Extract timezone on the fly

Introduction

In most scenarios, we recommend using the UTC timezone for your system logs, as this removes the need to adjust for DST changes. If multiple time zones are used within the same log, it is possible to extract the timezone offset from each event to ensure the original timestamp is preserved.

Example

Log events are from different time zones. Each timestamp contains the associated time zone:

Aug 06 08:10:23--7:00 data1 data2
Aug 06 10:10:24--5:00 data1 data2
Aug 06 11:10:24--4:00 data1 data2

Parser

1. We use pattern (defined within the patterns section) to extract the entire timestamp string (including timezone offset)

2. The timestamp string is placed in the attribute ts for further processing

3. A rewrites rule is used to extract the date and time in a format that the DataSet platform recognizes (most standard timestamp formats are automatically identified) and placed within the reserved timestamp field

4. The timezone is subsequently extracted from the ts attribute. Some minor restructuring is needed for it to be recognized

Result

Aug 06 08:10:23--7:00 data1 data2
message: Aug 06 08:10:23--7:00 data1 data2
timestamp: Aug 06 08:10:23 (parsed as: Fri Aug 6, 2021 3:10:23 PM GMT, i.e. 398 minutes ago)
timezone: GMT-7:00
ts: Aug 06 08:10:23--7:00

Aug 06 10:10:24--5:00 data1 data2
message: Aug 06 10:10:24--5:00 data1 data2
timestamp: Aug 06 10:10:24 (parsed as: Fri Aug 6, 2021 3:10:24 PM GMT, i.e. 398 minutes ago)
timezone: GMT-5:00
ts: Aug 06 10:10:24--5:00

Aug 06 11:10:25--4:00 data1 data2
message: Aug 06 11:10:25--4:00 data1 data2
timestamp: Aug 06 11:10:25 (parsed as: Fri Aug 6, 2021 3:10:25 PM GMT, i.e. 398 minutes ago)
timezone: GMT-4:00
ts: Aug 06 11:10:25--4:00

Despite the difference between timestamps, each log event is converted to the correct GMT time since to the timezone is adjusted