Introduction
lineGroupers are used to combine log events that occurred as separate lines. This preserves the order and association between log lines; however, multi-line events need to be delimited for this feature to work properly. This article provides some best practices when working with aggregated events.
Using lineGroupers
When configuring a parser, lineGroupers are one of the first sections that you will define. Consequently, when the lineGroupers statement is activated, any format rules within the parser will apply to the grouped block of text rather than individual lines.
Delimiting Multi-Line Log Events
As mentioned above, a delimiter at the beginning (and if necessary, the end) of each multi-line event are required. At minimum, delimiters should be consistent, recognizable patterns that can be defined as regular expressions and do not occur within the log. Whitespace (or sequences of whitespaces) should not be used for this purpose. For example, a timestamp followed by severity and other parameters signals the beginning of a new series of log events.
Parsing Individual Log Lines
Once log events have been grouped together, individual log lines can be extracted. The primary difference is that these events are separated by whitespace, so the regex is slightly different. Consider the following Python Traceback outputs:
2021-08-10 12:26:35 ERROR .::. test application : Traceback (most recent call last):
File "test.py", line 3, in <module>
hello_world(1)
File "test.py", line 5, in hello_world
...
TypeError: must be str, not int
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'tuple' object has no attribute 'append'
Time: Aug 10 12:35:28
Severity: ERROR
-----
Parser
Given the parser above, each log event is assembled from the lines that comprise it. The standard cascaded order of processing still applies.
Output
2021-08-10 12:26:35 ERROR .::. test application : Traceback (most recent call last):
File "test.py", line 3, in
hello_world(1)
File "test.py", line 5, in hello_world
...
TypeError: must be str, not int
message: ... Truncated ...
severity: ERROR
timestamp: 2021-08-10 12:26:35 (parsed as: Tue Aug 10, 2021 7:26:35 PM GMT, i.e. 33 minutes ago)
typeErrorMsg: must be str, not int
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'tuple' object has no attribute 'append'
Time: Aug 10 12:35:28
Severity: ERROR
-----
message: ... Truncated ...
severity: ERROR
timestamp: Aug 10 12:35:28 (parsed as: Tue Aug 10, 2021 7:35:28 PM GMT, i.e. 24 minutes ago)
Comments
0 comments
Please sign in to leave a comment.