Update Log Display with Parser

Introduction

A customer inquired about modifying log events which contain escaped newline characters ("\n") to display the newlines instead. Since each log event contained a lengthy stack trace, the escaped newlines made them nearly impossible to read. Furthermore, since everything was neatly stored within a single log event, using a lineGroupers statement wouldn't have been optimal.

Newline Handling

Our display routine will automatically escape any newlines it encounters within a given log event. However, the log event that was received will not be changed.

In this example, the customer uploaded log events which contained escaped whitespace characters (newline and tab).

Log Example

{"@timestamp": "2021-06-22T19:51:47Z", "@level": "error", "exception": "Exception in thread \"main\" java.lang.NullPointerException\n\tat com.example.myproject.Book.getTitle(Book.java:16)\n\tat com.example.myproject.Author.getBookTitles(Author.java:25)\n\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\n"}

Parser

{
  formats: [
    {
      format: "$=json{parse=json}$",
      rewrites: [
        {
          input: "@timestamp",
          output: "timestamp",
          match: "([0-9\\- ,:]+)",
          replace: "$1"
        },
        {
          input: "@level",
          output: "severity",
          match: "([a-zA-Z]+)",
          replace: "$1"
        }, 
        {
          input: "message",
          output: "message",
          match: "\\\\n",
          replace: "\n",
          replaceAll: true
        },
        {
          input: "message",
          output: "message",
          match: "\\\\t",
          replace: " ",
          replaceAll: true
        } 
      ]
    }
  ]
}

Notes

• The built-in JSON parser extracts all fields into their own attributes. We then use the rewrites statement to move some of these fields into the correct built-in DataSet fields (timestamp and severity).
• Why are we modifying the message field? Won't the stack trace be associated with the exception field (which is extracted by the JSON parser?)
  • The message field is the full and unmodified log event that is uploaded to DataSet. Each log event which is uploaded to DataSet by the Scalyr Agent will have an associated message field. Since our search UI displays it by default, we update the formatting of the message field
  • Depending on your use case, the Search display could also be modified to display the exception field. However, if the exception field is not present in the log line, an empty column will be displayed instead, which may not be desireable.
• This parser will replace any escaped newline (\n) or tab (\t) characters in the message field with actual newlines (or two spaces, for the tab).
• To improve performance, we deliberately avoided using wildcards (ex. .*) in the match regex of the rewrites statement.

Search UI

The log is displayed on a single line in the Search UI before the parser is applied:

Once the parser is applied, the same log is a lot easier to read!