Ignore fields / sections within log event

Introduction

Whenever possible, we recommend using key-value pairs to extract attributes from log events. This is only possible if the parameters and their values are clearly and consistently delimited. When extracting attributes by position, you may encounter situations where it's not desirable to extract all of the data within the log line. Inline regular expressions can be used to navigate around these unneeded fields.

Example

2021/08/09 08:35:54 192.168.4.3 PID 1823 Request ID: c24076f1 Msg : Loading application for user: rburgundy
2021/08/09 08:35:55 192.168.4.10 PID 915 Request ID: 3a25f816 Msg : Loading application for user: ckind bfantana btamland
2021/08/09 08:35:56 10.0.0.1 PID 1401 Initiating db connection 3, output was: test test test blah blah blah

As you can see from this example, the log event is not suitably delimited for key / value pair extraction. We could create a simple parser to extract the fields:

For the purposes of our explanation, the Request ID field is also not important to us. Since its value changes, we can't ignore "c24076f1" as a fixed value. Instead, we implemented a regular expression to do so. 

Output

2021/08/09 08:35:54 192.168.4.3 PID 1823 Request ID: c24076f1 Msg : Loading application for user: rburgundy
fromIP: 192.168.4.3
message: ... Truncated ...
pid: 1823
timestamp: 2021/08/09 08:35:54 (parsed as: Mon Aug 9, 2021 8:35:54 AM GMT, i.e. 918 minutes ago)
user: rburgundy

2021/08/09 08:35:55 192.168.4.10 PID 915 Request ID: 3a25f816 Msg : Loading application for user: ckind bfantana btamland
fromIP: 192.168.4.10
message: ... Truncated ...
pid: 915
timestamp: 2021/08/09 08:35:55 (parsed as: Mon Aug 9, 2021 8:35:55 AM GMT, i.e. 918 minutes ago)
user: ckind bfantana btamland

2021/08/09 08:35:56 10.0.0.1 PID 1401 Initiating db connection 3, output was: test test test blah blah blah
fromIP: 10.0.0.1
instance: 3
message: ... Truncated ...
pid: 1401
timestamp: 2021/08/09 08:35:56 (parsed as: Mon Aug 9, 2021 8:35:56 AM GMT, i.e. 918 minutes ago)