Keep it simple!

Introduction

As a best practice, individual log formats should be stored within a single log file. This reduces noise when analyzing log events and simplifies the parser configuration and maintenance process.

If multiple log formats are sent to the same log file, the parser will become increasingly complex with each additional format. This may also complicate or prevent the use of key-value pair parsing.

Example

If you are uploading HTTP access, firewall, and application logs from 20 hosts to DataSet, three separate log files are defined for this purpose ("access.log", "firewall.log", and "marketplace_app.log"). The serverHost attribute (or other user-defined attributes) can be used to identify where the events within each log file originated from. Each log file has its own parser, so three parsers are created. This configuration requires minimal overhead yet ensures that parsers are scalable and efficient.

Configuration

The serverHost, parser, and logfile associated with each log event are set at the Agent or API level (configuration varies slightly by method). These fields are realized as server-level attributes, which means that every log event originating from a specific host will have the attribute assigned to it.