Parser Limits

Message Size Limit

Log lines are limited to 99,900 characters in length. Any log events which exceed this limit will be split into multiple lines.

WARNING: If a log event is split up, it might not be parsed correctly. As a result, the lines may not be grouped together in the UI. We encourage our customers to optimize their log's formatting / structure prior to uploading it to DataSet, as this tends to reduce log volume while improving the usability of data.

Attribute Limits

What are Attributes?

Attributes are key-value pairs that are extracted from log events by parsers. They can also be assigned by the Scalyr Agent, CLI, or API. Dynamic parsers automatically extract the keys and values by the data's format (ex. JSON, Python dict, custom structure, etc.), while individual attributes are explicitly extracted by position (ex. $keyname$,$value1$,). When you click a log event, attributes are displayed in a panel. For example,

Example

In this example, 5 attributes (timestamp, severity, topic, trace, and version) were extracted with a parser, while 7 server level attributes (launchTime, logfile, parser, serverHost, serverIP, session, and threadId) were assigned by the Scalyr Agent.

If this account submitted 100 unique instances of this log over a 5 minute period, and

• No other log lines were sent, and
• Each attribute had a unique value

12 total attributes would be assigned. This is similar to assigning variables within a programming routine -- 12 variables would be initialized, each with one or more values assigned to it.

Attribute Limits

Total Attribute Limit

There is no limit to the number of attributes that can be received from a customer account within a given timeframe. Please refer to the previous section for an explanation of how attributes are created.

Extracting Dynamic Attributes

When you use a key-value pair, Python dict, or JSON parsing statement (for example, $=json{parse=json}$), you are extracting dynamic attributes. There no limit to the quantity of attributes that can be dynamically extracted, however:

• The maximum attributes per event (1350) applies, and
• There is no limit to the traversal depth when working with structured JSON, however, there is an attribute name length limit of 200 characters

Maximum Attributes per Event

Values that are extracted by their location are per-event attributes. For example, if you are extracting fields from a syslog log event, fields are extracted and statically assigned to an attribute by their location in a delimited string. Each log message can consist of a maximum of 1350 parsed attributes.

Pro Tips

If you notice that attributes are not consistently being extracted from your log data,

• Confirm that your parser works as expected,
• Verify that the overall format of the log in question hasn't changed