Parser Limits

Message Size Limit

Log lines are limited to 99,900 characters in length. If the message's size exceeds this limit, it is split into multiple lines.

WARNING: The message which is split up might not be parsed correctly. As a result, the lines may not be grouped together on the UI. We encourage our customers to optimize their log's formatting / structure prior to uploading it to DataSet, as this tends to reduce log volume while improving the usability of data.

Attribute Limits

What are Attributes?

Attributes are key-value pairs that are extracted from log events by parsers. They can also be assigned by the Scalyr Agent, CLI, or API. Dynamic parsers automatically extract the keys and values by the data's format (ex. JSON, Python dict, custom structure, etc.), while individual attributes are explicitly extracted by position (ex. $keyname$,$value1$, etc.). When you click a log event, attributes are displayed in a panel. For example,

Example

In this example, 5 attributes (timestamp, severity, topic, trace, and version) were extracted with a parser, while 7 server level attributes (launchTime, logfile, parser, serverHost, serverIP, session, and threadId) were assigned by the Scalyr Agent.

If this account submitted 100 instances of the same log over a 5 minute period, and

• No other log lines were sent, and
• Each attribute had a unique value

The account would use 12 of the 4000 total available (unique) attributes.

Attribute Limits

Total Attribute Limit

The maximum unique attributes per account is 4,000 over a rolling 5 minute interval. The total number of attributes must be greater than or equal to the sum of dynamic and event attributes; our platform will discard attributes when this limit is exceeded. This limit cannot be adjusted. 

There are two different attribute types:

• Dynamic Attributes
• Attributes per Event

Dynamic Attributes

When you use a key-value pair, Python dict, or JSON parsing statement (for example, $=json{parse=json}$), you are extracting dynamic attributes. There is a limit of 1,000 dynamic attributes per 5 minute interval. If you're primarily parsing key-value formatted data, this limit can be expanded upon request (but cannot exceed the total attribute limit).

Attributes per Event

Values that are extracted by their location are per-event attributes. For example, if you are extracting fields from a syslog log event, fields are extracted and statically assigned to an attribute by their location in a delimited string. Each log message can consist a maximum of 200 parsed attributes. Like the dynamic attributes, this limit can be increased on request.

Pro Tips

If you notice that attributes are not consistently being extracted from your log data,

• Confirm that your parser works as expected,
• Verify that the overall format of the log in question hasn't changed,

Try searching for tag = "attributeLimitInfo" to verify whether your account reached the attribute limit. Please note that the querytag = "attributeLimitInfo" only returns a result if your account is above half of the limit.

This feature allows you to directly view the number of attributes you're using.The search's response should include an attributeCount attribute which shows the number of attributes within its assigned 5-minute interval. If any attributes were discarded, the following fields will be available for review:

• discardedAttributes – the number of unique attributes which were discarded (i.e. how many different attribute names)
• discardedAttributeInstances – the number of discarded attribute instances. If we discard attributes "x" and "y" from 10 events, this will be 20 but discardedAttributes will be 2.
• eventsWithDiscardedAttributes – the number of events which had at least one discarded attribute
• sampleDiscardedAttribute – the name of one of the discarded attributes. Looking at these values might give us a general idea of which attributes are causing trouble.