Message Size Limit
Log lines are limited to 99,900 characters in length. If the message's size exceeds this limit, it is split into multiple lines.
WARNING: The message which is split up might not be parsed correctly. As a result, the lines may not be grouped together on the UI. We encourage our customers to optimize their log's formatting / structure prior to uploading it to DataSet, as this tends to reduce log volume while improving the usability of data.
Attribute Limits
What are Attributes?
Attributes are key-value pairs that are extracted from log events by parsers. They can also be assigned by the Scalyr Agent, CLI, or API. Dynamic parsers automatically extract the keys and values by the data's format (ex. JSON, Python dict, custom structure, etc.), while individual attributes are explicitly extracted by position (ex. $keyname$,$value1$,
etc.). When you click a log event, attributes are displayed in a panel. For example,
Example
In this example, 5 attributes (timestamp
, severity
, topic
, trace
, and version
) were extracted with a parser, while 7 server level attributes (launchTime
, logfile
, parser
, serverHost
, serverIP
, session
, and threadId
) were assigned by the Scalyr Agent.
If this account submitted 100 instances of the same log over a 5 minute period, and
- No other log lines were sent, and
- Each attribute had a unique value
The account would use 12 of the 4000 total available (unique) attributes.
Attribute Limits
Total Attribute Limit
The maximum unique attributes per account is 4,000 over a rolling 5 minute interval. The total number of attributes must be greater than or equal to the sum of dynamic and event attributes; our platform will discard attributes when this limit is exceeded. This limit cannot be adjusted.
There are two different attribute types:
- Dynamic Attributes
- Attributes per Event
Dynamic Attributes
When you use a key-value pair, Python dict, or JSON parsing statement (for example, $=json{parse=json}$
), you are extracting dynamic attributes. There is a limit of 1,000 dynamic attributes per 5 minute interval. If you're primarily parsing key-value formatted data, this limit can be expanded upon request (but cannot exceed the total attribute limit).
Attributes per Event
Values that are extracted by their location are per-event attributes. For example, if you are extracting fields from a syslog log event, fields are extracted and statically assigned to an attribute by their location in a delimited string. Each log message can consist a maximum of 200 parsed attributes. Like the dynamic attributes, this limit can be increased on request.
Pro Tips
If you notice that attributes are not consistently being extracted from your log data,
- Confirm that your parser works as expected,
- Verify that the overall format of the log in question hasn't changed,
Try searching for tag = "attributeLimitInfo"
to verify whether your account reached the attribute limit. Please note that the querytag = "attributeLimitInfo"
only returns a result if your account is above half of the limit.
This feature allows you to directly view the number of attributes you're using.The search's response should include an attributeCount
attribute which shows the number of attributes within its assigned 5-minute interval. If any attributes were discarded, the following fields will be available for review:
discardedAttributes
– the number of unique attributes which were discarded (i.e. how many different attribute names)discardedAttributeInstances
– the number of discarded attribute instances. If we discard attributes "x" and "y" from 10 events, this will be 20 butdiscardedAttributes
will be 2.eventsWithDiscardedAttributes
– the number of events which had at least one discarded attributesampleDiscardedAttribute
– the name of one of the discarded attributes. Looking at these values might give us a general idea of which attributes are causing trouble.
Comments
0 comments
Please sign in to leave a comment.