The log volume ingested by DataSet is the basis for billing, so it's important for customers/prospects to understand how this volume is calculated, in order to optimize your logs and agent configurations to get the best value out of the service.
Here is an explanation of what is counted toward your log volume and what is not:
- Raw log messages are counted
- Metalog events are not counted
sessionInfo/serverInfofields (app, launchTime, parser, session, machine, serverType, serverScope, sessionType, serverHost) are not counted- The logfile attribute, as well as internal k8s attributes ("containerName", "containerId", "pod_name", "pod_namespace", "namespace", "pod_uid", "k8s_container_name", "original_file", "scalyr-category", "k8s_node", "container_id") aren't counted
- Other than the above, attribute values that originate in the Agent or at the DataSet API are counted. Attribute name lengths aren't counted, but each attribute costs 1 byte. (Attributes that are created by the parser are not counted.)
- There is no separate charge for the contents of the message attribute. You are only billed once for the original log event.
- Having no attributes would theoretically cost 1 byte
Comments
1 comment
Hi Mark!
A bit unclear on attribute costs.
Was this meant to be "each attribute costs at least 1 byte?"
Cool so if from the log event that looks like this:
[29/Apr/2022:13:49:15 +0000] "POST /auth/api/15/feature-flags/context HTTP/1.1" 200 200 "-"We parse out the following part into the attribute called `msg`
"POST /auth/api/15/feature-flags/context HTTP/1.1" 200We will not pay extra for `msg`? (apart maybe for the `index:length` that you will have to store for each event to seek into the raw log to get the `msg` part)
And yeah another question was meant to be - do I understand correctly that the way you're storing attribute values is basically `index:length` pairs with which you can find the attribute's value in the raw message, and you only retrieve that value when needed and never store it separately? Do you then also have to index the attribute for search? I'd imagine indexing taking some space as well - if it does can we opt out of indexing for some attributes?
Please sign in to leave a comment.