Skip to main content

How DataSet Log Volume is Calculated


1 comment

  • Nikita Nefedov

    Hi Mark!

    A bit unclear on attribute costs.

    Attribute name lengths aren't counted, but each attribute costs 1 byte

    Was this meant to be "each attribute costs at least 1 byte?"

    Attributes that are created by the parser are not counted

    Cool so if from the log event that looks like this:

    [29/Apr/2022:13:49:15 +0000] "POST /auth/api/15/feature-flags/context HTTP/1.1" 200 200 "-"

    We parse out the following part into the attribute called `msg`

    "POST /auth/api/15/feature-flags/context HTTP/1.1" 200

    We will not pay extra for `msg`? (apart maybe for the `index:length` that you will have to store for each event to seek into the raw log to get the `msg` part)

    And yeah another question was meant to be - do I understand correctly that the way you're storing attribute values is basically `index:length` pairs with which you can find the attribute's value in the raw message, and you only retrieve that value when needed and never store it separately? Do you then also have to index the attribute for search? I'd imagine indexing taking some space as well - if it does can we opt out of indexing for some attributes?


Please sign in to leave a comment.