addEvents API gives customers full control over the structure of log messages that are submitted to DataSet. Due to the flexibility it offers,
addEvents can be used to:
- Submit log events with chosen attributes. A parser is not needed in this use case, as the customer's application extracts fields, packages them in a JSON structure, and supplies the JSON to the
addEventscall. Typically, a parser performs this routine after the log event has been received.
- Modify (or completely omit) the
- Upload (batch) multiple events per
addEventscall to improve efficiency
addEvents is a standalone API, it requires the user to employ fault tolerance (in most cases, a retry mechanism). However, it is well suited for integration within applications and other scenarios where a specific logging solution is needed.
This article contains some examples of how to best utilize the
Parsing logs that were uploaded by
As previously mentioned,
addEvents may allow you to bypass parsing if your application is configured to extract the fields you wish to use as attributes.
A customer inquired about how to upload batches of timestamped logs from text files. These text files were the result of a process that had completed, however there was no guarantee that the log events were in chronological order. Each log event contained an ISO-8601 timestamp, which came in handy.
You may recall reading the following in the DataSet API documentation for
DataSet uses timestamps internally to identify events, so the ts field must be strictly increasing — each event must have a larger timestamp than the preceding event. This applies to all /addEvents invocations for a given session; each session (identified by the session parameter to /addEvents) has an independent timestamp sequence. So one easy way to ensure valid timestamps is for each client to keep track of the last timestamp it used, and ensure that the next timestamp it generates is at least 1 (nanosecond) larger.
For this particular scenario, we recommend the following:
- Utilize the ISO-8601 timestamp value that is provided with each log event
- Specify a
parserattribute within the
addEventsAPI, set a recognizable value like "lambda_log"
- Set the
tsparameter as a Unix epoch (nanoseconds, so 20 digits long). Increment its value by 1 for each log event in the session
- Configure the parser from step 2 to extract the
timestampfrom each log event. Since you are using a parser, you will be setting the
timestampattribute (instead of
ts, which is specific to the
addEventsAPI). The parser will override the timestamp set by
tssince it occurs later in the ingestion process.
Numerical vs String values
DataSet will distinguish attribute values passed with either numbers or strings. For simplicity, further classifications (i.e. floats vs integers and lists vs strings) are not made. To ensure that numerical values are preserved, do not use quotations when defining the attribute. Conversely, string values should always be enclosed in quotation marks. For example,
fluff is a numerical value of 1000, while
greeting are both treated as strings.
Please sign in to leave a comment.