addEvents - Best Practices

Introduction

The addEvents API gives customers full control over the structure of log messages that are submitted to DataSet. Due to the flexibility it offers, addEvents can be used to:

• Submit log events with chosen attributes. A parser is not needed in this use case, as the customer's application extracts fields, packages them in a JSON structure, and supplies the JSON to the addEvents call. Typically, a parser performs this routine after the log event has been received.
• Modify (or completely omit) the message field
• Upload (batch) multiple events per addEvents call to improve efficiency

Since addEvents is a standalone API, it requires the user to employ fault tolerance (in most cases, a retry mechanism). However, it is well suited for integration within applications and other scenarios where a specific logging solution is needed.

This article contains some examples of how to best utilize the addEvents API.

Parsing logs that were uploaded by addEvents

As previously mentioned, addEvents may allow you to bypass parsing if your application is configured to extract the fields you wish to use as attributes.

A customer inquired about how to upload batches of timestamped logs from text files. These text files were the result of a process that had completed, however there was no guarantee that the log events were in chronological order. Each log event contained an ISO-8601 timestamp, which came in handy.

You may recall reading the following in the DataSet API documentation for addEvents:

DataSet uses timestamps internally to identify events, so the ts field must be strictly increasing — each event must have a larger timestamp than the preceding event. This applies to all /addEvents invocations for a given session; each session (identified by the session parameter to /addEvents) has an independent timestamp sequence. So one easy way to ensure valid timestamps is for each client to keep track of the last timestamp it used, and ensure that the next timestamp it generates is at least 1 (nanosecond) larger.

For this particular scenario, we recommend the following:

1. Utilize the ISO-8601 timestamp value that is provided with each log event
2. Specify a parser attribute within the addEvents API, set a recognizable value like "lambda_log"
3. Set the ts parameter as a Unix epoch (nanoseconds, so 20 digits long). Increment its value by 1 for each log event in the session
4. Configure the parser from step 2 to extract the timestamp from each log event. Since you are using a parser, you will be setting the timestamp attribute (instead of ts, which is specific to the addEvents API). The parser will override the timestamp set by ts since it occurs later in the ingestion process.

Numerical vs String values

DataSet will distinguish attribute values passed with either numbers or strings. For simplicity, further classifications (i.e. floats vs integers and lists vs strings) are not made. To ensure that numerical values are preserved, do not use quotations when defining the attribute. Conversely, string values should always be enclosed in quotation marks. For example,

...
      "attrs": {
        "fluff": 1000,
        "version": "1000",
        "greeting": "hello",
...

fluff is a numerical value of 1000, while version and greeting are both treated as strings.