Optimizing DataSet Ingestion API Usage

DataSet provides two types of RESTful APIs, addEvents and uploadLogs, for logs ingestions. Both APIs handle data ingestions, but the characteristics are different and a user should choose the one that is best suited for his/her use cases. 

Here is a brief summary of each API:

uploadLogs

• Easy to use
• Good for raw text ingestion
• Not to be used for intensive data ingestions (high average log volume or high-frequency ingestions)

uploadLogs is well suited to simple integrations and requires a minimal set of parameters to be supplied. Each call to uploadLogs sends a single log event to DataSet. Attributes are extracted from the log event upon ingestion by the specified parser. As such, it is important to use a well structured log format and extract the timestamp parameter (via the parser) to ensure that chronological order between events is maintained.

For instance, the quick-start app is implemented using uploadLogs API which reads the content of the file at once and makes a single request to bulk upload all messages to DataSet.

with open(file_path, "r") as f:
    content = f.read()

uploadLog_url = 'https://{scalyr_url}/api/uploadLogs?token={token}&host={host}&logfile={logfile}&parser={parser}' \
    .format(scalyr_url=scalyr_url, token=api_key, host='support.scalyr.com', logfile=raw_filename, parser=parser)

logging.info(uploadLog_url)
resp = requests.post(uploadLog_url, data=content.encode('utf-8'), headers={'content-type': 'text/plain'})

Note that the max size for a single uploadLogs API request is around 5MB, so if the data exceeds that limit, break it into chunks beforehand to ensure that requests are completed successfully. 

addEvents

addEvents has several distinct advantages, however, this comes at the expense of increased complexity:

• Offers granular control over attributes and contents of message field.
  • addEvents enables your application to define attributes. Rather than sending the entire log event and relying on a parser to extract fields, your application can directly create DataSet attributes.
    • Advantage: This approach may reduce your log volume, since only essential fields are transferred.
    • Disadvantage: Free-text search capabilities (ex. searching for "value0" wherever it occurs) may not work if you're sending an abbreviated message field. If this is the case, you will need to specify the attribute to search within when you are looking for a particular value (ex. attribute contains "value1").
    • Note: It is still possible to parse events that are sent via addEvents. To do so, include data that will be parsed in the message field, and specify the parser attribute in the attrs field.
• Can be optimized for higher log volume
  • addEvents is intended to send multiple log events at once. This reduces the frequency of API calls, as log events can be grouped and sent at fixed intervals (rather than 1 call per line).
  • However, there are several important considerations to note (see documentation).

IMPORTANT:

The rule of thumb for both uploadLogs and addEvents APIs is to group multiple log events within a single request, then send requests once every few seconds. DataSet is built on top of an extremely scalable and robust backend, so we have a stronger tolerance against malicious attacks relative to other platforms. To protect our servers from excessive bursts of traffic, we rate limit API calls from individual accounts. Consequently, requests which exceed the above limit could be dropped.

The log retrieval functions provided by Google Cloud prevent logs from being batched within the DataSet addEvents API. Please note that it is prohibited to use the uploadLogs or addEvents APIs to send one log event at a time from your Google Cloud application, as this is will have detrimental effects on your account's performance. Furthermore, issuing thousands of API requests over the course of several seconds is considered a misuse of our API. 

Instead, we recommend ingesting Google Cloud logs with our pub/sub monitor, as it is capable of grouping thousands of Google Cloud logs before making an ingestion request to DataSet.