Diagnose DataSet API Errors

Introduction

DataSet supports purpose-driven ingestion and query APIs. When you integrate our API into your applications, it's important to understand the status code responses and best practices in order to optimize the API's usage. We've created the following list of status codes and resolutions to streamline customer integrations.

HTTP code 500

This usually indicates a server-side issue. We push new code to production weekly, so if the error only lasts for a few seconds during US business hours (Pacific Time), this is likely the cause.

Please note that our SRE team monitors our platform 24/7, and we actively update the status page if any server issues occur. However, in the unlikely event that you receive status code 500s for an extended period of time, please reach out to us (support@dataset.com).

HTTP code 429

DataSet API enforces a rate limit on the number of requests per sec/min/hr and the limit varies for each API call. We also limit the allocated resources (ex. CPU execution time) per account. If your API activity exceeds one or more of these thresholds, a 429 error is returned. Generally speaking, these errors can be mitigated by:

• Batching API requests

  • If addEvents is returning 429 errors, please verify that your API isn't using a hard-coded session ID. Please see our documentation for more information. addEvents is not designed to support a high rate of requests, however it supports batching up to 6MB (before compression) of log events per call.
    

• Reducing complexity and / or timeframe of API-issued search queries
• Reducing frequency of initiated API calls

If your application requires additional resources, please contact us and we'll help you find a solution.

Logs missing despite HTTP code 200

You've issued the addEvents API and received the following response, but can't locate your logs:

{
    "bytesCharged": 0,
    "status": "success"
}

The "status":"success" message confirms the API had the correct syntax, but it doesn't guarantee the event was uploaded to DataSet with the specified timestamp.

1. Confirm that the timestamp (passed via the events.ts attribute) is correct (example). It should be a string that represents the UNIX epoch in nanoseconds (ex. "1667443947000000000").
2. Please note that if the timestamp is not current, you will need to expand your search to include the date range when the log event(s) in question occurred.
   
3. If the timestamp exceeds your log retention period (ex. timestamp is from 32 days ago and you have 30 day log retention), the log event will be ingested, but not stored