Diagnose DataSet API ingestion Errors

DataSet supports multiple types of APIs for both ingestion and query. It's so simple to use and it's very popular among our customers. When you integrate our API into your applications, it's important to understand the returning status code means and the best practice to optimize the API's usage.

Here are some of the common issues you might see and I am going to provide the general guidelines to resolve them.

Getting HTTP code 500

This usually indicates a server-side issue, so if you are getting status code 500 from either query or ingestion for an extended period of time, please reach out to us (support@dataset.com).

We push new code to production weekly, so if the error only lasts for a few seconds, it could just be your server is temporarily unavailable during deployment. In addition to that, our SREs also monitor the production servers 24/7, so we do actively update the status page as soon as we identify any server issues that may cause DataSet to be unavailable to any user.

Getting HTTP code 429

This means that your account reaches a limit. DataSet API enforces a rate limit on the number of requests per sec/min/hr and the limit varies for each API call. Besides that, we also limit the allocated resource (ex. CPU execution time) per account. If you could pause longer between each call, you might just see the error go away. If your application requires to go beyond the default limit assigned to your account, you could send us a ticket and we'll help you find a solution.

If you're getting 429 from addEvents API calls, it's worth verifying that your API doesn't have a hard-coded session ID. /addEvents is not designed to support a high rate of requests to begin with, so appending a single session ID will definitely cause the failures to manifest quickly.

Getting HTTP code 200 but I can't find the event on DataSet

Another common issue we've seen from users is that they don't see the event on their accounts despite the API call returns 200.

{
    "bytesCharged": 0,
    "status": "success"
}

The status:"success" message merely tells users that the call has the correct syntax, but it doesn't guarantee the event is uploaded to DataSet with the specified timestamp. The first thing to verify is that the timestamp ("events.ts") is a string type and the unit is in "nanoseconds" UNIX time. If the timestamp is older than 10-20 minutes from the current time, DataSet considers the event as a stale event and would assign the closest possible time as the event's timestamp.