Introduction
Video Explanation
We created the following video to explain the following process
Logfile from video
{"ts":1604429125651,"elapsed":0,"spanId":"043932c7-b0f7-498b-9ce7-4418f114f747","action":"start","operation":"profile","tags":{"request":"x610250375","user":"u3296379"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"043932c7-b0f7-498b-9ce7-4418f114f747"},"references":{"child_of":"60847f29-425b-4cf5-a4f9-6783c98fe73d"}}
{"ts":1604429125651,"elapsed":0,"spanId":"f3ea10d7-7e23-40d8-bfb0-91d3aa91ba91","action":"start","operation":"fetchUserRecord","tags":{"request":"x610250375","user":"u3296379","table":"users"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"f3ea10d7-7e23-40d8-bfb0-91d3aa91ba91"},"references":{"child_of":"043932c7-b0f7-498b-9ce7-4418f114f747"}}
{"ts":1604429125655,"elapsed":4,"spanId":"f3ea10d7-7e23-40d8-bfb0-91d3aa91ba91","action":"finish","operation":"fetchUserRecord","tags":{"request":"x610250375","user":"u3296379","table":"users","status":"success"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"f3ea10d7-7e23-40d8-bfb0-91d3aa91ba91"},"references":{"child_of":"043932c7-b0f7-498b-9ce7-4418f114f747"}}
{"ts":1604429125655,"elapsed":0,"spanId":"3c793000-8c92-42f9-bf9a-435e45e92265","action":"start","operation":"issueTableQuery","tags":{"request":"x610250375","query":"SELECT * FROM albums WHERE users contains '?'"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"3c793000-8c92-42f9-bf9a-435e45e92265"},"references":{"child_of":"043932c7-b0f7-498b-9ce7-4418f114f747"}}
{"ts":1604429125761,"elapsed":106,"spanId":"3c793000-8c92-42f9-bf9a-435e45e92265","action":"finish","operation":"issueTableQuery","tags":{"request":"x610250375","query":"SELECT * FROM albums WHERE users contains '?'","status":"success"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"3c793000-8c92-42f9-bf9a-435e45e92265"},"references":{"child_of":"043932c7-b0f7-498b-9ce7-4418f114f747"}}
{"ts":1604429125761,"elapsed":0,"spanId":"11e7eb86-b774-445e-b852-81aa37a3ea37","action":"start","operation":"issueTableQuery","tags":{"request":"x610250375","query":"SELECT * FROM photos WHERE owner = '?' ORDER BY date_created DESC limit 10"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"11e7eb86-b774-445e-b852-81aa37a3ea37"},"references":{"child_of":"043932c7-b0f7-498b-9ce7-4418f114f747"}}
{"ts":1604429125774,"elapsed":0,"spanId":"3dfe5fe0-8f42-4acc-a07a-ecaf295c4083","action":"start","operation":"album","tags":{"request":"x991787919","user":"u9439176"},"baggage":{"traceId":"5d23f0db-4f48-4479-9e14-8eb7fcb052d8","reporter.spanId":"3dfe5fe0-8f42-4acc-a07a-ecaf295c4083"},"references":{"child_of":"5d23f0db-4f48-4479-9e14-8eb7fcb052d8"}}
{"ts":1604429125774,"elapsed":0,"spanId":"2bfd6e8e-36a7-4a62-9ca0-be6aafb4d04a","action":"start","operation":"fetchUserRecord","tags":{"request":"x991787919","user":"u9439176","table":"users"},"baggage":{"traceId":"5d23f0db-4f48-4479-9e14-8eb7fcb052d8","reporter.spanId":"2bfd6e8e-36a7-4a62-9ca0-be6aafb4d04a"},"references":{"child_of":"3dfe5fe0-8f42-4acc-a07a-ecaf295c4083"}}
{"ts":1604429125790,"elapsed":16,"spanId":"2bfd6e8e-36a7-4a62-9ca0-be6aafb4d04a","action":"finish","operation":"fetchUserRecord","tags":{"request":"x991787919","user":"u9439176","table":"users","status":"success"},"baggage":{"traceId":"5d23f0db-4f48-4479-9e14-8eb7fcb052d8","reporter.spanId":"2bfd6e8e-36a7-4a62-9ca0-be6aafb4d04a"},"references":{"child_of":"3dfe5fe0-8f42-4acc-a07a-ecaf295c4083"}}
{"ts":1604429125790,"elapsed":0,"spanId":"dadf3222-9f78-4bb8-a07a-688410044796","action":"start","operation":"issueTableQuery","tags":{"request":"x991787919","query":"SELECT * FROM photos WHERE album = '?' ORDER BY date DESC limit 100"},"baggage":{"traceId":"5d23f0db-4f48-4479-9e14-8eb7fcb052d8","reporter.spanId":"dadf3222-9f78-4bb8-a07a-688410044796"},"references":{"child_of":"3dfe5fe0-8f42-4acc-a07a-ecaf295c4083"}}
{"ts":1604429125994,"elapsed":233,"spanId":"11e7eb86-b774-445e-b852-81aa37a3ea37","action":"finish","operation":"issueTableQuery","tags":{"request":"x610250375","query":"SELECT * FROM photos WHERE owner = '?' ORDER BY date_created DESC limit 10","status":"success"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"11e7eb86-b774-445e-b852-81aa37a3ea37"},"references":{"child_of":"043932c7-b0f7-498b-9ce7-4418f114f747"}}
{"ts":1604429125994,"elapsed":0,"spanId":"122d949a-f167-4de3-b0a1-022d4264d3ad","action":"start","operation":"issueTableQuery","tags":{"request":"x610250375","query":"SELECT * FROM friends WHERE a = '?' ORDER BY date DESC limit 10"},"baggage":{"traceId":"60847f29-425b-4cf5-a4f9-6783c98fe73d","reporter.spanId":"122d949a-f167-4de3-b0a1-022d4264d3ad"},"references":{"child_of":"043932c7-b0f7-498b-9ce7-4418f114f747"}}
{"ts":1604429126021,"elapsed":0,"spanId":"c04b85bd-5894-4941-85e6-792dbe0466a9","action":"start","operation":"photo","tags":{"request":"x654875134","user":"u4903159"},"baggage":{"traceId":"0e4b6c5d-564a-4fc8-9166-990034242779","reporter.spanId":"c04b85bd-5894-4941-85e6-792dbe0466a9"},"references":{"child_of":"0e4b6c5d-564a-4fc8-9166-990034242779"}}
This article explains how to use our log upload utility. We developed this handy web application to enable new customers to test out DataSet's features with their log data and minimal configuration.
Please note that this application was designed to quickly demonstrate DataSet's functionality to new customers by using their log data.
There are 3 production-ready methods to upload log data to DataSet:
- Scalyr agent
- DataSet API
- Integrations with various third-party platforms (ex. Kafka, Fluentd, AWS, etc.)
All 3 methods are capable of handling terabytes of daily log volume.
Getting Started
The upload file app can be pulled from the Docker hub to your local machine and run as a Docker container. Here are the steps to run the application:
1. Pull the Docker images to your machine
docker pull weili808/uploadfile
2. Run the Docker container and map the application to the local port 3000
docker run -d --name uploadfile -p 3000:3000 weili808/uploadfile
3. Verify that the app is up and running
weilil@C02F348AMD6R ~ % sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
aeab4dd24e2d weili808/uploadfile "gunicorn app:app --…" 18 seconds ago Up 18 seconds 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp uploadfile
4. Open a browser and go to 127.0.0.1:3000 to upload a file to DataSet
You can simply drag and drop or select a log (text) file from your local machine and all of the data will be available within your DataSet account in a matter of seconds. Note that the "serverHost" of the messages will be the Docker container ID.
Some other features of the app:
- Allows the uploading of multiple text-formatted files (simply drag and drop them into the box).
- File size limit of 25MB
- Enables selection of predefined (use the drop-down list), or user defined parsers
- Immediate deletion of files (after they have been uploaded to DataSet)
We recommend defining your own parser(s) in advance of uploading any logs. You can build the parser on your own, or ask us to edit it for you on the parsers page. In addition, we have a parser tester that makes it easy for you to debug your parser against actual log data before deploying it.
Our Support center has helpful knowledge base articles, a video series, and best practices for creating parsers and running searches.
Feel free to send us your feedback or report any issues you found by submitting a ticket to the Support team. We look forward to hearing from you!
Comments
0 comments
Please sign in to leave a comment.