Welcome!

Congratulations! If you're reading this, you have probably installed the Scalyr Agent and begun uploading your logs. (If not, you might want to hop back to the Agent Installation page and get that out of the way.)

Here, we'll give a quick introduction to the DataSet site. With DataSet's unified approach to server monitoring, you can perform a broad array of tasks — from basic uptime monitoring and error alerts, to capacity planning, bug forensics, performance investigations, and more. It's worth spending a couple of minutes to learn the basic concepts before you dive in.

ALL SERVER DATA IN ONE PLACE

DataSet provides a home for all of your server data, from system metrics to logs to monitoring probes. Each log message or monitoring report becomes an event, consisting of one or more fields. For instance, consider this entry from a web access log:

72.46.140.106 - - [06/Mar/2014:14:04:15 +0000] "GET /healthcheck HTTP/1.1" 301 - "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" 1 "-"

Some of the fields in this event:

ip: 72.46.140.106
method: GET
uriPath: /healthcheck
protocol: HTTP/1.1
status: 301
agent: Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)

Fields are a powerful tool for searching and analyzing data. You can group your web traffic by URL to find the most popular, largest, or slowest pages. You can see which pages are consuming the most bandwidth, or triggering the most errors. You can graph responses times and sizes, and alert if the average response time exceeds some threshold. And with all your server data in one place, you can combine system metrics with access logs in a single dashboard, or generate alerts from both log messages and external monitoring probes.

SERVERS, LOGS, AND EVENTS

Each event is associated with the server (aka "host") it came from. In the Scalyr Agent configuration, you can specify fields for a server — for instance:

server: staging-frontend-7
tier: frontend
group: staging
datacenter: aws-us-east-1a

A server's fields are attached to every event from that server. You can use these fields to organize data, graphing response times for staging servers in us-east, or alerting if there are errors on any production database server in any data center. DataSet does not dictate server fields; you can organize your servers using any field names and values you like. You can also specify fields for each log, to distinguish between services running on the same machine.

LOG PARSING

DataSet's tools gather some data for you automatically, such as server CPU and disk metrics. These events are automatically organized into fields:

metric: proc.stat.cpu_rate
type: iowait
value: 7.193473108

But for logs, a log parser is needed to identify fields. We provide parsers for web access logs and MySQL and PostgreSQL database logs, among others. For other logs, you can create your own parser using our powerful tools, or just click a button and we'll take care of it for you (no extra charge!).

WORKING WITH DATA

To dive into your server data, the usual place to start is the Overview link in the navigation bar. On this page, you should see each of your servers, with links for each log file on that server. Click on a log to view it:

From here, you can search your logs by typing in the Expression box. To search for a single word, just type it; to include multiple words or punctuation, place quotes around your search text. Note that the Expression box isn't empty — it has already been filled in with a terms that narrow your view to a particular log from a particular server. You can edit or remove these terms to explore multiple log files or logs from multiple servers.

To work with fields, click the Refine search by button. This will show fields that appear in this log. Move the mouse to any field to see the most common values for that field, and click to show events with that value. (Note that a field won't be listed here if all matching events have the same value in that field.)

Click on any event to see details, including all fields of the event, and the log and server it came from. Here you will also find an array of tools for finding events related to the selected event.

(To see everything you can do on this page, read the log viewing tools reference.)

ENOUGH READING

There's lots more to say, but we promised this introduction would be quick. From here, you can head to the Solutions Gallery and start getting things done. If you like to read more before diving in, the links in the upper-left side of the page provide complete documentation for everything you can do with DataSet. Or if you'd rather just dive in and explore, head to that Overview link in the navigation bar — but you might want to come back here later and explore some of the other links.