Evaluating Log Volume

Introduction

DataSet has a number of helpful features that make it easier for customers to manage their log volume. If you are concerned about the log volume of particular files, or would like to learn more about how you can quickly evaluate log volume, please continue reading!

New and Improved Log Volume Dashboard

As a starting point, our "Log volume dashboard" can provide some insight. To access it, click "Dashboards" -> "log Volume"

Log Volume (Over Time)

This graph represents the approximate (smoothed average) log volume in GB/day. In other words, the sumPerSecond values presented by the graph are interpreted as: "If the observed log volume were sustained for 24 hours, it would be approximately __ GB/day."

An explanation of how we calculated this (bytes/sec -> GB/day) can be found here.

By clicking on the "Expand" button (magnifying glass) in the upper right corner of the graph, you can modify the criteria in graph mode. For example, you could analyze the approximate log volume for all the /var/log/mysql-log file as follows:

sumPerSecond(value where tag='logVolume' metric='logBytes' forlogfile='/var/log/mysql-log' ) * 0.00008046627

Available parameters can be obtained by viewing the logs responsible for this calculation -- simply click "Show Logs":

Based on our example, fields like tag, path1, and metric can be used to improve the relevance of graphed results, while the value field is used for calculation.

Log Volume (by server/host)

This graph contains a breakdown of the logs with highest volume by the hosts they originated from.

You can generate your own graph to evaluate log volume as needed with a PowerQuery. For example, the PowerQuery below displays logs with the highest volume by host by hour:

tag='logVolume' metric='logBytes' !(k8s-cluster=*) forlogfile!="none"
| group bytes=sum(value) by timestamp=timebucket("1h"), forlogfile, host
| sort -timestamp,-bytes

For another PowerQuery-based example how how to evaluate log volume, please see: How to Breakdown Log Volume by a Custom Field

Graphing Log Volume (on the fly)

Starting from the "Search" page, identify the attributes associated with the log(s) you wish to evaluate. You can get a quick overview of any values and the frequencies associated with them by hovering over attributes in the left column of the Search:

When you identify a value that you wish to graph, click the "Graph <>:" button. The resultant graph contains additional aggregation functions for evaluating your data.

In the above example, the log volume reports rely on a metric that is captured in our metalog -- you can see it by searching for tag="logVolume" metric="logBytes". There is a second metric, metric="logEvents" which captures the number of log events.

To compare log events between weeks:

1. Do a new search for tag="logVolume" metric="logEvents" over the last 24 hours

2. Go to the "value" facet and click "Graph Values"

3. On the "Compare" menu, select "Prev Week"

Cost Management

DataSet provides a number of proactive methods that can be used to help reduce your log volume. For example, you can establish categories for the various log types that are uploaded to DataSet. From here, you can:

• Discard unwanted log categories -- discarded logs do not count against your monthly bill
• Send overage notifications if a category / categories exceed a defined amount
• Change your plan to accommodate your current / anticipated log volume

More information on our Cost Management features can be found here: Maximizing Value with Cost Management