Optimize your Search Queries: Attributes

Introduction

This article contains an overview of searching with attributes. Attributes are either assigned by the ingestion mechanism (more information on that below) or extracted via a parser. They are considered to be the most effective way to conduct searches on the DataSet platform. Attributes are also extremely versatile -- once created, they can be used in searches, graphs, dashboards, PowerQueries, and alerts.

The examples provided here originated from customer inquiries and our experience with the DataSet product.

Searching with Attributes

Whenever possible, it's recommended to use attributes to search for specific log events, as this significantly improves search speed and performance.

For example, given the complete log event:

Search the message attribute

Search parameters which do not specify an attribute are applied to the reserved message field (which contains the entire log event). The following would search the message field for a value of "75c0509a":

"75c0509a"

Search a specific attribute

A more effective search would target the spanId attribute (screenshot from "Inspect Log Line" dialog):

spanId contains "75c0509a"

This is provided you know the attribute contains the value you're searching for.

Locate an attribute

If you don't know which attribute contains the value you're looking for, you could review all attributes which contain the value you're searching for:

* contains "75c0509a"

From here, locate the attribute(s) in the facet bar of the Search page. Since you've identified the attribute(s), specify it by name within your query.

Excluding Values

Adding an exclamation point (!) as a prefix will negate the search query:

!(spanId contains "75c0509a")

Inspect Log Line Dialog

You can quickly review the attributes that are associated with a log event by clicking the event in the "Search" page. From there, review the "Inspect Log Line" dialog:

1. Complete Log Event (message attribute)

The complete log event is stored within the reserved message field and is displayed at the top of the dialog. This maintains the original log for analysis / review and is independent of parsing routines.

2. Attributes

Attributes displayed in this field are either assigned by the log ingestion mechanism or extracted from the log event by a parser.

Assigned Attributes

A quick way to identify assigned attributes is to check the message field.

• If no message field is present, or
• If the attribute value is not present within the message field,

The attributes below were assigned as no message field is present. The most common form of logs with assigned attributes is DataSet metalog data:

Otherwise, assigned attributes are explicitly configured by the customer when using the DataSet addEvents API, or our Logstash / Fluentd plugins. They do not occur without manual intervention. It is currently not possible to extract values from assigned attributes with a parser. However, you can use a PowerQuery with the parse from function to analyze these fields.

Parsed Attributes

The most common way that attributes are generated for log events. In order for an attribute to be parsed, its value must exist in the message field. DataSet parsers are comprised of regular expressions that extract values from log events. Please note that parsers do not support iteration (for, while), conditional logic (if-then-else, case), or mathematical operations. Some logical and mathematical operations can be performed by PowerQueries, however, this won't impact how log data is extracted by a parser.

3. "Server Fields" (server attributes)

Server attributes are assigned by the upload mechanism that processed the log event (ex. DataSet API, Agent, or integration (S3, CloudTrail, Okta, Logstash, etc.)). These fields are present for each log event and contain attributes which are related to the log's origin.

Server attributes are typically used for DataSet operations like log scrubbing, templated alerts (see byHosts section), and defining Cost Management categories since they're consistently available. Server fields like logfile, serverHost, and serverIP are commonly used to designate log sources.