This article contains an overview of searching with attributes. Attributes are either assigned by the ingestion mechanism (more information on that below) or extracted via a parser. They are considered to be the most effective way to conduct searches on the DataSet platform. Attributes are also extremely versatile -- once created, they can be used in searches, graphs, dashboards, PowerQueries, and alerts.
The examples provided here originated from customer inquiries and our experience with the DataSet product.
Searching with Attributes
Whenever possible, it's recommended to use attributes to search for specific log events, as this significantly improves search speed and performance.
For example, given the complete log event:
Search the message attribute
Search parameters which do not specify an attribute are applied to the reserved
message field (which contains the entire log event). The following would search the
message field for a value of "75c0509a":
Search a specific attribute
A more effective search would target the
spanId attribute (screenshot from "Inspect Log Line" dialog):
spanId contains "75c0509a"
This is provided you know the attribute contains the value you're searching for.
Locate an attribute
If you don't know which attribute contains the value you're looking for, you could review all attributes which contain the value you're searching for:
* contains "75c0509a"
From here, locate the attribute(s) in the facet bar of the Search page. Since you've identified the attribute(s), specify it by name within your query.
Adding an exclamation point (!) as a prefix will negate the search query:
!(spanId contains "75c0509a")
Inspect Log Line Dialog
You can quickly review the attributes that are associated with a log event by clicking the event in the "Search" page. From there, review the "Inspect Log Line" dialog:
1. Complete Log Event (
The complete log event is stored within the reserved
message field and is displayed at the top of the dialog. This maintains the original log for analysis / review and is independent of parsing routines.
Attributes displayed in this field are either assigned by the log ingestion mechanism or extracted from the log event by a parser.
A quick way to identify assigned attributes is to check the
- If no
messagefield is present, or
- If the attribute value is not present within the
The attributes below were assigned as no message field is present. The most common form of logs with assigned attributes is DataSet metalog data:
Otherwise, assigned attributes are explicitly configured by the customer when using the DataSet
addEvents API, or our Logstash / Fluentd plugins. They do not occur without manual intervention. It is currently not possible to extract values from assigned attributes with a parser. However, you can use a PowerQuery with the
parse from function to analyze these fields.
The most common way that attributes are generated for log events. In order for an attribute to be parsed, its value must exist in the
message field. DataSet parsers are comprised of regular expressions that extract values from log events. Please note that parsers do not support iteration (for, while), conditional logic (if-then-else, case), or mathematical operations. Some logical and mathematical operations can be performed by PowerQueries, however, this won't impact how log data is extracted by a parser.
3. "Server Fields" (server attributes)
Server attributes are assigned by the upload mechanism that processed the log event (ex. DataSet API, Agent, or integration (S3, CloudTrail, Okta, Logstash, etc.)). These fields are present for each log event and contain attributes which are related to the log's origin.
Server attributes are typically used for DataSet operations like log scrubbing, templated alerts (see
byHosts section), and defining Cost Management categories since they're consistently available. Server fields like
serverIP are commonly used to designate log sources.
Please sign in to leave a comment.