Using annotations with the K8s Agent

Introduction

This article pertains to the K8s Agent. As mentioned in our K8s Agent documentation, the Agent captures stdout / stderr output from pods within the same node and uploads them to DataSet. For ease of use within the K8s environment, a number of common Agent functions can be configured via annotations. This article explains how to assign a parser to pod logs and include / exclude logs by using annotations. More information can also be found here.

Examples

Note: Substitute pod-name-xyz123 with the full name of your application pod

Set the parser via the CLI

The "docker" parser is the default parser for logs which originate from pods where a parser is not explicitly set. The "docker" parser works well if your logs are in JSON format and no modifications need to be made. We generally recommend setting custom parsers if:

• Your logs are an unstructured format (ex. CSV)
• Specific values will be extracted from log events
• Key-value pair extraction will be used
• Logs consist of different formats. For example, you have HTTP access logs, application logs, and firewall logs. In this case, 3 parsers would be created, one to accommodate each log format.

Using custom parsers improves the performance, maintenance, and scalability of your DataSet configuration. Don't use a single parser to process multiple log formats!

Set the parser to "logGenTest" with an annotation

kubectl annotate pod pod-name-xyz123 --overwrite log.config.scalyr.com/attributes.parser=logGenTest

Set the parser via the deployment metadata

The parser attribute can also be set via the application pod's deployment configuration

List deployments

kubectl get deployments --show-labels

Add the annotation to the template section of the deployment metadata, then apply the updates:

application-pod.yaml

...
spec:
  ...
  template:
    metadata:
      annotations:
        log.config.scalyr.com/attributes.parser: logGenTest
        ...
      ...
    ...
  ...
...

Confirm that the parser attribute was correctly set

kubectl get pods pod-name-xyz123 -o custom-columns=ANNOTATIONS:.metadata.annotations

This should return output similar to

map[... log.config.scalyr.com/attributes.parser:logGenTest ...]

Edit or create the "logGenTest" parser within the DataSet UI by clicking the User Menu -> "Manage Logs" -> "Parsers"

Any changes made to the parser will be applied the associated log events as they are ingested

Unset the parser attribute

kubectl annotate pod pod-name-xyz123 log.config.scalyr.com/attributes.parser-

This command will unset the parser attribute. The default "docker" parser will be applied to the associated log events.

Only upload logs from specified pods

Logs can be restricted to specified pods in order to reduce chatter and log volume. Similar to configuring the parser, this is achieved by setting annotations.

Add SCALYR_REPORT_CONTAINER_METRICS: "false" to the K8s Agent ConfigMap

...
apiVersion: v1
data:
  SCALYR_REPORT_CONTAINER_METRICS: "false"
  ...
...

Restart the K8s Agent for the change to take effect

kubectl delete pods -n scalyr --field-selector=status.phase=Running

Set the following annotation to include logs from specific pods

kubectl annotate pod pod-name-xyz123 --overwrite log.config.scalyr.com/include=true

or add via the deployment metadata

...
spec:
  ...
  template:
    metadata:
      annotations:
        log.config.scalyr.com/include: "true"
        ...
      ...
    ...
  ...
...

This setting will only upload logs from pods where the include attribute is set.