Skip to main content

K8s - Scalyr Agent Installation and Configuration

John Tan
  • Updated

Introduction

DevOps and SREs will be able to install and configure the Scalyr K8s Agent. This guide outlines the following topics of interest by citing the relevant sections from our official documentation and providing additional details (where appropriate).

Configure Metrics

Metrics and Events

By default, the Scalyr Agent gathers Kubernetes cluster and container metrics and Kubernetes events. This section explains how to configure them accordingly.

To change this, you can simply modify the following environment aware variables in a ConfigMap. See the "Modifying the ConfigMap" section below for step-by-step instructions.

Turn off Cluster Metrics

SCALYR_K8S_EVENTS_DISABLE: "true"

Turn off Container Metrics

SCALYR_REPORT_CONTAINER_METRICS: "false"

Turn off Events

SCALYR_K8S_EVENTS_DISABLE: "true"

Environment Aware Variables

The authoritative list of the Scalyr Agent's environment aware variables can be found here

Prometheus Metrics

We have a working Prometheus integration, which is presently in the alpha stage of development. Reach out to support@dataset.com if you are interested in beta testing.

Modifying the ConfigMap

The following commands detail how to manage the Scalyr Agent's ConfigMap

View ConfigMap

kubectl get configmap scalyr-config -n scalyr -o yaml

Export ConfigMap to file

kubectl get configmap scalyr-config -n scalyr -o yaml > /tmp/scalyr-agent-2-configmap.yaml

Edit

vim /tmp/scalyr-agent-2-configmap.yaml

Save

kubectl create configmap scalyr-config --from-file=/tmp/scalyr-agent-2-configmap.yaml -n scalyr -o yaml --dry-run=client | kubectl apply -f -

 

Redeploy agent

kubectl delete -f scalyr-agent-2-daemonset.yaml
Then
kubectl create -f scalyr-agent-2-daemonset.yaml

 

Configuring Parsers

This tutorial will help you configure and name parsers in Kubernetes. If you would like to learn about parsers and how to build them, visit our parsing video series or check out our parser documentation.

By default, deployment logs being read by the agent will default to the "docker" parser. It is important to change this and separate your parsers logically. This will simplify the eventual configuration when we define custom parsers for different log patterns.

For example, if Application A is emitting JSON logs, Application B is emitting access logs, and Application C is emitting error logs, we recommend implmenting a "JSON_applicationA" parser and an "accessLog_applicationB" parser (and so on). Some clusters have tens or even hundreds of different services. As we scale and begin to add more microservices, using one parser for all logs will become unmanageable quickly and increase in complexity due to the accommodation of various patterns.

Introduction

In this exercise, we will take our existing cluster, assign the frontend a parser, and configure this parser in DataSet.

Configuration

List deployments

kubectl get deployments

Next, find the "nginx-deployment" deployment (or any other deployment in your environment). Apply a pod annotation to this deployment. This can be done in one of two ways:

CLI

kubectl annotate pod <pod-name> --overwrite log.config.scalyr.com/attributes.parser=custom-parser

YAML

metadata:annotations:        
     "log.config.scalyr.com/attributes.parser": "custom-parser"

For this example, we want this to apply dynamically to all pods (not just a single pod), so we will change this in the manifest.

Update the manifest for "nginx-deployment"

kubectl edit deployment nginx-deployment

Add the following to the pod annotations section (in bold)

log.config.scalyr.com/attributes.parser: "custom-parser"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      annotations:
        log.config.scalyr.com/attributes.parser: "nginx-whee"
      labels:
        app: nginx
  spec:
    containers:
    - name: nginx
      image: nginx:1.14.2
      ports:
      - containerPort: 80
View your cluster's logs in DataSet and confirm that the parser is set correctly (select a log line and click the "Inspect Fields" button):
mceclip0.png

Setting Parsers with Annotations

Please see our official documentation and scroll down to the "Setting Parsers" section.

 

Attributes

Attributes can be attached in 3 ways:

  1. Agent Attributes - via annotations, scroll down to "Attaching Attributes" section
  2. Pod Labels - Displayed as log attributes within DataSet
  3. Deployment Labels - Displayed as server attributes and metadata can be broken down within the Log Volume dashboard (click "Dashboards" -> "Log Volume")

Related articles

Comments

0 comments

Please sign in to leave a comment.