Configure Kubernetes Scalyr Agent to Ship Pod Logs Written to Files

Note: The recommended approach to ingest pod logs written to files is setting up "sidecar" mode for your application pods. Please refer to this KB article for more information. 

The Scalyr K8s agent collects pod logs output via stdout/stderr. This has worked really well for just about all of our users as that's the standard approach to build containerized applications. That being said, we do occasionally receive requests from users who don't have the flexibilities to tune the application using "sidecar" or stdout excessive pod log volume could cause negative impacts on the cluster's performance. To workaround those use cases, the fallback approach is to mount the pod's directory to the node's directory and add another configMap to specify the log paths for the Scalyr agent to read from. 

Here is an example to demonstrate how it works. The application container homelogger writes logs to /mnt/dummy.log, so the first thing to do is to edit the application's manifest to mount the file path to the node.

apiVersion: v1
kind: Pod
metadata:
  name: mc1
spec:
  volumes:
  - name: logger
    hostPath:
      path: /mnt
  containers:
  - name: 1st
    image: weili808/homelogger
    volumeMounts:
    - name: logger
      mountPath: /mnt

Next, a new configMap (scalyr-config-agent-d) is needed for configuring the log path, parser, and any additional attributes associated with the messages. Refer to our documentation for more information.

....  
attributes.json: |
    {
      "logs": [
        {
          "path": "/mnt/dummy.log",
          "parser": "dummy"
        }
      ]
    }
....
kind: ConfigMap
metadata:
  creationTimestamp: "2021-06-25T05:39:44Z"
  name: scalyr-config-agent-d
  namespace: scalyr
...

Finally, editing the agent's Daemonset to mount to the host directory and add the configMap we just created so that the application logs are visible to the Scalyr agent.  

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: scalyr-agent-2
  name: scalyr-agent-2
  namespace: scalyr
....
....
        volumeMounts:
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
...
...
        - mountPath: /mnt
          name: logger
        - name: scalyr-config-agent-d
          mountPath: /etc/scalyr-agent-2/agent.d
...
...

      volumes:
      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: varlibdockercontainers
      - name: scalyr-config-agent-d
        configMap:
            name: scalyr-config-agent-d
      - hostPath:
          path: /mnt
        name: logger
      # comment this section if you do not want to run on the master
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists

 Spinning up the pods and navigating to Scalyr UI to confirm the pod logs are available on Scalyr.

You may notice that the k8s attributes such as "k8s-cluster", "k8s-deployment", etc. are not available. That is the main drawback of mounting the file path directly to the Scalyr agent pod for ingesting messages written to files. Thus, we still strongly recommend our users to stdout pod logs to get the most out of Scalyr K8s support.