Does Kubernetes Agent Ingest Logs Written to Files? – DataSet Customer Portal

Scalyr Kubernetes agent currently only supports reading logs from stdout, so the logs written to files aren't ingested by the Scalyr agent. The recommended solution is to modify the application to log to stdout rather than a file, so the Scalyr agent can pick up those logs automatically with k8s labels (ie. pod name, container id, k8s-node, etc.) attached.

Alternatively, you could also run the application as a sidecar that uses another container to read the file and output them to stdout. Here is a quick example to demonstrate how it works:

The application (log.sh) logs date output

#/bin/sh

# log.sh - run an infinite loop,
# and print the container hostname as well as the time every second.

i=0; while true; do echo "[$(uname -n)] $(date)"; i=$((i+1)); sleep 1; done

Create a DockerFile to redirect the output to a file (ie. /var/log/dummy.log)

FROM busybox:latest
ADD log.sh /
RUN mkdir /var/log/
VOLUME /var/log
RUN ["chmod", "+x", "/log.sh"]
CMD /log.sh > /var/log/dummy.log

Build the image and publish it to docker hub (my image name is weili808/dummylogger:v1)
Create a Pod YAML file for a multi-container Pod with a shared volume (i.e logger). One container (i.e 1st) writes logs to the shared directory(i.e. /var/log) and the other container (i.e 2nd) reads from the shared directory and prints to stdout.
Assigning a custom parser "dummy-parser" to the 2nd container so the logs ingested to DataSet can be parsed properly (the syntax log.config.scalyr.com/<container name>.attributes.parser can be reused if more than 1 log file needs to be parsed and ingested to DataSet ).

apiVersion: v1
kind: Pod
metadata:
  name: mc1
  annotations:
      "log.config.scalyr.com/2nd.attributes.parser": "dummy-parser"
spec:
  volumes:
  - name: logger
    emptyDir: {}
  containers:
  - name: 1st
    image: weili808/dummylogger:v1
    volumeMounts:
    - name: logger
      mountPath: /var/log/
  - name: 2nd
    image: debian
    volumeMounts:
    - name: logger
      mountPath: /var/log
    command: ["/bin/sh", "-c"]
    args:
      - while true; do
          tail -n+1 -f  /var/log/*.log;
        done

Creating this pod and you'll find logs written to /var/log/dummy.log in the first container is collected by the Scalyr agent and the messages are available in the app.

Screen_Shot_2021-03-29_at_2.54.58_PM.png

The messages are ingested from the 2nd container that reads the logs from the file and output to stdout.

weilil@C02F348AMD6R k8s % kubectl describe pod mc1
Name: mc1
Namespace: default
Priority: 0
Node: ip-172-31-47-33.us-west-2.compute.internal/172.31.47.33
Start Time: Mon, 29 Mar 2021 14:48:08 -0700
Labels: <none>
Annotations: kubernetes.io/psp: eks.privileged
             log.config.scalyr.com/2nd.attributes.parser: dummy-parser
Status: Running
IP: 172.31.45.23
IPs:
    IP: 172.31.45.23
Containers:
  1st:
    Container ID: docker://d2138f6da1894e7362086799f64eaf3bd3e3613ce476f0995159303d816fe753
    Image: weili808/dummylogger:v1
    Image ID: docker-pullable://weili808/dummylogger@sha256:2e7041bc2e5747600cdcc38e36e6bcfa3951ecbb79a8754afda48dea32ded28a
    Port: <none>
    Host Port: <none>
    State: Running
        Started: Mon, 29 Mar 2021 14:48:09 -0700
    Ready: True
    Restart Count: 0
    Environment: <none>
    Mounts:
      /var/log/ from logger (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pzlmv (ro)
  2nd:
    Container ID: docker://9c7d1b0d58d5d7f68d76bfe04666a1773956ca1f5468968c7b90b689dbfdac70
    Image: debian
    Image ID: docker-pullable://debian@sha256:ed28974b37a2ee07bc8d43249c7396eee05a210d92aa784871e3bed715671d4f
    Port: <none>
    Host Port: <none>
    Command:
      /bin/sh
      -c
    Args:
      while true; do tail -n+1 -f /var/log/*.log; done
    State: Running
      Started: Mon, 29 Mar 2021 14:48:11 -0700
    Ready: True
    Restart Count: 0
    Environment: <none>
    Mounts:
      /var/log from logger (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pzlmv (ro)

Related articles