Minimizing Duplicate Log Entries when Agent Pod / Container Restarted

Introduction

As of June, 2020, version 2.1.7+ of the Kubernetes and Docker Scalyr Agent incorporate additional functionality to minimize duplicate logs on restart. This was achieved by storing the Agent's checkpoints on a volume mount so that they persist. The latest versions of the Agent's DaemonSet configuration (scalyr-agent-2.yaml) contain the updates below, so no further action is needed if you're running a recent version of the Agent.

The steps below are maintained for legacy purposes.

Configuration

1. Update to v2.1.7 (or later) of the Scalyr Agent
2. Review the Scalyr Agent DaemonSet configuration that is presently implemented on your cluster:
   1. You can run kubectl -n scalyr edit daemonsets scalyr-agent-2 to review and make changes to its contents. Please add the following sections if they are not already implemented:
      
      1. Confirm that you have a volumeMounts section as follows:

         volumeMounts:
         ... 
          - mountPath: /var/lib/scalyr-agent-2
            name: checkpoints

      2. Next, confirm that you have the following volumes section:

         volumes:
         ...
         - hostPath:
             path: /tmp/scalyr-agent-2
             type: DirectoryOrCreate
             name: checkpoints

   2. You can also apply the latest version of the Scalyr Agent DaemonSet configuration file if additional changes are not required
3. Redeploy the Scalyr Agent DaemonSet

The Scalyr Agent will use this volume mount to store its checkpoints. In the event that it is restarted, it will resume logging where it left off.