Missing K8s Agent Metrics or InsecureRequestWarning Errors

Introduction

When upgrading from a version of the Scalyr Agent that is < v2.1.3, you may notice that metrics are no longer available. This may be accompanied by an "InsecureRequestWarning" or "CERTIFICATE_VERIFY_FAILED" error in the the Scalyr Agent logs. For example:

$ kubectl logs scalyr-agent-2-XXXXX | tail 
...
InsecureRequestWarning) 
/usr/share/scalyr-agent-2/py/scalyr_agent/third_party/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
...

Or,

$ kubectl logs scalyr-agent-2-XXXXX | tail 
...
WARNING [monitor:kubernetes_monitor()] [kubernetes_monitor.py:3909] Error connecting to kubelet API: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727). No Kubernetes stats will be available
...

Solution

As of v2.1.3 of the Scalyr Agent, the K8s monitor will verify https connections made to the local Kubelet. You can either

• Set SCALYR_K8S_KUBELET_CA_CERT in your scalyr-config ConfigMap to the path of the relevant certificate (be sure that path is properly mapped into the Scalyr Agent container's filesytem), or

• Disable verification by setting SCALYR_K8S_VERIFY_KUBELET_QUERIES in your scalyr-config configmap to false