ThanosReceiveHighHashringFileRefreshFailures

This alerts is triggered when Thanos Receive is failing to refresh hashring file.

Troubleshooting Hints

Note

stagingproduction

• NAMESPACE=monitoring-stack-corp-obs-corp-stg

• NAMESPACE=monitoring-stack-corp-obs-corp-prd

• Check the hashring presence and syntax:

  kubectl -n ${NAMESPACE} get cm thanos-receive -o yaml
  

  The expected result is (extract):

  hashrings.json: |-
    [
      {
        "endpoints": [
          "thanos-receive-0.thanos-receive-headless.monitoring-stack-corp-obs-corp-XXX.svc.cluster.local:10901",
          "thanos-receive-1.thanos-receive-headless.monitoring-stack-corp-obs-corp-XXX.svc.cluster.local:10901",
  ...
          "thanos-receive-NN.thanos-receive-headless.monitoring-stack-corp-obs-corp-XXX.svc.cluster.local:10901"
        ]
      }
    ]
  

  The hashring is static and is generated at the deployment of the helm chart. It should not be edited by hand.

  If the hashring needs to be redeployed, follow the documentation to redeploy the helm chart.

• Check the logs of Thanos Receive. They will probably give some information about the problem.

  kubectl get pods -n ${NAMESPACE}
  kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=thanos