ThanosReceiveNoUpload

This alert is trigerred when Thanos Receiver has not uploaded latest data to object storage.

Troubleshooting Hints

Note

ProductionStaging

• NAMESPACE=monitoring-stack-corp-obs-corp-prd

• NAMESPACE=monitoring-stack-corp-obs-corp-stg

• Notice if one or many Thanos Receivers are impacted.

  • if one is impacted, the problem probably happens on that receiver or to the node it is running on,
  • if many are impacted, the problem may be either on the cloud provider or on the S3 backend.

• Check when the problem started. In Grafana, this expression shows the Thanos Receivers that have not uploaded any data for more than 3 hours.

  increase(thanos_shipper_uploads_total{job=~".*thanos-receive.*"}[3h]) == 0
  

  Enlarge the visualisation time window to see 24h or more: you should see when the broken pods have stopped uploading for the first time. We consider this as the beginning of the issue.

• Check the pods state.

  kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/component=receive
  

  • Check the state of the pod mentionned in the alert (label pod). Is it in the expected state?

  • Check if it is one of the last pods? If yes, this may be a failed scale-up. Check the pod number definition (where svc-monitoring-stack-corp-xxx-n is from label ngot_service in the alert):

    • Prod: svc-monitoring-stack-corp-prd-1 or svc-monitoring-stack-corp-prd-2,
    • Staging: svc-monitoring-stack-corp-stg-1 or svc-monitoring-stack-corp-stg-2.

  Check the history (git blame) of the replicaCount (in section receive) line. It may have been modified recently. It may also show inconsistency in case of a scale-down operation.

• Check the pods "distributor" state.

  kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/component=receive-distributor
  

  • Check the state of the pod mentionned in the alert (label pod). Is it in the expected state?

• Check the logs of an impacted Thanos Receiver

  kubectl logs -n ${NAMESPACE} thanos-receive-xx
  

  or in Grafana, datasource Loki (replace kub-53 with kub-34 if it is on "mon4"):

  {cluster="kub-53", container="receive", pod="thanos-receive-xx"}
  

  The logs should explain the problem.

  In some cases, restarting the pod (kubectl delete pod) may be sufficient.

• Check the alerts history. Some problem may have happened on Thanos Receive (including Thanos Distributors) or on the network (including the Ingresses and loadbalancers). It may have been repaired yet but if something was broken for too long, there is no data to upload and here is the alert ThanosReceiveNoUpload.