Aller au contenu

PrometheusRemoteWriteRequestErrors#

This alert is implemented only on NGOT and has the role to monitor Prometheus Remote Write on client clusters.

This alert is fired by destination Prometheus (PF side, service zone type) when source Prometheus (client side) fails to send metrics via Remote Write protocol.

The alert is based on the prometheus_http_requests_total metric which allows to know the return code (code label in alert) of the request on the Prometheus destination.

The criticity of this alert depends on the http error code:

  • critical: for 5xx http error codes
  • warning: for 4xx http error codes (except http code 400 because the problem is on client side).

Troubleshooting hints#

  • Check the logs of destination Prometheus (PF side, service zone type). It is probably the one not working well. 

    kubectl get logs <destination_prometheus_pod> -n <destination_prometheus_ns>

  • Check the load of destination Prometheus (CPU/Mem...)

Known cases#

HTTP response is 404#

  • Check that the endpoint remote write receiver is enabled: 
    kubectl get prometheus -n <destination_prometheus_ns> # get the name of destination prometheus object (PF side, service zone type) 
kubectl get prometheus <destination_prometheus_object> -n <destination_prometheus_ns> -o json | jq .spec.enableFeatures # `remote-write-receiver` must be visible

Prometheus Remote Write known cases#

Some solutions are described here.