Aller au contenu

LokiRemoteWriteBehind#

When Loki Ruler sends data to Prometheus via Remote Write protocol, if the destination has a problem, the source will retry to send the data.

This alert show that those data are accumulating on Loki Ruler.

Warning

Prometheus cannot receive metrics older than one hour. Any data older than 1 hour will be definitively lost if they could not be sent.

If label ngot_contract has other value than obs-corp-prd, it means there is a customer impact. In this case, the client needs to be informed about the data loss.

Note

It's normal to see fake in the labels or description of the alert: this corresponds to the default name of the tenant in Loki.

Troubleshooting Hints#

Tip

In most cases, there is no problem on Loki Ruler. Prefer investigating on Prometheus side first.

  • Check the logs of Loki and Prometheus

    • Prometheus (destination)

      Tip

      Below, <monitoring_namespace> can be:

      • monitoring-stack-client-<ngot_contract>
      • monitoring-stack-<ngot_contract>

      Note

      The ngot_contract label can be obtained from the alert.

      For each Prometheus where there are errors check the Prometheus logs: 

      kubectl logs statefulset/prometheus-<ngot_contract>-prometheus -n <monitoring_namespace> -c prometheus

    • Loki Ruler (source) can give clues why it cannot send the data 

      kubectl logs statefulset/loki-ruler -n <logging_namespace>  # Notice the `namespace` label in the alert

  • Check the URL of the Remote Write configuration in Loki configuration :

    • in the url label of the alert ;
    • or in the remote_write section of loki configmap.

  • Test that URL locally with GET method

    Tip

    There is no authentication : a 405 is expected.

    There is an authentication : without a username/password a 401 is expected.

  • Check the load of Prometheus (CPU/Mem...).