Aller au contenu

LokiRemoteStorageFailures#

Some samples (aka metrics) failed to be sent via Remote Write protocol.

Because we are loosing data (the failed samples) this alert is critical.

Warning

If this alert is triggered, metrics are lost.

If label ngot_contract has other value than obs-corp-prd, it means there is a customer impact. In this case, the client needs to be informed about the data loss.

Note

It's normal to see fake in the labels or description of the alert: this corresponds to the default name of the tenant in Loki.

Troubleshooting Hints#

In most cases, this alert comes with errors HTTP 4xx : the source Loki failed to connect to the destination Prometheus.

  • Check the logs on Prometheus destination

    Tip

    Below, <monitoring_namespace> can be:

    monitoring-stack-client-<ngot_contract>

    monitoring-stack-<ngot_contract>

    Note

    The ngot_contract label can be obtained from the alert.

    For each Prometheus where there are errors check the Prometheus logs: 

    kubectl logs statefulset/prometheus-<ngot_contract>-prometheus -n <monitoring_namespace> -c prometheus

  • Check Remote Write URL in Loki configuration :

    • in the url label of the alert ;
    • or in the remote_write section of loki configmap.

  • Test that URL locally with GET method :

    Tip

    • There is no authentication : a 405 is expected
    • There is an authentication : without a username/password a 401 is expected

  • For Corp zones, check that Loki has the correct username/password to connect to the URL :

    Vault secret path containing username/password : 

    secret/zones/fe/<service_zone>/prometheus-ingress
    Example for obs-pf.

    Secret in Loki Ruler pod : 

    kubectl exec -it statefulset/loki-ruler -n loki-<ngot_contract> -- cat /etc/secrets/secret-ruler-prometheus

  • Check the load of Prometheus destination (CPU/Mem...).