Aller au contenu

Troubleshooting Loki#

LokiRequest400Errors#

First, you need to know which job it is, such as querier or distributor. This information is provided in Karma with job label.

The issue with job Querier#

Request latencies are also observed (error LokiRequestLatency)#

We are in this case when

  • the alert label job is querier
  • either another alert LokiRequestLatency is happening or when some latencies are noticed in Grafana dashboard Loki/Reads.

Troubleshooting hints

  • In Grafana, check the dashboard Loki/Reads. Check the latencies graphs : they show the severity of the problem.
  • Querier logs may give some additional information. LogQL query : {container="querier", namespace="logging-client"} | logfmt | level="error" (change namespace for logging if needed).

Warning

20/04/2022 : the Loki dashboards are broken : the metrics are aggregated for all Loki namespaces. See PF-111.

Explanation

The user is querying on too many or too big indexes.

Solutions

  • If possible, reduce the cardinality of the logs. However, this is only a best practice and cannot be considered as the ultimate solution.
  • Reduce the time windows of the query. However, if the user is our client, this can only be an advice.
  • There is no known good solution for this scalability problem (20/04/2022).

Tip

ESAGSO was known to have a high cardinality on logs and metrics due to their deployed cronjobs. This could be observed in logging-client namespace.

The issue with job Distributor#

Method with metric loki_discarded_samples_total#

Find the reason#

Check Grafana using the following query (replace XX with the cluster ID):

sum by (cluster, namespace, reason, job) (increase(loki_discarded_samples_total{reason!='', cluster="kub-XX"}[5m]) > 0)

Find the label reason and its value.

The reasons are explained in the Loki documentation.

Some cases are well known and documented below:

  • too_far_in_future
  • greater_than_max_sample_age
  • rate_limited
Reason too_far_in_future or greater_than_max_sample_age#

In the case of {reason="too_far_in_future} or {reason="greater_than_max_sample_age} in Grafana

  • The alert label job is distributor.
  • The reason for the issue is too_far_in_future or greater_than_max_sample_age in Grafana.

Explanation :

If a sample's timestamp is later than the current time (with a grace period) or older than the reject_old_samples_max_age parameter, Loki will reject the samples.

Solution:

You need to send an email to the customer, adapting the content as necessary.

Bonjour,

Nous avons constaté des erreurs 400 sur votre instance Loki, dues à des logs ayant un timestamp dans le futur ('too_far_in_future').

Loki rejette les logs dont le timestamp est dans le futur (avec une petite marge autorisée).

Nous vous prions de vérifier et de synchroniser les horloges de vos machines.

Merci de bien vouloir nous informer dès que le problème est résolu. N'hésitez pas à nous contacter pour toute information complémentaire.

Cordialement,
NGOT Support Team

Bonjour,

Nous avons constaté des erreurs 400 sur votre instance Loki, dues à des logs ayant un timestamp trop ancien ('greater_than_max_sample_age').

Loki rejette les logs dont le timestamp est trop ancien, c'est-à-dire que ceux-ci dépassent la limite d'âge maximale configurée.

Nous vous prions de vérifier et de synchroniser les horloges de vos machines.

Merci de bien vouloir nous informer dès que le problème est résolu. N'hésitez pas à nous contacter pour toute information complémentaire.

Cordialement,
NGOT Support Team
Reason rate_limited#

In the case of {reason="rate_limited"} in Grafana

  • The alert label job is distributor.
  • The reason for the issue is {reason="rate_limited"} in Grafana.

Explanation :

If the sample ingestion limit is exceeded, Loki will reject the samples.

Solution :

Increase the value of the following parameters limits_config in Loki configuration. 

ingestion_rate_mb:       1024
ingestion_burst_size_mb: 1024

There are two ways to do this :

  • increase limits for all zones, in loki.cue
  • increase the limit for a particular zone, in ngot-zones, see models.cue

Sample for 'logs' has timestamp too old#

Note

This case is similar as too_far_in_future or greater_than_max_sample_age.

The method of detection differs and can be applied only on our clusters with full access to Promtail.

Case if Promtail returns a 400 Bad Request error.

  • the alert label job is distributor
  • some logs say server returned HTTP status 400 Bad Request (400): sample for 'logs' has timestamp too old:....

Troubleshooting hints

See the Promtail logs on grafana. LogQL query : {app="promtail"}.

Errors already encountered :

  • server returned HTTP status 400 Bad Request (400): sample for 'logs' has timestamp too old:... due to a timestamp parsing problem. See MVPMON-167.

  • 'entry out of order' for stream:. For one stream, a log is received with a timestamp older than the most recent log received.

Explanation :

Some logs are not sent in loki due to their timestamp too old.

Solutions:

You can fix it by adding labels to get different streams.

See current best practices.

WARNING: Despite of the solution exposed above, we added label for splitting stream, but in the case of kubelets it's not enough because for a node, kubelets logs are not well ordered.

In this particular case, those errors must be ignored.

LokiRequest500Errors#

ingester ready is experiencing ...% errors.#

If we can see in the logs of the ingesters with an ingester pod id that no longer exists:

Not ready: ingester ingester-xxxx past heartbeat timeout.

This error can happen when ingesting pods end abruptly.

Ingesters keep data in memory before flushering it in the storage backend: S3.

Before ending, an ingester does several actions: - puts itself in a leaving state - tries to transfer its data to the new ingester which is in a pending state - otherwise, writes its data in the storage backend - deleted from the ring stored in consul

If an ingester stops suddenly, as with an OOM kill, all of these actions are not performed.

Query to see all the ingesters pending (new ingesters awaiting the data of the old) on Grafana : cortex_ring_members{name="ingester",namespace=${NAMESPACE},state="PENDING"}.

Query to see all the ingesters leaving on Grafana : cortex_ring_members{name="ingester",namespace=${NAMESPACE},state="LEAVING"}.

Sometimes the pod is killed so quickly that the ingester doesn't have time to pass by leaving.

Explanations with : ingester handover.

Consequences :

  • Ingester loses its data in memory. The risk is mitigated thanks to the replication which is currently 2.
  • Ingester doesn't remove itself from the ring. And the new ingester remains in non ready.
  • As long as the percentage in the alert message remains below 33%, there is no interruption of service.

Solutions :

To fix that, run loki_remove_ghosts_ingesters_from_ring.sh script in caascad-loki. 

./loki_remove_ghosts_ingesters_from_ring.sh <logging/logging-client>

All ingester pods should become ready.

LokiStreamMemoryUsageHigh#

How to fix#

  1. Set max_global_streams_per_user and loki_stream_memory_usage in ingestion_limits_config in any case
  2. Check the cardinality for possible improvements with the client
  3. Adapt the max_global_streams_per_user and loki_stream_memory_usage if the client could reduce his cardinality

Set parameters in loki client configuration#

The parameter max_global_streams_per_user defines the total number of active streams allowed per user across all ingesters. The parameter loki_stream_memory_usage defines the threshold for a preventive alert.

  1. Retrieve the value of max_global_streams_per_user in Grafana (NGOT): 

    sum(loki_ingester_memory_streams{namespace="loki-client-obs-<client>"}) / 2 / 0.64
    max_global_streams_per_user is the highest value on at least 24h of business hours.

    Note

    • the / 2 is the replication factor, fixed to 2 on NGOT
    • the / 0.64 is the margin of 80% of the preventive alerte (see loki_stream_memory_usage below) and an additional margin of 20% (0.8 * 0.8 = 0.64).

  2. Retrieve the value of loki_stream_memory_usage in Grafana (NGOT):

    max(loki_ingester_memory_streams{namespace="loki-client-obs-<client>"}) / 0.8

    Note

    • the / 0.8 is an additional margin of 20%.

  3. Edit the client config file in zones/ngot_zones/client-xxx.cue.

    Find or add the parameters max_global_streams_per_user and loki_stream_memory_usage with the values above:

    "svc-loki-client-xxx": {
    ...
    parameters: {
        loki: {
            ingestion_limits_config: {
                ...
                max_global_streams_per_user: xxx
            }

            loki_stream_memory_usage: xxx
        }
    }
}

    Deploy the new configurations with the usual worflow:

    • loki
    • prometheus-rules

    Example:

    generate-static-zones-files
(cd contexts/ngot && trackbone apply -z svc-loki-client-<client> -c loki -c prometheus-rules --add-services)

    Note

    This step is a pre-requisite before any possible improvment with the client (see below).

Cardinality#

Warning

Escalate this incident to Monitoring Team.

Monitoring team:

  • we need to investigate and document the "cardinality" section with more real use-cases.
  • we need to explain how to qualify high cardinality.

Study the cardinality#

Note

Install logcli on your machine, you can find the official installation instructions here.

Use logcli with --analyze-labels to identify high-cardinality labels. Consider removing or aggregating them.

Example: 

kubectl -n loki-client-obs-<client> port-forward svc/loki-query-frontend 3100:3100
export LOKI_ADDR=http://localhost:3100
logcli series '{}' --since=1h --analyze-labels

Hint:

Also try this (It may help, or not): 

logcli series '{}' --since=1h --analyze-labels | awk '{if ($3 + 0 != 0) {printf "%-40s %-10s %-10s %f\n", $1, 0+$2, 0+$3, $2 / $3} else {print $1, $2, $3}}'

Contact the client#

If high cardinality is identified, contact the client to reduce it if they can.

If there is nothing to do, the incident can be closed.

When the cardinality has reduced#

When the client has reduced the cardinality, we can adjust the max_global_streams_per_user and loki_stream_memory_usage parameters.

Re-apply the section Set parameters in loki client configuration above.

Note

If the max_global_streams_per_user is calculated as lower than the default value, remove both max_global_streams_per_user and loki_stream_memory_usage parameters from the config file instead of setting it.

The default value is defined upstream. Search for max_global_streams_per_user in docs/loki/configure/#limits_config.