PrometheusHighQueryLoad

Prometheus can handle multiple queries simultaneously. There is a maximum value. When queries use more than 80% of this maximum, the alert fires.

Queries come from :

• PrometheusRules (recording and alerting rules)
• user queries.

However Grafana connects to Thanos-querier that connects to the Thanos sidecar. So user queries from Grafana do not impact Prometheus.

Note

The default value is 20 for the maximum of concurrency queries.

Troubleshooting hints

• Check the logs
• If the solution is to increase the number of max concurrency queries, edit envs-ng/kube-prometheus-stack.cue and add :

  prometheus: {
      ...
      prometheusSpec: {
         ...
         query: maxConcurrency: <new limit>
      }
      ...
  }
  

Note

15/09/2022 : this is a new alert and has never fired yet. When it fires, please update this documentation.

Advanced debugging

• Enable QueryLog in the Prometheus object (kubectl edit prometheus caascad-prometheus for example). In the spec section, set QueryLog: "/dev/stdout". Then check the logs if there were some unusual queries.

Warning

Don't forget to remove the QueryLog line ASAP as it generates lots of logs.