PrometheusContainerReachedMaxAllowedOpenSockets#
This alert indicates that prometheus container has reached the maximum allowed open connections in the last 5 minutes.
If this alert fires, the destination Prometheus is not able to accept new connexions in order to receive metrics from source prometheus via Remote Write protocol.
This could imply lost of metrics, which exmplains de criticity of the alert.
Warning
The number of open connexion on destination Prometheus depends on the number of client clusters that send metrics to it. At the end of the incident don't forget to let in place a configuration that works with the number of cluster sending metrics to the destination prometheus (if default value is modified, you need to update the alert threshold too).
Troubleshooting Hints#
- Check the destination Prometheus (prometheus mentioned in the alert) using
kubectl describe ...command (check for liveness/readiness probe failure) - Check the destination Prometheus logs
- In Grafana (Explore menu) check the following metric:
container_sockets{container="prometheus"}and try to identify any unusual behaviour.
Solutions#
Some solutions are described here.