PrometheusContainerTooManyOpenSockets

This alert indicates that prometheus container has a high number of open connections.

Prometheus client

Explanations

The number of open connexion on destination Prometheus depends on the number of client clusters that send metrics to it.

The threshold for this alert has been evaluated for 30 client clusters. Make sure that there are less than 30 clusters sending metrics to the prometheus for which this alert is firing before starting the investigation on this alert. If the number is higher than 30 client cluster, than the alert threshold needs to be increased.

Troubleshooting Hints

Some solutions are described here.

Prometheus central (mon3/mon4)

Explanations

Prometheus central (mon3/mon4) open a lot of connections:

• for clients of "monitoring stack": 3 (with Blackbox-Exporter, test remote-write-0, remote-write-1 and grafana)
• for clients of "loki": 1
• for clients of "mutualized Grafana": 1 per datasource
• for clusters: 2 per cluster (monitoring of monitoring)

On 2025-03-10, with 132 clients of "monitoring stack", 19 clients of "loki", 24 datasources for all "mutualized Grafana" and 77 clusters, plus/minus some minor variations, the number of connections is 600.

Troubleshooting Hints

1. Theory: count the theorical number of connections: 3*<nb clients> + <nb loki> + <nb datasources of mutualized grafana> + 2*<nb clusters> as described above. However, because each Grafana has one datasource, we prefer this formula:

   2 * <nb clients> + <nb_loki> + <nb datasources> + 2 * <nb clusters>
   

   where these numbers can be got like this:

   # Nb clients
   sd get zones | jq -r '.[] | select(.product == "ngot") | select(.metadata.line == "prod") |.name'|grep svc-monitoring-stack-client |wc -l
   # Do not forget to double this number!
   
   # Nb loki
   sd get zones | jq -r '.[] | select(.product == "ngot") | select(.metadata.line == "prod") |.name'|grep svc-loki |wc -l
   
   # Nb datasources
   cd envs-ng/gen/ngot_zones_static
   cue export --out json |jq -r '.zones | to_entries | .[]| select(.value.metadata.line == "prod") | select(.value.subtype == "grafana-client") | .value.parameters.thanos_query_connected_services | to_entries |.[]|.value.name' | wc -l
   
   # Nb clusters
   sd get zones | jq -r '.[] | select(.metadata.line == "prod") | select(.type == "cluster")|.name' |wc -l
   # Do not forget to double this number!
   

2. Reality: in Grafana, with the following expression, get the real number of connections:

   container_sockets{container="prometheus",namespace="monitoring-stack-corp-obs-corp-prd"}
   

Then compare the theorical value and the real one. They should match (not exactly, but with a difference < 5)

Theory vs Reality: what to do?

• Case 1 : theory and reality match. The problem is known: the threshold is too small. However changing the threshold would change it also for Prometheus Clients. You need to escalate the issue to Monitoring Team.

  • Monitoring Team may try to make the difference between incoming and outgoing connections and set specific thresholds
  • Monitoring Team may try to set a different threshold for Prometheus Central and Prometheus Clients

• Case 2 : reality is far bigger than theory. You need more investigation about the connections. This is probably not a problem with Monitoring.

  • get into the pod and check with netstat -tunelap or netstat -tuelap

  • get on the node and investigate with sudo lsof -p <pid> where the PID of Prometheus can be found with ps -ef | grep prometheus; ignore prometheus-config-reloader; then identify your Prometheus thanks to the --web.external-url argument of the output.

    Note

    In the node, lsof works well but netstat will not give good results because of cgroups.

  • as usual, check the Prometheus logs

  • as usual, check the node logs (dmesg, systemctl --failed, journalctl..., /var/log/..., ...)