PodRestarts

Fires when a pod restarts.

Troubleshooting hints

• investigate why the pod restarted (check the logs, check the kubectl decribe pod xxx Last State status, rason and exit code...).
• If the pod does not exist any more, in Grafana, choose Loki or Loki-client source and check logs :
  • {job="kube-system/eventrouter"}|="<pod name>"
  • {job="kubelet"}|="<pod name>"
  • {namespace="<namespace of the pod>"}|="<pod name>" WARNING : this show the pod logs. DO NOT DO IT IN A CLIENT POD !
  • in Grafana, if you zoom on the alert date, zoom at least 5 minutes before.

Note : the alert is raised for 30 minutes after the pod restarted. But it fires only after 5 minutes. One may think that we can fire the alert before 5 minutes. However, those 5 minutes let people silence the alert if they are working on that pod.

Known case

A known case is when liveness/readiness probe failures are causing pod restart.

In order to decide if the incident matches this particular case, perform the following checks:

• check for liveness/readiness probe failures: kubectl describe <prometheus_pod> ...
• in Grafana (Explore menu) check the following metric: container_sockets{container="prometheus"}. It has been noticed that when the number of open connections reaches the maximum allowed limit, it is causing liveness/readiness to fail.

In this case, in order to reduce the impact (lost of metrics), you can temporary perform one of the following actions:

• remove the liveness and readiness probe: this action will immediatelly stop the cause that is producing the pod restart
• increase the allowed number of open connections on prometheus container: this action can avoid the liveness/readiness probe failures

Important

If you decide to increase the number of allowed open connections on prometheus container, but the liveness/readiness are still failling, you will probably want to disable the liveness/readiness probe.

Remove Prometheus liveness/readiness probe

Important

Prometheus is deployed with Prometheus-Operator so you need to modify the prometheus object. Modifying prometheus pod is useless as prometheus-operator will detect the configuration diff and will set back the default values (your change will not be taken into consideration).

When Prometheus is deployed, its liveness and readiness probes are configured for a normal usage.

If you decide to temporarily disable the liveness/readiness probe, follow the next steps:

• get the prometheus object name: kubectl get prometheus -n <prometheus_namespace>
• edit the prometheus object: kubectl edit prometheus <prometheus_object_name> -n <prometheus_namespace>
• disable the liveness/readiness probe by adding the following configuration in prometheus object. Attention: when editing the prometheus object, the liveness/readiness configuration is not visible when it has the default value set; simply adding the following code, will modify the default configuration:

spec:
  containers:
  - livenessProbe:
      failureThreshold: 1000000 # to deactivate the probe, you can just put a very high value
    name: prometheus
    readinessProbe:
      failureThreshold: 1000000 # to deactivate the probe, you can just put a very high value

Important

At the end of the incident,don't forget to rollback the workaround by removing the added liveness/readiness configuration.

Increase Prometheus container number of connections

If Grafana indicates that prometheus container has reached the maximum allowed number of open connections (container_sockets{container="prometheus"} metric), you can try to increase this limit. However, this solution does not guarantee the success of liveness/readiness probes.

If you decide to temporarily increase maximum limit of open connections, follow the next steps:

• get the prometheus object name: kubectl get prometheus -n <prometheus_namespace>
• edit the prometheus object: kubectl edit prometheus <prometheus_object_name> -n <prometheus_namespace>
• increase the maximum allowed number of open connections by adding the following configuration in prometheus object. Attention: when editing the prometheus object, the --web.max-connections configuration is not visible when it has the default value set; simply adding the following code, will modify the default configuration:

spec:
  web:
    maxConnections: 2048

• wait and verify the resolution of the incident by checking container_sockets{container="prometheus"} in Grafana. If everything comes back to normal you can rollback the changes you made to stabilise Prometheus.

Warning

At the end of the incident, don't forget to rollback the workaround by removing the added --web.max-connections configuration.