Aller au contenu

ContainerOutOfMemory#

Memory consumption of container is close to memory limit defined in container manifest.

Notes about the metrics used :

  • There are two alerts. They both use the metric container_memory_set_working_bytes.
  • The metric container_memory_set_working_bytes excludes the cache (for example the filesystem cache). We use this metric because kubernetes also uses it for OOM kill the pod when the limit we specified is crossed.
  • container_memory_working_set_byte = container_memory_usage_bytes - total_inactive_file : with this metric we exclude the cache which has not been used for some time and which can be reclaimable. We don't exclude the recently used cache (total_active_file).

Note about the alerts :

  • The first alert checks that the percentage of memory used on the limit is greater than 80%. This alert is the original alert. We may remove it later when we are sure that the second alert is really better.
  • The second alert checks the previous condition and also checks that the available memory is less than 500M.

Troubleshooting#

  • When the alert fires, the message looks like memory usage above 80% and free memory below 500MB. In your case, only one of these conditions is significant. Explanation :

    • on containers with little memory, you may want to be alerted when the memory usage is above 80% and you can safely ignore free memory below 500MB (because your container is defined with little memory).
    • on containers with large memory definition, you do not want to be alerted when memory usage is above 80%. In this case you can safely ignore it. However you have to pay attention why the free memory is below 500 MB.
    • the alert is defined with the 2 conditions and one of them is usually filled in all cases. This is why we define the alert to fire on both conditions and this is also why you can safely ignore one of the 2 conditions.

  • Also check Known Cases below.

Solutions#

  • Solution 1 : check why the container needs so much memory and fix that problem (if it is a problem).
  • Solution 2 : increase the container memory resource limit (except in the known case "Thanos-Receiver" below).

Known Cases#

Thanos-Receiver#

Warning

Never update the memory R/L or volume size of a thanos-receiver-X pod/Statefulset. Follow the documentation and add a new replica instead.

In Grafana, check memory usage with this expression (replace the labels if they are not in the predefined ones below):

sum(container_memory_working_set_bytes{cluster="kub-34",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-prd", pod=~"thanos-receive-[0-9]*",container="receive"}) by (pod,container)

sum(container_memory_working_set_bytes{cluster="kub-53",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-prd", pod=~"thanos-receive-[0-9]*", container="receive"}) by (pod,container)

sum(container_memory_working_set_bytes{cluster="kub-10002",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-stg", pod=~"thanos-receive-[0-9]*", container="receive"}) by (pod,container)

sum(container_memory_working_set_bytes{cluster="kub-10010",job="kubelet", metrics_path="/metrics/cadvisor", namespace="monitoring-stack-corp-obs-corp-stg", pod=~"thanos-receive-[0-9]*", container="receive"}) by (pod,container)

Ensure on a 48h or 30 days time window that the memory consumption is growing over the time. If the growing is fast and cannot be explained with new clients or new zones, the operation needs to be communicated to the Monitoring Team. However, you can follow the procedure below.

Info

PF-3028 : a Grafana dashboard, named PF-2640 can also help instead of typing the PromQL expressions above. It should be renamed (in PF-3028). The expressions above may be replaced with a link to the new dashboard.

If the pod is a thanos-receive-X, use this procedure in order to scale-up Thanos Receiver.

Thanos-storegateway#

Warning

Always set the request/limit memory to the same value

The changes are made in the Git repository Here. You need to modify the files mon3.yaml and mon4.yaml.

The procedure for the upgrade is to be followed here.

Promtail#

If the pod is related to Promtail, also check Applications/Promtail/troubleshooting.

Alertmanager#

If the issue is related to Alertmanager (Caascad and NGOT), please check if the problem is related to CAASINC-569:

  • the memory usage is growing very slowly (check evolution in the last 30 days) ?
  • there is at least one ACK! silence in that Alertmanager
    • the ACK! silence can be on any alert, not limited to ContainerOutOfMemory
    • for Caascad zone, you can see it in Karma : select cc_client=<client> and cc_prom=<same cloud-XXX as in the alert> and check if you see silenced alerts with ACK!.

Tip

30/05/2023, when the NGOT Boyana will be deployed you have to filter silenced alerts on Karma using obs_client=<client>

Either you restart Alertmanager pod, or you put an ACK! silence in Karma and do nothing - it will restart by itself due to OOMKill.

Suggested resolution: restart the Alertmanager pod then close the CAASINC as duplicate as CAASINC-569.

Tip

For Caascad zone, if the memory usage graph has stopped growing and if you don't see any ACK! in Karma, please check the silence history with a command like this:

amtool-caascad --alertmanager.url ocb-test06/caascad silence query --expired
If you see expired ACKs, you were in the case described above (CAASINC-569). Prefer to kill the pod.