Disable alerts#

There are many possibilities:

Disable alerts for pod#

Alerts are generic. However some alerts are written in a way that they will not fire for some pods with specific labels.

This documentation explains how to deactivate specific alerts on a specific pod.

To deactivate the alert on a pod, add the pod label caascad_container_ignore_memory_alerts with the value "true".

In pod template:

spec:
  template:
    metadata:
      labels:
        caascad_container_ignore_memory_alerts:true

Warning

This solution excludes all the containers of a pod from the alert. There is no way to deactivate for specific containers only.

There is a trick in PromQL to disable alerts in some cases.

You can create a recording rule like :

expr:   "count by(cc_prom_source) (kube_pod_info{pod=~\".*cluster-autoscaler.*\"})"
record: "cluster_autoscaler:kube_pod_info:count_by"

Here we create a recording rule that counts the number of autoscaler pods.

And use this recording rule in the expression of the alert :

<expr> unless on(cc_prom_source) cluster_autoscaler:kube_pod_info:count_by != 0

The value of the cc_prom_source label must be equal on the left and on the right of the expression.

If there are autoscaler pods in the cluster (represented by cc_prom_source label), then the expression on the left is not evaluated.

Note

This is an example, other metrics can be used.