Aller au contenu

InhibitorClusterAlerts#

This alert is triggered when the number of alerts in NGOT is superior to a defined threshold. It will mute all the alerts in NGOT with severity warning and critical.

Important

You need to identify the alerts that triggered the inhibitor by following the procedure below. For each issue, open a CAASINC with the name of the real alert (not with InhibitorClusterAlerts).

Troubleshooting Hints#

Display critical inhibited alerts#

List critical alerts that are inhibited with the filters @inhibited=true and severity=critical:

Warning

When the InhibitorClusterAlerts is not active in Karma, the filter @inhibited=true will hide most of the alerts. In this case, remove this filter.

This will happens when some problems get solved.

List problems#

  1. List all the alerts by their name and clients: this gives the list of distinct issues to fix.
  2. Create a CAASINC for each distinct issue.
  3. In Karma, remove the filter severity=critical: some issues come with additionnal alerts with different severities. It may helps to know them.
  4. To fix each issue, refer to the dedicated help_alerts documentation for its alert.

Tip

You may silence alerts to remove the InhibitorClusterAlerts one. You may write the same comment for all the alerts that relate to the same issue (same CAASINC).

Known cases#

Many PrometheusXXX alerts#

You have many alerts like:

  • PrometheusDown
  • PrometheusRemoteStorageFailure
  • PrometheusRemoteWriteBehind
  • xxxDown

You may follow this doc