Aller au contenu

ConnectionKarmaAlertmanagerFailed#

This alert can appear only in Karma infra zone (infra-stg or infra-prd) and it indicates that in cloud-app zone, Karma cannot communicate with Alertmanager.

Note

The severity of the alert is critical until we have a better alert when a cluster is down

When this alert is firing outside working hours, check these 2 critical possible causes :

  • is the zone down ?
  • is the alertmanager still up and running ?

Any other possible cause will be investigated at working hours.

Troubleshooting#

Promtail parses Karma logs (cloud-app zone) and triggers this alert when it encounteres one of the following errors:

  • no such host or failed with 403 Forbidden : can be caused by a wrong Alertmanager URL configured in Karma (both in cloud-app zone), wrong DNS resolution, etc
  • failed with 401 Unauthorized : this is caused when the token configured in Karma to access Alertmanager is not correct (both in cloud-app zone).
  • context deadline exceeded : this error can appear when the whitelist of cloud-app zone does not contain NAT GW of infra-stg

Check cluster:#

You need to check the status of the cluster masters from the FE console or cmd Openstack

os s ${zone}
os cce cluster list
os cce cluster node list <ID cluster>

If you notice a malfunction open an incident ticket to FE support here.

Check Karma (cloud-app zone) logs#

kswitch <zone_name>
kubectl -n monitoring-app get pods | grep karma #get karma pod name (karma-xxxxx-xxxxx)
kubectl -n monitoring-app logs karma-xxxxx-xxxxx

Check Alertmanager URL configured in Karma (cloud-app zone)#

kubectl -n monitoring-app get configmap karma-config -o yaml

View Rancher token configured in Karma (infra-stg or infra-prd) to acces Alertmanager (cloud-app zone )

kubectl exec -it -n monitoring deploy/karma -- bash
cat /tmp/karma/karma.conf