PromitorScraperScrapeError

Alert is triggered when promitor_scrape_error is equal to one over a period of 10 minutes.

This alert either indicates that the scraping of one (or more) Azure resource has failed.

To identify the resource for which the scraping failed, several labels are available :

• tenant_id
• subscription_id
• resource_type (example: VirtualMachine)
• resource_name (example: vm-linux)
• resource_group

Troubleshooting Hints

Kubernetes side

• set variables

  # namespace and pod are labels in alert
  namespace=xxx
  pod=xxx
  

• check status/logs of Promitor Scraper

  kubectl get pod -n ${namespace} ${pod}
  kubectl logs -n ${namespace} ${pod}
  kubectl logs -n ${namespace} ${pod} --tail=-1 | grep FTL
  

• delete the scraper pod and see if the alert is triggered again :

  kubectl delete pod -n ${namespace} -l app=promitor-scraper-tenant1
  

Azure side

Check Promitor Troubleshooting in order to verify resource availability.

Known cases

Chanel

We check that we are in the known case of resources deleted on the client side :

NAMESPACE="monitoring-stack-client-obs-chanel"
kswitch svc-monitoring-stack-client-chanel
kubectl logs -n ${NAMESPACE} $(kubectl get pod -n ${NAMESPACE} | grep promitor-scraper | awk '$1 {print$1}') | grep FTL | grep ResourceNotFound

If the command returns lines, delete the scraper and resource-discovery pods :

kubectl delete pod -n ${NAMESPACE} $(kubectl get pod -n ${NAMESPACE} | grep promitor-scraper | awk '$1 {print$1}')
kubectl delete pod -n ${NAMESPACE} $(kubectl get pod -n ${NAMESPACE} | grep promitor-resource-discovery | awk '$1 {print$1}')

Warning

If deleting the pods is not enough, silence alert and notify the monitoring team.

Authentification failed

• set variables

  # namespace and pod are labels in the alert
  cluster=kub-xxx
  namespace=xxx
  pod=promitor-scraper-xxx
  

• check logs of the Promitor Scraper

  kswitch ${cluster}
  kubectl logs ${pod} -n ${namespace} | grep "error"
  

  When we find a log error for The provided client secret keys for app 'XXX-XXX-XXX-XXX-XXX' are expired, the root cause is that the promitor authentification token has expired. We can do nothing to repair it by ourselves.

ClientSecretCredential authentification failed (key expiry)

The root cause is a problem with the Azure authentification. We can do nothing to repair by ourselves.

• Communicate to the client. The message could be like this :

  Bonjour,

  Nous avons constaté une perte de métriques issues d'Azure de votre zone OBS-XXX. L'incident a démarré à HHhMM.

  Cet incident a pour origine un dysfonctionnement sur l'authentification d'Azure avec l'expiration de la secret key de l'application: XXXXX

  Merci de nous communiquer un nouveau WrapToken avec les nouvelles informations d'authentification.

  Nous attirons votre attention sur le fait qu'en l'absence de ces métriques, les alertes basées sur celles-ci ne fonctionnent pas.

  Nous restons à votre disposition pour toute information complémentaire.

  Cordialement

• The client provides the token:

  Unwrap token here

  Replacing the old token with the new one in Vault=>svc-monitoring-stack-client-XXX/promitor-client/tenant1. No merge is needed.

  Perform a trackbone apply from master on the contexts/ngot

  trackbone apply -z "svc-monitoring-stack-client-xxx" -c promitor-client 
  

  Check if pods restart:

  kubectl get pod -n monitoring-stack-client-obs-XXX | grep promitor
  

  If they do not restart, kubectl delete to force them restart.

  kubectl delete pod -n monitoring-stack-client-obs-XXX
  

No metrics for azure_application_gateway_backend_response_status

In Grafana, check if there are metrics for azure_application_gateway_backend_response_status{subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"} where the subscription_id is given in the alert. When we get no metrics in Grafana, the root cause is a problem with the Azure Application Gateway. We can do nothing to repair it by ourselves. If metrics are missing during the incident, you should inform the client by following the procedure below.

• Communicate to the client. The message could be like this:

  Bonjour,

  Nous avons constaté une perte de métriques issues d'Azure avec le tenant ID xxxxxx, subscription ID xxxxxxxx, resource type xxxxx, resource name xxxxxxxx et resource group xxxxxx. L'incident a démarré à HHhMM.

  Cet incident a pour origine un dysfonctionnement de la Application Gateway d'Azure.

  Nous attirons votre attention sur le fait qu'en l'absence de ces métriques, les alertes basées sur celles-ci ne fonctionnent pas.

  Si la durée de l'incident est supérieure à 2 heures, nous vous recontacterons avec de plus amples informations.

  Nous restons à votre disposition pour toute information complémentaire.

  Cordialement

• Set a silence (not a ACK). The downtime period must be calculated to be exactly 2h after the beginning of the incident (do not set 2h without calculating !)

• If the alert comes back after 2 hours, restart the investigations and communicate again to the client.

Azure resource temporarily stopped by the client

• Set variables:

  # namespace and pod are labels in the alert
  cluster=kub-xxx
  namespace=xxx
  pod=promitor-scraper-xxx
  

• Check logs of the Promitor Scraper:

  kswitch ${cluster}
  kubectl logs ${pod} -n ${namespace} | grep "FTL"
  

  When we find a log FTL like:

  ResourceNotFound: The Resource `XXX-XXX-XXX-XXX-XXX-XXX` under resource group `XXX-XXX-XXX-XXX-XXX-XXX` was not found
  

  It usually indicates that the resource no longer exists or has been stopped.

• Contact to the client to confirm whether the resource (e.g., resource group XXX-XXX-XXX-XXX-XXX-XXX) has been temporarily stopped. The message could be like this :

  Bonjour,

  Nous avons constaté une perte de métriques issues d'Azure avec le tenant ID xxxxxx, subscription ID xxxxxxxx, resource type xxxxx, resource name xxxxxxxx et resource group xxxxxx. L'incident a démarré à HHhMM. Pouvez-vous nous confirmer si cette ressource groupe XXX-XXX-XXX-XXX-XXX-XXX est toujours active ?

  Nous restons à votre disposition pour toute information complémentaire.

  Cordialement

• Set a silence (not a ACK). Since the client has confirmed the intentional stop of the resources group, and provided a known reactivation date.

• If the client does not confirm that the resource group has been stopped, restart the investigations and communicate again to the client.

Persistent Karma errors

If errors persist, confirm that the client-provided metric_name exist within Azure. For this, consult the following link:

https://learn.microsoft.com/en-us/azure/azure-monitor/reference/metrics-index

Other cases

Note

If this incident happens, please note your investigation and update this documentation.