PromitorResourceDiscoveryRequestErrors#

This alert is triggered when requests sent to Azure Resource Graph have a status other than 200.

For more information, several labels are available :

status_code: identifies the status of requests
path: identifies the resource group for which queries are failing (/api/v2/resources/groups/<group_name>/discover)

Troubleshooting Hints#

set variables

# naespace and name are labels in alert
namespace=xxx
name=xxx

check status/logs of Promitor Resource Discovery

kubectl get pod -n ${namespace} -l app.kubernetes.io/name=promitor-resource-discovery-${name}
kubectl logs -n ${namespace} -l app.kubernetes.io/name=promitor-resource-discovery-${name}

check health of Promitor Resource Discovery

kubectl port-forward svc/promitor-resource-discovery-${name} -n ${namespace} 8889:8889 &
curl "http://localhost:8889/api/v1/health?includeDependencies=true" | jq .

Solutions#

ClientSecretCredential authentification failed (key expiry)#

The root cause is a problem with the Azure authentification. We can do nothing to repair by ourselves.

Communicate to the client. The message could be like this :

Bonjour,

Nous avons constaté une perte de métriques issues d'Azure de votre zone OBS-XXX. L'incident a démarré à HHhMM.

Cet incident a pour origine un dysfonctionnement sur l'authentification d'Azure avec l'expiration de la secret key de l'application: XXXXX

Merci de nous communiquer un nouveau WrapToken avec les nouvelles informations d'authentification.

Nous attirons votre attention sur le fait qu'en l'absence de ces métriques, les alertes basées sur celles-ci ne fonctionnent pas.

Nous restons à votre disposition pour toute information complémentaire.

Cordialement
The client provides the token:

Unwrap token here

Replacing the old token with the new one in Vault=>svc-monitoring-stack-client-XXX/promitor-client/tenant1. No merge is needed.

Perform a trackbone apply from master on the contexts/ngot
```
trackbone apply -z "svc-monitoring-stack-client-xxx" -c promitor-client 
```
Check if pods restart:
```
kubectl get pod -n monitoring-stack-client-obs-XXX | grep promitor
```
If they do not restart, kubectl delete to force them restart.
```
kubectl delete pod -n monitoring-stack-client-obs-XXX
```