PromitorResourceDiscoveryDown

Alert is triggered when promitor_runtime_http_request_duration_seconds_count{job="promitor-agent-resource-discovery"} metric is missing from Prometheus cluster.

This alert either indicates that :

• the Promitor Resource Discovery application does not return the promitor_runtime_http_request_duration_seconds_count metric
• or Prometheus client does not scrape Promitor Resource Discovery's metrics
• or Prometheus cluster does not federate Promitor Resource Discovery's metrics from Prometheus client

Troubleshooting Hints

Set variables :

# namespace and name are labels in alert
namespace=xxx
name=xxx

Promitor Resource Discovery metrics from Prometheus Resource Discovery application

To check if Promitor Resource Discovery metrics are available :

kubectl get svc -n ${namespace} promitor-resource-discovery-${name} # service should be present
kubectl port-forward svc/promitor-resource-discovery-${name} -n ${namespace} 8889:8889 &
curl -sL http://localhost:8889/metrics
curl -sL http://localhost:8889/metrics | grep promitor_runtime_http_request_duration_seconds_count

If Promitor Resource Discovery metrics are not available, try to find the reason :

• check status/logs of Promitor Resource Discovery

  kubectl get pod -n ${namespace} -l app.kubernetes.io/name=promitor-resource-discovery-${name}
  kubectl logs -n ${namespace} -l app.kubernetes.io/name=promitor-resource-discovery-${name}
  

Promitor Resource Discovery metrics in Prometheus client

To check if Promitor Resource Discovery metrics are available in Prometheus client, check promitor_runtime_http_request_duration_seconds_count{job="promitor-agent-resource-discovery"} in Grafana client (thanos datasource).

If Promitor Resource Discovery metrics are not available in Prometheus client, try to find the reason why Prometheus client fails to scrape them :

• check status/logs of Prometheus client

  kubectl get pods -n ${namespace} | grep prometheus
  kubectl logs <prometheus_pod> -n ${namespace}
  

• check the target, by doing a port-forward on the Prometheus client of the cluster.

  kubectl -n ${namespace} get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n ${namespace} port-forward svc/<prometheus_svc> 12345:9090 &
  

  Connect to http://localhost:12345/targets.

  There are two use cases :

  1. The Promitor target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

  2. The Promitor target is absent, in this case you need to check why the target is not discovered by Prometheus.

     One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

     kubectl get servicemonitor -n ${namespace} promitor-client-promitor-resource-discovery-${name} # servicemonitor should be present
     

Promitor Resource Discovery metrics in Prometheus cluster

If Promitor Resource Discovery metrics are available in Prometheus client but not available in Prometheus cluster, try to find the reason why Prometheus cluster fails to scrape them :

• check status/logs of Prometheus cluster

  kubectl get pods -n monitoring | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring
  

• check the target, by doing a port-forward on the Prometheus cluster.

  kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &
  

  Connect to http://localhost:12345/targets.

  There are two use cases :

  1. The Promitor target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

  2. The Promitor target is absent, in this case you need to check why the target is not discovered by Prometheus.

     One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

     kubectl get servicemonitor -n ${namespace} promitor-client-multi-promitor-federate # servicemonitor should be present