StackdriverExporterClientMetricsMissing

Alert is triggered when stackdriver_monitoring_scrapes_total metric is missing from Prometheus cloud-caascad for Caascad / Prometheus cluster for NGOT.

This alert either indicates that :

Tip

CaascadNGOT

• the Stackdriver-Exporter application does not return the stackdriver_monitoring_scrapes_total metric
• or Prometheus cloud-app does not scrape Stackdriver-Exporter's metrics
• or Prometheus cloud-caascad does not federate Stackdriver-Exporter's metrics from Prometheus cloud-app

• the Stackdriver-Exporter application does not return the stackdriver_monitoring_scrapes_total metric
• or Prometheus client does not scrape Stackdriver-Exporter's metrics
• or Prometheus cluster does not federate Stackdriver-Exporter's metrics from Prometheus client

Troubleshooting Hints

PROJECT_NUMBER is built with the project_number label from the karma alert.

PROJECT_ID is built with the project_id label from the karma alert.

Stackdriver-Exporter metrics from Stackdriver-Exporter application

Tip

NAMESPACE can be:

CaascadNGOT

• stackdriver-exporter-client

• monitoring-stack-client-<contract_name>

To check if Stackdriver-Exporter metrics are available :

kubectl get svc -n ${NAMESPACE} stackdriver-exporters-client-se-${PROJECT_ID}-${PROJECT_NUMBER} # service should be present
kubectl port-forward stackdriver-exporters-client-se-${PROJECT_ID}-${PROJECT_NUMBER} -n ${NAMESPACE} 9255:9255 &
curl -sL http://localhost:9255/metrics
curl -sL http://localhost:9255/metrics | grep stackdriver_monitoring_scrapes_total

If Stackdriver-Exporter metrics are not available, try to find the reason :

• check status/logs of Stackdriver-Exporter

  kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=se-${PROJECT_ID}-${PROJECT_NUMBER}
  kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=se-${PROJECT_ID}-${PROJECT_NUMBER}
  

Stackdriver-Exporter metrics in Prometheus cloud-app for Caascad / Prometheus cluster for NGOT

Tip

CaascadNGOT

To check if Stackdriver-Exporter metrics are available in Prometheus cloud-app, check stackdriver_monitoring_scrapes_total{project_number="${PROJECT_NUMBER}", project_id="${PROJECT_ID}"} in Thanos-app datasource.

If Stackdriver-Exporter metrics are not available in Prometheus cloud-app, try to find the reason why Prometheus cloud-app fails to scrape them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring-app | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring-app
  

• check the target, by doing a port-forward on the Prometheus cloud-app of the cluster.

  kubectl -n monitoring-app get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring-app port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. For project number, the Stackdriver-Exporter target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. For project number, the Stackdriver-Exporter target is absent, in this case you need to check why the target is not discovered by Prometheus.

   One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

   kubectl get servicemonitor -n stackdriver-exporter-client -l app.kubernetes.io/name=se-${PROJECT_ID}-${PROJECT_NUMBER} # servicemonitor should be present
   

To check if Stackdriver-Exporter metrics are available in Prometheus client, check stackdriver_monitoring_scrapes_total{project_number="${PROJECT_NUMBER}", project_id="${PROJECT_ID}"} in Thanos datasource.

If Stackdriver-Exporter metrics are not available in Prometheus client, try to find the reason why Prometheus client fails to scrape them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring-stack-client-<contract-name> | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring-stack-client-<contract-name>
  

• check the target, by doing a port-forward on the Prometheus cloud-app of the cluster.

  kubectl -n monitoring-stack-client-<contract-name> get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring-stack-client-<contract-name> port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. For project number, the Stackdriver-Exporter target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. For project number, the Stackdriver-Exporter target is absent, in this case you need to check why the target is not discovered by Prometheus.

   One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

   kubectl get servicemonitor -n monitoring-stack-client-<contract-name> -l app.kubernetes.io/name=se-${PROJECT_ID}-${PROJECT_NUMBER} # servicemonitor should be present
   

Stackdriver-Exporter metrics in Prometheus cloud-caascad for Caascad / Prometheus cluster for NGOT

Tip

CaascadNGOT

If Stackdriver-Exporter metrics are available in Prometheus cloud-app but not available in Prometheus cloud-caascad, try to find the reason why Prometheus cloud-caascad fails to federate them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring
  

• check the target, by doing a port-forward on the Prometheus cloud-caascad of the cluster.

  kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. The Stackdriver-Exporter federation target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. The Stackdriver-Exporter federation target is absent, in this case you need to check why the target is not discovered by Prometheus.

One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

kubectl get servicemonitor -n stackdriver-exporter-client stackdriver-exporters-client-multi-stackdriver-exporter-federate # servicemonitor should be present

If Stackdriver-Exporter metrics are available in Prometheus client but not available in Prometheus cluster, try to find the reason why Prometheus cluster fails to federate them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring
  

• check the target, by doing a port-forward on the Prometheus cluster.

  kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. The Stackdriver-Exporter federation target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. The Stackdriver-Exporter federation target is absent, in this case you need to check why the target is not discovered by Prometheus.

One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

kubectl get servicemonitor -n monitoring-stack-client-<contract_name> stackdriver-exporters-client-multi-stackdriver-exporter-federate # servicemonitor should be present

Known Case

• GCP backend returns 500 error

This issue is caused by an HTTP 500 error from GCP Monitoring API, typically indicating an internal server-side failure on Google's infrastructure. Possible reasons include:

• Stackdriver Monitoring backend temporarily failing to serve requests

• Metric service timeouts or delays in retrieving custom metrics

• Internal load or regional service disruption (not always visible publicly)

Pods State Check

kubectl get pods -n monitoring-stack-client-obs-<Client>

Result: All stackdriver-exporters-client-* pods are in Running state with 0 restarts. Logs check

kubectl logs stackdriver-exporters-client-se-<Client>-com-sandbox-services-XXXX -n monitoring-stack-client-obs-<Client>

If the return logs match the expected results, then you're on the right track.

googleapi: Error 500: Internal error encountered. Please retry after a few seconds. 
If internal errors persist, contact support at https://cloud.google.com/support/docs., backendError

Exporter pods are healthy, listening correctly on :9255, and project ID is correctly configured. The error is returned by the GCP Monitoring backend — it's an issue on GCP’s side.

Action

Pod restarts are not recommended at this point, as all exporter pods are currently running normally with no signs of abnormal restarts.

Check the GCP Status Dashboard to see if there are any ongoing regional or service-wide incidents affecting Cloud Monitoring.

As no action will be taken from our side, the customer should directly contact GCP official support for further investigation and resolution