CloudeyeExporterClientMetricsMissingFE

This alert is triggered when cloudeye-exporter-client metrics are missing for more than 10 minutes.

Warning

There are three alerts.

• CloudeyeExporterMetricsMissingFE: Fires when cloudeye-exporter is not running correctly. Fires only for Caascad clients and for mon3/mon4 on Ngot.
• CloudeyeExporterClientMetricsMissingFE: Fires when cloudeye-exporter is not running correctly. Fires only for Ngot clients.
• CloudeyeExporterResourcesMetricsMissingFE: Fires when cloudeye-exporter does not collect any metrics. Fires only on Ngot (both client and mon3/mon4).

When CloudeyeExporterResourcesMetricsMissingFE and another alerts are both firing, first investigate on the other one.

Cloudeye-Exporter doc here.

This component is stateless, you can kill the pod safely.

Troubleshooting hints

SERVICE_NAME is built with the service label from the Karma alert.

Tip

NAMESPACE can be:

CaascadNGOT

• cloudeye-exporter-client

• monitoring-stack-client-<contract_name>

• Check that servicemonitor is present

  kubectl get servicemonitor -n ${NAMESPACE} "$SERVICE_NAME-metrics"
  

• Check if the exporter is healthy

  export POD_NAME=$(echo "$SERVICE_NAME" | sed -e "s/ce-//")
  kubectl get pods -l "app.kubernetes.io/name=$POD_NAME" -n ${NAMESPACE}
  kubectl logs -l "app.kubernetes.io/name=$POD_NAME" -n ${NAMESPACE}
  

• Check that cloudeye-exporter-client metrics (ex: go_info{job="$SERVICE_NAME"}) are available on the Grafana.

Tip

CaascadNGOT

Choose either the Thanos datasource if the problem concerns the cloud zone or Thanos-client if it is about a client zone (the only client zones where cloudeye-exporter is deployed is corp)

Connect to https://grafana.$contract_name.cloudservicesfactory.com and choose the Thanos datasource.

• Check if cloudeye-exporter-client metrics are available in Prometheus

Tip

CaascadNGOT

kubectl port-forward svc/caascad-prometheus -n monitoring 9090:9090 &

kubectl port-forward svc/<contract_name>-prometheus -n ${NAMESPACE} 9090:9090 &

Access Prometheus URL (http://localhost:9090) and check:

• In the main menu, metrics for cloudeye-exporter-client (ex: go_info{job="$SERVICE_NAME"}), should be available.

• In Status > Targets menu, cloudeye-exporter-client target should be up. If it's not, there are two use cases:

  1. If the target is present but in failure state (click on Unhealthy button), check the logs to find the reason for which the target is failing.

  2. The target is absent, in this case you need to Troubleshoot why the target is not discovered by Prometheus.