YaceClientMetricsMissing

Alert is triggered when yace_cloudwatch_requests_total metric is missing from Prometheus cloud-caascad for Caascad / Prometheus cluster for NGOT.

This alert either indicates that :

Tip

CaascadNGOT

• the YACE application does not return the yace_cloudwatch_requests_total metric
• or Prometheus cloud-app does not scrape YACE's metrics
• or Prometheus cloud-caascad does not federate YACE's metrics from Prometheus cloud-app

• the YACE application does not return the yace_cloudwatch_requests_total metric
• or Prometheus client does not scrape YACE's metrics
• or Prometheus cluster does not federate YACE's metrics from Prometheus client

Troubleshooting Hints

ACCOUNT_NAME is built with the account_name label from the Karma alert.

ACCOUNT_ID is built with the account_id label from the Karma alert.

YACE metrics from YACE application

Tip

NAMESPACE can be:

CaascadNGOT

• yace-client

• monitoring-stack-client-<contract_name>

To check if YACE metrics are available :

kubectl get svc -n ${NAMESPACE} yace-client-yace-${ACCOUNT_NAME}-${ACCOUNT_ID} # service should be present
kubectl port-forward svc/yace-${ACCOUNT_NAME}-${ACCOUNT_ID} -n ${NAMESPACE} 5000:80 &
curl -sL http://localhost:5000/metrics
curl -sL http://localhost:5000/metrics | grep yace_cloudwatch_requests_total

If YACE metrics are not available, try to find the reason :

• check status/logs of YACE

  kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=yace-${ACCOUNT_NAME}-${ACCOUNT_ID}
  kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=yace-${ACCOUNT_NAME}-${ACCOUNT_ID}
  

YACE metrics in Prometheus cloud-app for Caascad / Prometheus client for NGOT

Tip

CaascadNGOT

To check if YACE metrics are available in Prometheus cloud-app, check yace_cloudwatch_requests_total{account_name="${ACCOUNT_NAME}", account_id="${ACCOUNT_ID}"} in Thanos-app datasource.

If YACE metrics are not available in Prometheus cloud-app, try to find the reason why Prometheus cloud-app fails to scrape them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring-app | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring-app
  

• check the target, by doing a port-forward on the Prometheus cloud-app of the cluster.

  kubectl -n monitoring-app get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring-app port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. For account name, the YACE target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. For account name, the YACE target is absent, in this case you need to check why the target is not discovered by Prometheus.

One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

kubectl get servicemonitor -n yace-client -l app.kubernetes.io/name=yace-${ACCOUNT_NAME}-${ACCOUNT_ID} # servicemonitor should be present

To check if YACE metrics are available in Prometheus client, check yace_cloudwatch_requests_total{account_name="${ACCOUNT_NAME}", account_id="${ACCOUNT_ID}"} in Thanos datasource.

If YACE metrics are not available in Prometheus client, try to find the reason why Prometheus client fails to scrape them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring-stack-client-<contract_name> | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring-stack-client-<contract_name>
  

• check the target, by doing a port-forward on the Prometheus client of the cluster.

  kubectl -n monitoring-stack-client-<contract_name> get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring-stack-client-<contract_name> port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. For account name, the YACE target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. For account name, the YACE target is absent, in this case you need to check why the target is not discovered by Prometheus.

One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

kubectl get servicemonitor -n monitoring-stack-client-<contract_name> -l app.kubernetes.io/name=yace-${ACCOUNT_NAME}-${ACCOUNT_ID} # servicemonitor should be present

YACE metrics in Prometheus cloud-caascad for Caascad / Prometheus cluster for NGOT

Tip

CaascadNGOT

If YACE metrics are available in Prometheus cloud-app but not available in Prometheus cloud-caascad, try to find the reason why Prometheus cloud-caascad fails to federate them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring
  

• check the target, by doing a port-forward on the Prometheus cloud-caascad of the cluster.

  kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. For account name, the YACE federation target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. For account name, the YACE federation target is absent, in this case you need to check why the target is not discovered by Prometheus.

One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

kubectl get servicemonitor -n yace-client yace-client-multi-yace-federate # servicemonitor should be present

If YACE metrics are available in Prometheus client but not available in Prometheus cluster, try to find the reason why Prometheus cluster fails to federate them :

• check status/logs of Prometheus

  kubectl get pods -n monitoring | grep prometheus
  kubectl logs <prometheus_pod> -n monitoring
  

• check the target, by doing a port-forward on the Prometheus cluster

  kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
  kubectl -n monitoring port-forward svc/<prometheus_svc> 12345:9090 &
  

Connect to http://localhost:12345/targets.

There are two use cases :

1. For account name, the YACE federation target is present but in failure state (click on Unhealthy button). In this case check the logs to find the reason for which the target is failing

2. For account name, the YACE federation target is absent, in this case you need to check why the target is not discovered by Prometheus.

One possible reason could be the absence of the servicemonitor. Check that the servicemonitor is present:

kubectl get servicemonitor -n monitoring-stack-client-<contract_name> yace-client-multi-yace-federate # servicemonitor should be present