HarborMetricsMissing#
This alert is triggered when Harbor metrics are missing for more than 5 minutes.
Troubleshooting hints#
Harbor service#
Identify the cluster and check if the harbor services are still available in the cluster.
Tip
kubectl get svc -n harbor-svc-oci-registry-prd -l app=harbor
kubectl get svc -n harbor-svc-oci-registry-stg -l app=habor
Harbor pod#
Identify the cluster and check the status/logs of the pods.
Tip
kubectl get pods -n harbor-svc-oci-registry-prd | grep harbor
kubectl logs -n harbor-svc-oci-registry-prd -l app=harbor,component=core
kubectl get pods -n harbor-svc-oci-registry-stg | grep harbor
kubectl logs -n harbor-svc-oci-registry-stg -l app=harbor,component=core
Servicemonitors#
-
Check that servicemonitor is present
kubectl get servicemonitor -A | grep harbor # Because the names of the servicemonitors usually contain harbor -
Check that Blackbox-Exporter metrics
probe_success{}are available in Grafana.Tip
Don't forget to select the correct Datasource in Grafana.
Probe#
Check that probe is present.
Note
Probes are defined only on NGOT and only on Thanos Receiver central monitoring stacks (mon3/mon4).
On NGOT central monitoring stacks mon3 (svc-monitoring-stack-corp-prd-1) and mon4 (svc-monitoring-stack-corp-prd-2):
```bash
kswitch svc-monitoring-stack-corp-prd-1 # or kswitch svc-monitoring-stack-corp-prd-2
kubectl get probe -n monitoring-stack-corp-obs-corp-prd | grep harbor
On NGOT central monitoring stacks mon3 (svc-monitoring-stack-corp-stg-1) and mon4 (svc-monitoring-stack-corp-stg-2):
kswitch svc-monitoring-stack-corp-stg-1 # or kswitch svc-monitoring-stack-corp-stg-2
kubectl get probe -n monitoring-stack-corp-obs-corp-stg | grep harbor
Blackbox-exporter#
Check if Blackbox-Exporter works correctly and probes the good targets. More information here.