Aller au contenu

VaultMetricsMissing#

Alert is triggered when vault_runtime_alloc_bytes metric is missing from Prometheus-cluster.

This alert either indicates that:

  • Vault does not return the vault_runtime_alloc_bytes metric
  • Or Prometheus-cluster does not scrape vault's metrics.

Troubleshooting Hints#

NAMESPACE=vault

Case: Vault does not return metrics#

To check if vault metrics are available:

kubectl get svc -n ${NAMESPACE} # service should be present
kubectl port-forward svc/vault -n ${NAMESPACE} 8200:8200 &
curl -k https://localhost:8200/v1/sys/metrics
curl -k https://localhost:8200/v1/sys/metrics | grep -o '"Name":"vault.runtime.alloc_bytes"'

If metrics are not available, try to find the reason by checking status/logs of vault:

kubectl get pod -n ${NAMESPACE} -l app.kubernetes.io/name=vault
kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=vault

Case: Prometheus-cluster does not scrape vault's metrics#

  • check status/logs of Prometheus

    kubectl get pods -n monitoring | grep prometheus
kubectl logs <prometheus_pod> -n monitoring

  • check that the servicemonitor is present

    kubectl get servicemonitor -n ${NAMESPACE} vault

  • check the target, by doing a port-forward on the Prometheus-cluster

    kubectl -n monitoring get svc -l app=kube-prometheus-stack-prometheus
kubectl -n monitoring port-forward svc/<prometheus_svc> 54321:9090 &

    Connect to http://localhost:54321/targets, search vault target, and verify error message.

    Warning

    Be careful of clear selection before filtering on vault.

Case: Vault does not reload its internal certificate correctly#

  • check in the corresponding grafana if there are some error logs about bad certificate.

    {namespace="vault", app="vault"} |~ "cert"
    - if there are some logs about certificate error, restart the vault to reload all its internal certifcate correctly.

    kubectl rollout restart statefulset -n vault -l app.kubernetes.io/name=vault
    - ensure that all the pod are in a running state.

    kubectl get pod -A -l app.kubernetes.io/name=vault

    Note

    If that's insufficient, terminate the Vault pods one at a time (allowing time in between).