BlackboxGRPCDown#

The Blackbox exporter conducts tests, known as probes, on endpoints and provides metrics based on the outcomes. This alert is based on those metrics.

To determine where is the problem, refer to the target label in the alert.

The target label provides you two informations:

the failing endpoint (example: thanos-querier-proxy-mon3.obs-corp-stg.csfpriv.com, in this case the issue is Thanos-querier on mon3 zone);
the name of the failed module (example: grpc_thanos_querier_proxy_mon3_tls,in this case the issue is GRPC communication with mon3).

This alerts happens when a target of Blackbox-Exporter is down.

Important

When this alert happens, THIS IS NOT AN ISSUE WITH BLACKBOX-EXPORTER. This alert cannot appear if Blackbox-Exporter is broken.

Check points#

Identify the target (label target in alert)
Follow debug documentation
Identify the problem :
- If you don't get a GRPC response your probably have a DNS problem
- If you can find a "Probe failed", PROBE with which you are trying to dialog is not available. Make sure this service are in state running Natively, the Blackbox-Exporter will prefer an IPv6 address if one is returned by DNS. You can adjust this behaviour by adding preferred_ip_protocol: "ip4" to the module configuration if you are not using IPv6.
- It may be an expired Thanos certificate.
- ...

Known cases#

Certificate in Vault secrets is expired#

Blackbox-Exporter uses certificates to connect to Thanos endpoints:

Thanos sidecar;
Thanos store gateway;
Thanos querier (only on central zones).

Important

Thanos querier certificates on client zones are managed with zerossl so they are not concerned by this known case.

To identify if your issue is in this known case:

Check if the module label name contains tls.

Examples mon3 and mon4:
- grpc_thanos_querier_proxy_mon*_tls.

Check the certificate.

Identify to Vault:

LINE=prd # or stg
export VAULT_ADDR=https://vault.infra-$LINE.caascad.com/
vault token lookup > /dev/null 2>&1 || vault login -method oidc

Variables declaration:

clientmon3mon4

CLIENT=xxx # exemple: CLIENT=pf
ZONE=svc-monitoring-stack-client-$CLIENT
CERTIFICATE=thanos-sidecar-certificate
# or CERTIFICATE=thanos-store-gateway-certificate

LINE=prd # or stg
ZONE=svc-monitoring-stack-corp-${LINE}-1
CONTRACT=obs-corp-${LINE}
CERTIFICATE=thanos-querier-proxy-certificate

LINE=prd # or stg
ZONE=svc-monitoring-stack-corp-${LINE}-2
CONTRACT=obs-corp-${LINE}
CERTIFICATE=thanos-querier-proxy-certificate

Check the certificate:

vault read /secret/zones/fe/$ZONE/$CERTIFICATE -format=json| jq -r '.data.certificate' |jq -r '.["tls.crt"]' |base64 -d| openssl  x509 -enddate  -noout

Check the CA certificate:

vault read /secret/zones/fe/$ZONE/$CERTIFICATE -format=json| jq -r '.data.certificate' |jq -r '.["ca.crt"]' |base64 -d| openssl  x509 -enddate  -noout

Warning

Continue with the next steps only if the tls.crt certificate is expired.

At this date (28/06/2024) the CA certificate expiration is not documented.

Delete the secrets.

Variables declaration:

clientmon3/mon4

NS=monitoring-stack-client-obs-$CLIENT
SECRET=thanos-sidecar-tls-secret
# or SECRET=thanos-store-gateway-tls-secret

NS=monitoring-stack-corp-obs-corp-$LINE 
SECRET=thanos-querier-proxy-tls-secret

Delete the secret:

kswitch $ZONE
kubectl -n $NS delete secret $SECRET

Regenerate the secrets.

For Thanos sidecar:

trackbone apply -z $ZONE -c kube-prometheus-stack -t bootstrap=true

For Thanos store gateway:

trackbone apply -z $ZONE -c  thanos-store

For Thanos querier:

trackbone apply -z $ZONE -c  thanos_certificates

Then, synchronize the new certificate with Vault:

querier_proxy_certificate=$(kubectl -n "monitoring-stack-corp-${CONTRACT}" \
get secret thanos-querier-proxy-tls-secret -o jsonpath="{ .data }")

vault write "secret/zones/fe/${ZONE}/thanos-querier-proxy-certificate" \
certificate=${querier_proxy_certificate} # Write it to Vault

Check the validity of the new certificate.

Follow the same procedure at step 2.

List the secrets and delete new ones.

Note

Because the secrets have been generated with new passwords, it is necessary to keep only the old ones.

vault list secret/zones/fe/$ZONE/prometheus-ingress/

Output example:

Keys
----
init-directory
prometheus-ingress-20230515-150858
prometheus-ingress-20240610-172019

In this example, you need to delete the newest one:

vault delete secret/zones/fe/$ZONE/prometheus-ingress/prometheus-ingress-20240610-172019

Redeploy Prometheus and Blackbox-Exporter on central zones.

trackbone apply -z $ZONE -c kube-prometheus-stack
trackbone apply -z svc-monitoring-stack-corp-${LINE}-1 -c blackbox-exporter-core -t blackbox_exporter_refresh_cache=true
trackbone apply -z svc-monitoring-stack-corp-${LINE}-2 -c blackbox-exporter-core
trackbone apply -z svc-grafana-central-corp-${LINE} -c thanos_certificates

Check that the alert has not reappeared in Karma after 10 minutes.

Redeploy Thanos Querier for a Mutualised Grafana Instance#

A shared Grafana instance is connected to multiple supervision environments.

To determine if the issue is related to Thanos certificate expiration, you can verify the data sources on the client Grafana UI: https://grafana.obs-$CLIENT.cloudservicesfactory.com/connections/datasources.

Note

Proceed only if the data source URLs include the names of other clients.

Use Cases#

There are two scenarios based on the location of the remote supervision environment:

When Grafana is configured to connect to a supervision environment located in the same cluster (local connection).
When Grafana is configured to connect to a supervision environment located in a different cluster (remote connection).

Local Connection#

If the remote supervision environment is in the same cluster as the shared Grafana instance, the connection uses a local URL and does not require a secure connection via a certificate. In this case, the expiration of the Thanos sidecar or Thanos store gateway certificates on the remote environment is not an issue.

Remote Connection#

If the remote supervision environment is in a different cluster, if Thanos sidecar or Thanos store gateway certificates expiration is an issue because mutualised Grafana instance will no longer be able to connect to it.

After renewing the Thanos Sidecar and/or Thanos Store Gateway certificates in the remote environment, as described in the Known Cases - Certificate in Vault Secrets is Expired section, you will need to redeploy the Thanos Querier in the shared Grafana environment using the following command:

trackbone apply -z $MUT_GRAFANA -c thanos-query

Examples of $MUT_GRAFANA zone names:

Staging:
- stgteam01 (connected to staging01 and staging02), devteam01 (connected to test04 and test06)
Production:
- rsc (connects to adh, avt, and ocd)
- evp, evpcco, evpcns (all connected to evpo)

Certificate in Vault secrets is not expired#

In this case you need to go further with the investigation.

Connect to Blackbox-Exporter with the failed probe and check the logs.

instance label in the alert provides you the information regarding the Blackbox-Exporter with a failed probe.

LINE=prd # or LINE=stg
kswitch $ZONE # from label ngot_service in the alert
kubectl port-forward svc/blackbox-exporter-obs-corp-${LINE} -n monitoring-stack-corp-obs-corp-${LINE} 9115:9115 &

Check the logs of the Blackbox-Exporter failed probe.

Go to http://localhost:9115, identify the failed target that generates the alert and click on Logs.

The following log indicate that Blackbox-Exporter cannot connect to Thanos endpoint:
```
desc = unexpected HTTP status code received from server: 400 (Bad Request); transport: received unexpected content-type \"text/html\""
```
The following log indicate that there is an issue with SSL. It could be the certificate.
```
probe_grpc_ssl 0
```

Control the certificate in the Kubernetes secrets

Variables declaration:

clientmon3/mon4

NS=monitoring-stack-client-obs-$CLIENT
SECRET=thanos-sidecar-tls-secret
# or SECRET=thanos-store-gateway-tls-secret

NS=monitoring-stack-corp-obs-corp-$LINE 
SECRET=thanos-querier-proxy-tls-secret

Control:

kubectl -n $NS get secret $SECRET -o json |jq -r '.data["tls.crt"]' | base64 -d | openssl  x509 -enddate  -noout

Warning

Continue with the next steps only if Kubernetes secrets are expired.

Follow the steps 3 to 8 in Known case : Certificate in Vault secrets is expired.
Additional step: this situation should never happen. Additional investigation is needed to explain why it happened.