Aller au contenu

Use Blackbox-Exporter#

Introduction#

Here are the steps to do to monitor an URL with Blackbox-Exporter.

  1. Check the modules already available in blackbox-exporter. If these modules are not sufficient to check the service: contact the team that manages Blackbox-Exporter. In most cases, http_2xx is what you need.
  2. Create a serviceMonitor with one of the modules and your service.
  3. If you need new rules that aren't already deployed in your cluster, create the new rules and redeploy them.

There are 2 ways to create a serviceMonitor :

  • standalone : the application you monitor with Blackbox-Exporter is not deployed in the same cluster and namespace as the serviceMonitor.
  • integrated in an application : you can deploy the serviceMonitor at the same time you deploy the application, for example in the same helm chart.

Adding a new standalone serviceMonitor for Blackbox-Exporter#

Deploying a standalone serviceMonitor can be done with the helm chart caascad-servicemonitor.

In envs-ng there are 2 facilities to deploy serviceMonitors with the caascad-servicemonitor chart :

  • #CaascadServicemonitorHelmConfig for generic serviceMonitors
  • #CaascadServicemonitorBlackboxexporterHelmConfig dedicated for generic Blackbox-Exporter serviceMonitors. This is the preferred method.

Method with #CaascadServicemonitorBlackboxexporterHelmConfig#

envs.cue#

In envs.cue add a line like this : 

"blackbox-exporter-servicemonitor-<your app>": #CaascadServicemonitorBlackboxexporterHelmConfig

blackbox-exporter-servicemonitor-<your_app>.cue#

Create a new file named blackbox-exporter-servicemonitor-<your app>.cue.

Tip

The name of the configuration and the file should be the same, as usual. The name does not matter. However, in order to be able to find it easily, it is better to prefix with blackbox-exporter-servicemonitor-.

Here is the contents of the file :

package terraform

envs: [string]: {
    zone:        _
    parent_zone: _
    configurations: ["blackbox-exporter-servicemonitor-<your app>"]: {
        owners: ["<your team>"]
        providers: kubernetes: "\(zone.name)": _

        helm: {
            release_name: "blackbox-exporter-servicemonitor-<your app>"
            namespace:    "<namespace where to deploy>"
            values: {
                fullnameOverride: "blackbox-exporter-<your app>"

                servicemonitorMap: {
                    "blackbox-exporter-<your app>": {
                        prometheusZone: "caascad"
                        spec: {
                            endpoints: [ #CaascadServicemonitorBlackboxexporterEndpoints & {
                                interval: "60s"
                                params: {
                                    module: [ "http_2xx"]
                                    target: [ "https://<your target>.corp.caascad.com"]

                                }
                                relabelings: #CaascadServiceMonitorBlackboxexporterRelabelingsZoneInfra
                            }]
                            namespaceSelector: matchNames: [ "monitoring"]
                        }
                    }
                }
            }
        }
    }
}
  • helm.namespace : this is the namespace where you want to deploy the serviceMonitor. This can be any namespace. In most cases, you do NOT want to write monitoring there.
  • helm.values.servicemonitorMap.prometheusZone : this is the name of the Prometheus that will scrap the Blackbox-Exporter metrics of your target. It will become the value of caascad.com/prometheus-monitor label. Check the nomenclature Caascad for possible values (usually caascad or client).
  • helm.values.servicemonitorMap...spec.endpoints :
    • .interval and .params are easy to understand. However, notice that .params.module and .params.target are arrays, but they cannot have more than one value. Any additionnal value will either be ignored or even break the scrapping.
    • .relabelings : in most cases, you should use a standard relabelings. There are 3 possible values defined in envs-ng schema_helm_caascad_servicemonitor.cue file. If you need additional relabelings, you cannot use #CaascadServicemonitorBlackboxexporterHelmConfig as the method of defining your serviceMonitor : you will probably fall back to #CaascadServicemonitorHelmConfig. Here are the 3 possible values :
      • #CaascadServiceMonitorBlackboxexporterRelabelingsZoneInfra
      • #CaascadServiceMonitorBlackboxexporterRelabelingsZoneCloud
      • #CaascadServiceMonitorBlackboxexporterRelabelingsZoneClient
    • .namespaceSelector.matchNames : this array should always defined as one of ["monitoring"] (infra and cloud zones) or ["caascad-monitoring"] (client zones). This is the namespace where Blackbox-Exporter is deployed.

Deploying#

Deploy as usual with Trackbone.

Add ruleset in applications/caascad-prometheus-rules in order to alert in case the metrics of blackbox servicemonitors are missing or have a bad configuration.

Method with #CaascadServicemonitorHelmConfig#

In most cases, you choose this method because you want to set specific values in relabelings or metricRelabelings that you cannot do with the method with #CaascadServicemonitorBlackboxexporterHelmConfig.

envs.cue#

In envs.cue add a line like this : 

"blackbox-exporter-servicemonitor-<your app>": #CaascadServicemonitorHelmConfig

blackbox-exporter-servicemonitor-<your_app>.cue#

Create a new file named blackbox-exporter-servicemonitor-<your app>.cue.

Tip

The name of the configuration and the file should be the same, as usual. The name does not matter. However, in order to be able to find it easily, it is better to prefix with blackbox-exporter-servicemonitor-.

Here is the contents of the file :

package terraform

envs: [string]: {
    zone:        _
    parent_zone: _
    configurations: ["blackbox-exporter-servicemonitor-<your app>"]: {
        owners: ["<your team>"]
        providers: kubernetes: "\(zone.name)": _

        helm: {
            release_name: "blackbox-exporter-servicemonitor-<your app>"
            namespace:    "<namespace where to deploy>"
            values: {
                fullnameOverride: "blackbox-exporter-<your app>"

                servicemonitorMap: {
                    "blackbox-exporter-<your app>": {
                        prometheusZone: "caascad"
                        spec: {
                            endpoints: [{
                                interval:          "60s"
                                path:              "/probe"
                                port:              "http"
                                scheme:            "http"
                                metricRelabelings: [...#PrometheusOperatorRelabelConfig] & [
                                            {
                                        action: "labeldrop"
                                        regex:  "pod"
                                    },
                                ]
                                params: {
                                    module: [
                                        "http_2xx",
                                    ]
                                    target: [
                                        "https://<your target>.corp.caascad.com",
                                    ]

                                }
                                relabelings: [...#PrometheusOperatorRelabelConfig] & [
                                        {
                                        separator: "/"
                                        sourceLabels: [
                                            "__param_target",
                                            "__param_module",
                                        ]
                                        targetLabel: "target"
                                    }, {
                                        separator: "/"
                                        sourceLabels: [
                                            "__param_target",
                                            "__param_module",
                                        ]
                                        targetLabel: "instance"
                                    }, {
                                        sourceLabels: [
                                            "__param_module",
                                        ]
                                        targetLabel: "module"
                                    }, {
                                        // Do not set cc_prom_source in client zones like corp
                                        replacement: "cloud-caascad"
                                        targetLabel: "cc_prom_source"
                                        // Do not set cc_prom_source in client zones like corp
                                    },
                                ]
                            }]
                            jobLabel: "blackbox-exporter"
                            namespaceSelector: matchNames: [ "monitoring"]
                            selector: matchLabels: {
                                "app.kubernetes.io/instance": "blackbox-exporter"
                                "app.kubernetes.io/name":     "prometheus-blackbox-exporter"
                            }
                        }
                    }
                }
            }
        }
    }
}

For most values, check the section above (Method with #CaascadServicemonitorBlackboxexporterHelmConfig).

Values specific to #CaascadServicemonitorHelmConfig :

  • helm.values.servicemonitorMap...spec.endpoints :
    • .relabelings : at the end of the array, you need to specify the cc_prom_source label (targetLabel) and its value (replacement). The possible values are specified in the nomenclature Caascad. Note : do not set it in client zones like corp.
  • helm.values.servicemonitorMap...spec.jobLabel : don't change it. This is a label defined in the Blackbox-Exporter service. You should not set another value.
  • helm.values.servicemonitorMap...spec.selector.matchLabels : don't change it. This is the labels selector to target the Blackbox-Exporter service. You should not set another value.

Deploying#

Deploy as usual with Trackbone.

Add ruleset in applications/caascad-prometheus-rules in order to alert in case the metrics of blackbox servicemonitors are missing or have a bad configuration.

Adding a new serviceMonitor integrated in an application for Blackbox-Exporter#

Copy serviceMonitor template#

Create a new serviceMonitor from this file :

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/instance: blackbox-exporter
    caascad.com/prometheus-monitor: caascad
  name: blackbox-exporter-OVERRIDEME # Exemple blackbox-exporter-Grafana
spec:
  endpoints:
  - interval: 60s
    params:
      module:
      - OVERRIDEME # Exemple http_2xx
      target:
      - OVERRIDEME # Exemple https://grafana.ZONE_NAME.caascad.com
    path: /probe
    port: http
    relabelings:
    - sourceLabels:
      - __param_target
      - __param_module
      separator: /
      targetLabel: target
    - sourceLabels:
      - __param_target
      - __param_module
      separator: /
      targetLabel: instance
    - sourceLabels:
      - __param_module
      targetLabel: module
    - targetLabel: "cc_prom_source" # do not set cc_prom_source in
      replacement: "cloud-caascad"  # client zones like corp
    metricRelabelings:
     - regex: 'pod'
       action: labeldrop
    scheme: http
    scrapeTimeout: 60s
  jobLabel: blackbox-exporter
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app.kubernetes.io/instance: blackbox-exporter
      app.kubernetes.io/name: prometheus-blackbox-exporter

Caution

The .spec.endpoints[].params.target[] is an array. However you can set only one URL there. Any other URL will be ignored.

You must edit the file and replace OVERRIDEME value inside the file.

Testing the new serviceMonitor#

Set the namespace and run:

NAMESPACE=<OVERRIDEME>
kubectl apply -n ${NAMESPACE} -f servicemonitor.yaml

After few minutes, there should be metrics available. The Grafana dashboard Blackbox Exporter will update automatically and show your new target.

  • Datasource=Thanos
  • cluster=cloud-caascad
  • target=all (or your target)
  • modules=all (or the module your configured)

Check the status#

Checking the status is the first step for debugging when you don't have metrics.

NAMESPACE=<OVERRIDEME>

# Check blackbox is running correctly (in client zones, the namespace is caascad-monitoring)
kubectl get pod -n monitoring | grep blackbox

# Check you found your new serviceMonitor
kubectl get servicemonitor -n ${NAMESPACE} | grep blackbox

# Check your serviceMonitor is correctly set.
kubectl describe servicemonitor -n ${NAMESPACE} <YOUR SERVICE MONITOR>

Deploying the new serviceMonitor#

Insert your serviceMonitor yaml file in your helm chart or in your deployment chain.

Add ruleset in applications/caascad-prometheus-rules in order to alert in case the metrics of blackbox servicemonitors are missing or have a bad configuration.

Alerting rules#

Applicative alerting rules#

There are 2 kinds of alerting rules for Blackbox-Exporter

  • generic rules for all targets : BlackboxHTTPDown and BlackboxHttpSlowProbe
  • specific rules for some Blackbox-Exporter targets (example : BastionUnreachableICMP)

In most cases, generic rules will fit your needs.

If you have a specific need, you can create a new PrometheusRules with the usual workflow. There is nothing specific to Blackbox-Exporter here.

Note

When you need a specific alerting rule, and because it is specific to your serviceMonitor, you should NOT add the rule in the blackbox-exporter.cue file. Either create a new file or use the file dedicated to your application.

Sometimes you need to use generic rule with a different severity for you component. You can add cc_severity or cc_component in your servicemonitor.

How to use it :

  • cc_component : create a new specific rule for you component and filter on the label (e.g. ThanosQuerierHTTPDown)
  • cc_severity : duplicate your alert and filter on this label, adapt your severity.

Monitoring of monitoring alerting rules#

Blackbox-exporter targets are supervised using servicemonitors.

For all the definitions of Blackbox-exporter servicemonitors, an alert must be created in case the servicemonitor is missing or badly configured.

The alert definitions for Blackbox-exporter servicemonitors have been facilitated in applications/caascad-prometheus-rules.

Add a block in _BlackboxExporterAppServiceMonitorRulesets in caascad-blackbox-exporter-servicemonitors.cue and then follow the usual workflow to deploy prometheus rules.

List all Blackbox-exporter servicemonitor targets#

The servicemonitor targets are different from one type of environement to another.

Blackbox-exporter servicemonitor targets example on infra-stg:

kubectl get servicemonitor -A -o json | jq -r '.items[] | select(.metadata.labels["caascad.com/prometheus-monitor"] == "caascad") | select (.spec.selector.matchLabels["app.kubernetes.io/instance"] == "blackbox-exporter") | select (.spec.selector.matchLabels["app.kubernetes.io/name"] == "prometheus-blackbox-exporter") | select(.spec.endpoints[].path == "/probe") | .spec.endpoints[].params | .target + .module |join("/")'https://rancher.infra-stg.caascad.com/http_2xx_ssl
keycloak.infra-stg.caascad.com/auth/realms/infra-stg/account/http_2xx_ssl
bst.infra-stg.caascad.com/icmp_check
https://grafana-infra.infra-stg.caascad.com/http_2xx
https://jira.corp.caascad.com/http_2xx
https://karma-infra.infra-stg.caascad.com/http_2xx
https://vault.infra-stg.caascad.com/http_2xx_ssl