Aller au contenu

Generate new secret for Prometheus ingress#

Important

This procedure applies only for products for which Prometheus Remote Write has been implemented.

Prometheus customer (zone svc-monitoring-stack-client-XXXX)#

CLIENT=xxxx       # obs-pf : CLIENT=pf

In order to generate a new ingress password, follow the next steps:

  1. Create a new secret and push it into Vault by redeploying Central Prometheus with bootstrap=true option:

    trackbone apply -c kube-prometheus-stack -z svc-monitoring-stack-client-${CLIENT} -t bootstrap=true

    Warning

    At this time, the two secrets coexist (this avoids the interruption of service).

  2. Communicate the new password to the client

  3. Update PROMETHEUS_PASSWORD and PROMETHEUS_USER in the External Gitlab > Settings > CI/CD > Variables

    xdg-open "https://sourcehub.orange-business.com/cs-factory/cs-factory-environments/obs-${CLIENT}/-/settings/ci_cd"

    Important

    Wait for customer feedback to ensure that they are using the new passwords correctly before continuing.

  4. Clean-up. After making sure that the new secret works and the customer is aware of it, you can delete the old secret in Vault. The secret is stored in the following path: secrets/secret/zones/fe/<service_zone_name>/prometheus-ingress/prometheus-ingress-YYYYMMDD-HHMMSS, where YYYYMMDD-HHMMSS indicates the date and hour when the secret has been generated.

  5. Redeploy the client Prometheus so it can take into consideration the removal of the old password:

    trackbone apply -c kube-prometheus-stack -z svc-monitoring-stack-client-${CLIENT}

  6. Redeploy Blackbox-Exporter and prometheus-rules, because it uses Prometheus secrets to monitor it:

    trackbone apply -z svc-monitoring-stack-client-"${CLIENT}" -c blackbox-exporter-probe-monitoring-stack-client-mon3 --add-services
trackbone apply -z svc-monitoring-stack-client-"${CLIENT}" -c blackbox-exporter-probe-monitoring-stack-client-mon4 --add-services
trackbone apply -z svc-monitoring-stack-corp-prd-1 -c prometheus-rules -c blackbox-exporter-core -t blackbox_exporter_refresh_cache=true --add-services
trackbone apply -z svc-monitoring-stack-corp-prd-2 -c prometheus-rules -c blackbox-exporter-core --add-services

    trackbone apply -z svc-monitoring-stack-client-"${CLIENT}" -c blackbox-exporter-probe-monitoring-stack-client-mon3 --add-services
trackbone apply -z svc-monitoring-stack-client-"${CLIENT}" -c blackbox-exporter-probe-monitoring-stack-client-mon4 --add-services
trackbone apply -z svc-monitoring-stack-corp-stg-1 -c prometheus-rules -c blackbox-exporter-core -t blackbox_exporter_refresh_cache=true --add-services
trackbone apply -z svc-monitoring-stack-corp-stg-2 -c prometheus-rules -c blackbox-exporter-core --add-services

Thanos stack#

mon3/mon4 (Thanos-Receive)#

This is not Prometheus. Follow the Thanos documentation.