Aller au contenu

Upgrade "upstream" rules and dashboards#

Introduction#

Upgrading "upstream" rules and dashboards should be done as the same time because they depend on each other.

Rules on Caascad and NGOT have the same origin and are generated with the same wrapper scripts. However, the generated results will differ.

Caascad rules are generated in a unique helm chart in helm/upstream-rules-dashboards-caascad directory.

Note

03/05/2023 : Rules compatibility with multiple Kubernetes versions is a pending subject on Caascad.

NGOT rules are generated in helm charts that depend on the Kubernetes version. These charts are in helm/upstream-rules-dashboards-ngot-vX.XX directories.

Because upstream rules follow latest versions of kubelet, kube-state-metrics and node-exporter, and because there is no maintained history of rules, there is no way to maintain older rules with our scripts. For this reason, you may not upgrade older rules for older Kubernetes versions if newer are incompatible with older versions of kubelet, kube-state-metrics or node-exporter.

Upgrading rules#

Ensure being running in a clean environment#

➜ git clone git@git.corp.caascad.com:caascad/applications/caascad-prometheus-rules-dashboards-upstream.git
➜ cd caascad-prometheus-rules-dashboards-upstream
➜ git switch main
➜ git pull
➜ git status

Control that you are in a safe environment.

Then git switch -c <your working branch>.

Run the upgrade script#

➜ cd scripts
➜ ./generate_prometheus_rules.sh <caascad|ngot-vX.XX>
➜ ./generate_dashboards.sh

If the script fails, you will have to repair it now and run it until it works.

Note

23/09/2022 / Caascad : the script uses a "hack" script from https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/hack. The sync_prometheus_rules.py has been hacked by our teams (e.g. we hacked the hack script). The fork is too big to be able to resynchronize from upstream. The PF-977 user story should solve this problem. Before that, we just repair what we can how we can.

Control the changes#

➜ git status
➜ git diff

Note each changed rule in a draft.

Rules : Study the changes#

The draft may be organised in these sections :

  • Alerting rules
    • new rules
    • changed/renamed rules
    • removed rules
  • Recording rules
    • new rules
    • changed/renamed rules
    • removed rules

For each change :

  • understand what the change is and explain it
  • find out what would be the impact
  • check the impact on the staging and production environments

How to find which environment use which rule ?#

Adapt and run this script :

  • METRICS is the most important variable. Set the metrics you want to control there
  • OBJECTS should not be modified. cm is for Grafana dashboards and prometheusrules is for Prometheus Rules.
  • ZONES may be set another way. The current expression uses sd (from Caascad Toolbox) to get all infra and cloud zones (client zones do not have rules)
# METRICS : list of metrics to analyse on all zones
METRICS=(kubelet_runtime_operations_duration_seconds pod_preemption_metrics storage_operation_errors_total etcd_object_counts)

# OBJECTS : configmaps (dashboards) and prometheusrules
OBJECTS=(cm prometheusrules)

# ZONES : infra and cloud zones
ZONES=($(sd get zones | jq -r '. | to_entries[] | .value | select (.type != "client") | .name' | sort -u))

# Check the zones
echo "${ZONES[@]}"
infra-prd infra-stg ocb-sbma ... ocb-test06

# If you run kswitch, protect yourself against some KUBECONFIG already set from somewhere else
unset KUBECONFIG

# Go ! (result will go to "/tmp/r")
for zone in "${ZONES[@]}"; do 
  kswitch "${zone}"; 
  for metric in "${METRICS[@]}"; do 
    for object in "${OBJECTS[@]}"; do
      kubectl get "${object}" -A -o json | jq --arg zone "${zone}" --arg object "${object}" --arg metric "${metric}" -r '.items[] | select(. | tostring | contains($metric)) | @sh "metric:\($zone):\($metric):\($object):\(.metadata.namespace)/\(.metadata.name)" ';
    done;
  done;
done > /tmp/r

The result is a CSV (Colon, not Coma, Separated Values) list.

The first field is metric in order to make it easier to filter results with grep ^metric:.

Following fields include :

  • zone
  • metric (as wanted in the METRICS array)
  • Kubernetes type (as specified in the OBJECTS array)
  • object namespace and name

This may help to identify the impact of a rule change.

Dashboards : Study the changes#

The draft may be organised in these sections :

  • New dashboards
  • Updated dashboards
  • Removed dashboards

For each change :

  • understand what the change is and explain it
  • find out what would be the impact
  • check the impact on the staging and production environments
  • for each change on an expression, ensure that
    • the metric exists on our clusters
    • the recording rule exists (or will exist after the upgrade of rules above)

Commit the changes#

Note

23/09/2022 : running the script to generate rules will retrieve the latest rules. There is no way to retrieve the rules at a specific date. So running the script twice may generate distinct rules.

Commit the changes as usual : git commit -m <message> and git push.

The message should reference the current commit of the upstream repository.

Rules changes#

Some rules may need to be changed for any reason (rollback a rule to the older expression, add a label...).

On latest NGOT and on Caascad, when this is needed, don't update the rule directly. Create a patch script in the prometheusrules/prometheusrules-patches or dashboards/dashboards-patches directory of the repository. It will be automatically used at the next run of the generate_prometheus_rules.sh or generate_dashboards.sh scripts.

Iterate and re-run the generate_prometheus_rules.sh and generate_dashboards.sh scripts until the rules and dashboards are correct.

On older NGOT rules, prefer doing the same as latest rules (see above). But when it becomes too difficult to maintain with patches, it may be easier to edit the rules directly.

Then git commit and git push the new changes.

Update the hack script#

The hack scripts are synchronized from upstream.

Download them (from here and here).

The script for Caascad rules needs to be renamed as sync_prometheus_rules-caascad.py.

Check the differences (git diff).

If the new version requires a new library

  • update the kubernetes/hack/requirements.txt file,
  • deactivate the virtualenv,
  • update and relaunch the virtual env as described above.

Deploy on a test environment#

Deploy the rules and the dashboards.

In Grafana, check each dashboard :

  • control there are metrics
  • visualize the changes identified above
  • take screenshots that will be used later when listing the changes

Tip

You may take screenshots in grafana-infra (caascad Grafana) with Thanos-client datasource and in grafana (client Grafana) with Thanos datasource. Select the same cluster. With dashboards updated on one Grafana and not on the other you can easily see the difference and screenshot them.

Finish the update#

When the scripts are run, when the differences are explained and when git contains the latest rules and dashboards, the update can be finished :

  • if there are new Alerting Rules or significative changes, present them to the Supervision Team
  • deploy in all staging environments
  • deploy in all production environments
  • rewrite the draft of the changes in a safe place
  • present the significative changes to the Monitoring Team