Aller au contenu

Workflow for writing Prometheus Rules#

Actors :

  • Creator : the developer of the rule
  • Supervision : the OPS and Support teams

Workflow :

  1. (Developer) Create or update a PrometheusRule cue definition.
  2. (Developer) Create a Merge Request with the changes
  3. (Developer) Comes to Supervision DSM and present the new rule (including a screenshot of the rule firing and the Help_alerts documentation)
  4. (Supervision) Validate the deployment of the rule
  5. (Developer) Deploy the new rule (following the usual deployment methods with CAASCHR)

Warning

Do not merge your MR in caascad-prometheus-rules too soon.

When that MR is merged, it will block any new deployment of rules. Some may wait after you to merge their MR. Moreover, in case of an incident in Production, your MR could be rollbacked !

Suggestion : when the MR is validated, try to merge it, update the MR in envs-ng and deploy (CAASCHR/stg) the same day. And the day after (CAASCHR/prd) in Production.

Create or update PrometheusRules#

Follow the documentation.

PrometheusRules Documentation#

Screenshot your rule#

When your rule is ready to deploy, you can invert the test (== becomes != or < becomes >=) to force the alert to fire. Then you can take a screenshot.

You can then add the screenshot as a comment in the discussion of your Merge Request so it's easier for all to see how the alert looks like when you present it.

Document your alert in Help_alerts documentation#

The alert description document is located here

Best practices for your alert :

  • When you describe the alert, think as an Operator who has a problem to solve and not as a Developer who is proud to describe his work.
  • The alert expression (expr) is defined in the rule and can easily be found with kubectl get prometheusrules -A. You don't need to copy/paste it in the description.
  • The description needs at least troubleshooting hints to help the Operator understand where the problem is.
  • The description may suggest solutions that the Operator can apply. If the solutions are easy, writing them will be appreciated.
  • Some alerts show problems that have no easy solution. In this case, do not spend time to imagine a solution with no use case. When the Operator fix a problem, (s)he may improve the help_alerts.md with the real use case (s)he just fixed.
  • An Operator may not know the application where a problem appears. You may add a link to the documentation of the product (or it source code) and another link to a document that explain how it was deployed at Caascad.

Test your alert#

Follow the README.md file documentation to test your alert.

Note

  • first you do not need a specific zone to test. You can use the default testzoneXX.
  • then you can create your own namespace and deploy there (as described in the README.md file). There is no need to deploy in the prometheusrules namespace to test. It will work in any namespace.

Checklist before you submit your PrometheusRule#

  • Is your alert firing when the problem appears
  • Is your alert down when the problem is fixed (if not, explain why in the help_alerts.md)
  • Does your alert have a annotation.message ?
  • Is the annotation.message meaningful ? With all needed labels to investigate ?
    • Namespace of the exporter
    • Namespace of the monitored component (may differ of the exporter)
    • id or name of the failed component
    • severity
    • mandatory labels (see above)
  • Have you created/updated the alert document in Help_alerts ? Is the document meaningful ?
    • description
    • troubleshooting hints
    • solutions (if any)