Custom metric for customer

Custom metric management

Two types of requests :

• creation of a custom metric
• modification of an existing custom metric

Creation of a custom metric

Create a custom metric by following manage custom metrics in promtail with trackbone documentation.

It's necessary that :

• the name of the stage starts with tmp_
• the selector is {filename=~\"/var/log/pods/.*prometheus.*\"}

Modification of an existing custom metric

Step 1

Retrieve the existing custom metric.

Example :

client_dcclight_server_error: {
                match: {
                        selector: "{app=\"dcclight-server\"}"
                        stages: [
                                {
                                        json: expressions: log: null
                                },
                                {
                                        json: {
                                                source: "log"
                                                expressions: {
                                                        level: null
                                                }
                                        }
                                },
                                {
                                        labels: {
                                                level: ""
                                        }
                                },
                                {
                                        metrics: warning_log: {
                                                description: "Dcclight application warning log report"
                                                source:      "level"
                                                type:        "Counter"
                                                config: {
                                                        action: "inc"
                                                        value:  "WARN"
                                                }
                                                prefix: "dcclight_"
                                        }
                                },
                        ]
                }
        }

Step 2

Create a new custom metric from the existing one.

It's necessary :

• modify the name of the stage, by adding a tmp prefix
• modify the selector, by putting {filename=~\"/var/log/pods/.*prometheus.*\"}
• add the change requested by the customer

Exemple :

tmp_client_dcclight_server_error: {
                match: {
                        selector: "{filename=~\"/var/log/pods/.*prometheus.*\"}"
                        stages: [
                                {
                                        json: expressions: log: null
                                },
                                {
                                        json: {
                                                source: "log"
                                                expressions: {
                                                        level: null
                                                        message: null
                                                }
                                        }
                                },
                                {
                                        labels: {
                                                level: ""
                                                message: ""
                                        }
                                },
                                {
                                        metrics: warning_log: {
                                                description: "Dcclight application warning log report"
                                                source:      "level"
                                                type:        "Counter"
                                                config: {
                                                        action: "inc"
                                                        value:  "WARN"
                                                }
                                                prefix: "dcclight_"
                                        }
                                },
                        ]
                }
        }

In this example, the client request was to add the message label.

Step 3

Specify this new metric in the list of custom metrics in the promtail.cue file :

snippets: _pipelineStages: *[
    ...
    prom.stages.tmp_client_dcclight_server_error,
] | [...]

Warning

The list is different for infra, cloud and client zones.

Test on a staging zone

Step 1

Deploy Promtail on the desired staging zone.

Step 2

Write a line of logs on one of the nodes where a Prometheus pod is scheduled.

To do this, follow the create_log_for_pod documentation with :

• source_pod_name=<customer_application>
• destination_pod_name=prometheus

Step 3

Go to Grafana and check that the metric is present.

Final preparation

Step 1

For the new custom metric :

• remove the tmp prefix in the name of the stage
• modify the selector to put the client's application

If it was a metric modification, you must also delete the old stage.

Step 2

Modify the list of custom metrics in the promtail.cue file.

Two cases :

• metric creation
• metric modification

If it was a metric creation, add a condition to deploy only for the desired zone.

Exemple :

snippets: _pipelineStages: *[
        ...
        if zone.name == "inttac" {prom.stages.client_dcclight_server_error},
] | [...]

And if it was a metric modification, remove the stage starting with tmp.

Step 3

Prepare for deployment as usual.