Create a custom metrics from logs in Promtail#
Why do you need to create a custom metric ?#
You may need to create a custom metric for:
- monitor/graph an application using information not present in the application metrics
- create a custom alert from the logs in Prometheus Rules
How to create a metric ?#
Use-case#
We need to be alerted when there's duplicated dashboard in Grafana. Based on the following Grafana's log, we have to create the metrics in Promtail that will be used for the Prometheus Rules.
t=2021-04-29T08:21:34+0000 lvl=eror msg="the same 'title' is used more than once" logger=provisioning.dashboard type=file name=Kubernetes title="Kubernetes / Networking / Pod" provider=Kubernetes
t=2021-04-29T08:21:34+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=Kubernetes uid="frKUKDfd" provider=Kubernetes
For these log lines, the useful information is the msg/title and msg/uid pairs.
It is then necessary to create two different metrics from these logs, grafana duplicated_title_errors and grafana_duplicated_uid_errors. These metrics will have to be incremented at each log line corresponding to the messages:
"the same 'title' is used more than once"forgrafana duplicated_title_errors"the same 'uid' is used more than once"forgrafana_duplicated_uid_errors
How it has to be implemented in Promtail configuration#
The configuration of Promtail is now carried by a secret. The metrics are to be declared in the scrape_configs map of the secret. For each job, new pipeline steps will be created, as follows (You can refer to the official documentation here.).
This is a quick description of the field:
selector: This is the request to select logs file.stage.regex.expression: Regex for filtering the logs filelabels(Optional): Labels that will be add to the metric. Useful to bring informations to alertmanager (Optional)metrics.<metric_name>.config.action: What the metric will do when a filtered line log appearmetrics.<metric_name>.description: Description of the metricmetrics.<metric_name>.source: Source of the metric #FIXME : c'est pas très clair, il faudrait peut-être préciser que ça correspond à la capture dans la regexmetrics.<metric_name>.type: Type of the metricmetrics.<metric_name>.prefix: Prefix of the metric
- match:
selector: '{job=~"monitoring.*/grafana"}'
stages:
- regex:
expression: ^.*(?P<msg_uid>the same 'uid' is used more than once).*uid=(?P<uid>[a-zA-Z0-9]+).*$
- labels:
msg_uid: ""
uid: ""
- metrics:
duplicated_uid_errors:
config:
action: inc
description: count duplicated dashboard UID errors
source: msg_uid
type: Counter
prefix: grafana_
- match:
selector: '{job=~"monitoring.*/grafana"}'
stages:
- regex:
expression: ^.*(?P<msg_title>the same 'title' is used more than once).*title=\\"(?P<title>.*)\\".*$
- labels:
msg_title: ""
uid: ""
- metrics:
duplicated_title_errors:
config:
action: inc
description: count duplicated dashboard title errors
source: msg_title
type: Counter
prefix: grafana_
Using Trackbone to deploy#
Trackbone will be used to deploy a new custom metric. It could be deployed globally or just or specific environments. All the declaration of custom metrics will be added into the file promtail/custom_metrics.cue of the caascad/terraform/envs-ng repository. The list of custom metrics by environment will be managed into the file promtail.cue of the caascad/terraform/envs-ng repository.
As prerequisite, create a new branch from master to work on.
Declare your metric first#
To deploy a new custom metric, you'll need to declare it into stages struct first.
Example:
stages: {
grafana_duplicate_dashboards_errors: {
match: {
selector: "{job=~\"monitoring.*/grafana\"}"
stages: [
{
regex: expression: "^.*(?P<msg>the same 'title' is used more than once|the same 'uid' is used more than once).*$"
},
{
metrics: duplicate_dashboards_errors: {
config: action: "inc"
description: "count duplicate dashboard errors"
source: "msg"
type: "Counter"
prefix: "grafana_"
}
},
]
}
}
grafana_duplicated_uid_errors: {
match: {
selector: "{job=~\"monitoring.*/grafana\"}"
stages: [
{
regex: expression: "^.*(?P<msg_uid>the same 'uid' is used more than once).*uid=(?P<uid>[a-zA-Z0-9]+).*$"
},
{
labels: {
msg_uid: ""
uid: ""
}
},
{
metrics: duplicated_uid_errors: {
config: action: "inc"
description: "count duplicated uid errors in logs of grafana"
source: "msg_uid"
type: "Counter"
prefix: "grafana_"
}
},
]
}
}
Then you want to setup your custom metric in the main configuration (promtail.cue). The place where you add it depends on where the logs come from.
- logs coming from files (including Kubernetes components, containers...)
- if the logs are specific to a cloud provider, add the custom metric to
_promtailExtraScrapeConfigMap[<zone>-<cloud provider>].extraScrapeConfigs.customMetricspipelineStages. See example files/specific to a CP below. - if the logs are generic, add the custom metric to
...helm.values.promtail.config.snipets._pipelineStages. See example files/generic below.
- if the logs are specific to a cloud provider, add the custom metric to
-
logs coming from loki_push_api : add the custom metric to
_promtailExtraScrapeConfigMap[<zone>-<cloud provider>].extraScrapeConfigsPushAPI.customMetricspipelineStages. See example loki_push_api below.Example with the previous
grafana_duplicated_uid_errorsandgrafana_duplicated_title_errorson infra and cloud zone. Notice you can also deploy on client zone.envs: [string]: { zone: _ configurations: ["promtail"]: { helm: ... if zone.type == "infra" || zone.type == "cloud" { values: promtail: { config: { snippets: _pipelineStages: *[ // Place you custom metrics here for Zone Infra or Zone Cloud // Specify env targeted with `if zone.metadata.line == <env> {<custom_metrics>}` ... prom.stages.grafana_duplicated_uid_errors, prom.stages.grafana_duplicated_title_errors, ... ] | [...prom.#PipelineStageMatch] } } } if zone.type == "client" { values: promtail: { config: { snippets: _pipelineStages: *[ // Place you custom metrics here for Zone Client // Specify targeted env with `if zone.metadata.line == <env> {<custom_metrics>}` ... ] | _ } } } } }Example with a custom metric specific to the cloud zone on AWS :
_promtailExtraScrapeConfigMap: ["cloud-aws"]: { ingressPushAPI: true extraScrapeConfigsPushAPI: [...#extraScrapeConfigsFull] & [{ esc: prom.#PromtailESCPushAPIAWS customMetricspipelineStages: [] }] extraScrapeConfigs: [...#extraScrapeConfigsFull] & [{ esc: prom.#PromtailExtraScrapeConfigsKubeletAWS customMetricspipelineStages: [ // add the line here ] }] }Example with a custom metric for Loki Push API specific to the infra zone on FE :
_promtailExtraScrapeConfigMap: ["infra-fe"]: { ingressPushAPI: false extraScrapeConfigsPushAPI: [...#extraScrapeConfigsFull] & [{ esc: prom.#PromtailESCPushAPICaascadInternals customMetricspipelineStages: [ // add the line here ] }] extraScrapeConfigs: [...#extraScrapeConfigsFull] & [{ esc: prom.#PromtailExtraScrapeConfigsKubeletFE customMetricspipelineStages: [] }, { esc: prom.#PromtailExtraScrapeConfigsKubeProxyFE customMetricspipelineStages: [] }, { esc: prom.#PromtailExtraScrapeConfigsKubeMessagesFE customMetricspipelineStages: [] }] }
Deploying metric for a list of environments only#
Sometimes, it could help to quickly set up a new metrics to monitor or troubleshoot a specific environment.
For example, we want to have a counter that will increase each time there's a log error in promtail's log, on the ocb-test05 cluster.
First declare your metric as before in promtail/custom_metrics.cue.
Then in the end of the file promtail.cue you can add your custom metric as follows:
// ------------------------------
// SPECIFIC CONFIGURATION BY ENV
// ------------------------------
// OCB-TESTO4
envs: ["ocb-test05"]: {
zone: _
configurations: ["promtail"]: {
helm: values: {
//-----------------------------------------------
// CREATE METRICS FOR PROMTAIL ERRORS
//-----------------------------------------------
"promtail": {
config: snippets: _pipelineStagesByEnv: [
// Place your custom metrics here
...
]
}
}
}
}
Tip
You can also use this technique with the flavourmap _promtailExtraScrapeConfigMap.
Run Trackbone to deploy#
nix-shell
trackbone plan -c promtail -z <zone_name>
trackbone apply -c promtail -z <zone_name>
Useful trackbone command#
Visualize custom metrics configuration :
trackbone show -c promtail -z <zone> helm.values.promtail.config.snippets.pipelineStages