Collect metrics from new applications#

This documentation applies on the following environements without restrictions:

Caascad: infra and cloud zones
NGOT: cluster and service "central" zones

On client zones, if the application is a Caascad application, it also applies.

Important

If the application is a client application, you should refer to the documentation exposed to our clients.

Requirements#

The application should expose the /metrics path in HTTP. You can check it using port-forward on your pod/service.

Exporter port#

If you have the possibility of choosing the port of the exporter, any port should fit. However, some ports are reserved. See the full list here.

Recommandations:

If you find your exporter in the port list, prefer to use that port number.
Otherwise, avoid the 9000-9999 port interval.

Scraping metrics#

ServiceMonitor vs PodMonitor#

Usually we have a pod with its associated service, then ServiceMonitor should be the default choice but, in some case, you may want to scrape metrics of a set of pods which all have a certain label which is not consistent between different services.

We will only discuss about ServiceMonitor here, PodMonitors configuration is very close.

Kubernetes labels#

Prometheus-operator will scan for ServiceMonitors and PodMonitors that have the following labels set with indicated specific values.

Caascad: caascad.com/prometheus-monitor
NGOT: cloudservicesfactory/managed-by.

More information about monitoring labels:

Caascad
NGOT.

Upstream information:

ServiceMonitor with Helm#

Pre-requisites#

This template needs a _helpers.tpl file generated from helm create <YOURCHART>.
A service already exists and is defined with labels like {{- include "YOURCHART.labels" . | nindent 4 }}
The metrics port name is metrics. Actually you can use any name but in the serviceMonitor, the .spec.endpoints[].port must match that name.
The exporter metrics endpoint path is /metrics. If you need anoter path, change it in .spec.endpoints[].path.

File `template/servicemonitor.yaml`#

Replace YOURCHART with the name of your chart as defined in Chart.yaml (the included helpers are defined in _helpers.tpl).

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ include "YOURCHART.fullname" . }}
  labels:
{{- include "YOURCHART.labels" . | nindent 4 }}
{{- if .Values.serviceMonitor.labels}}
{{ toYaml .Values.serviceMonitor.labels | indent 4 }}
{{- end }}
{{- if .Values.serviceMonitor.annotations }}
  annotations:
{{ toYaml .Values.serviceMonitor.annotations | indent 4 }}
{{- end }}
spec:
{{- if .Values.serviceMonitor.jobLabel }}
  jobLabel: {{ .Values.serviceMonitor.jobLabel }}
{{- end }}
  endpoints:
  - port: metrics
{{- if .Values.serviceMonitor.scheme }}
    interval: {{ .Values.serviceMonitor.scheme }}
{{- end }}
{{- if .Values.serviceMonitor.interval }}
    interval: {{ .Values.serviceMonitor.interval }}
{{- end }}
{{- if .Values.serviceMonitor.timeout }}
    scrapeTimeout: {{ .Values.serviceMonitor.timeout }}
{{- end }}
{{- if .Values.serviceMonitor.proxyUrl }}
    scrapeTimeout: {{ .Values.serviceMonitor.proxyUrl }}
{{- end }}
    path: /metrics
{{- if .Values.serviceMonitor.relabelings }}
    relabelings:
{{ toYaml .Values.serviceMonitor.relabelings | indent 6 }}
{{- end }}
{{- if .Values.serviceMonitor.metricRelabelings }}
    metricRelabelings:
{{ toYaml .Values.serviceMonitor.metricRelabelings | indent 6 }}
{{- end }}
  namespaceSelector:
    matchNames:
    - {{ .Release.Namespace }}
  selector:
    matchLabels:
      {{- include "YOURCHART.selectorLabels" . | nindent 6 }}

Warning

port :"" is the port name. It will not work with a port number.

File `values.yaml`#

serviceMonitor:
  labels: {}
  annotations: {}
  scheme: ""
  interval: ""
  timeout: ""
  proxyUrl: ""
  jobLabel: ""
  relabelings: []
  metricRelabelings: []

The values can be set when deploying the helm chart.

Feel free to adapt these files to your needs. Examples:

Deploy more than one serviceMonitor. In this case, pay attention to the name of the serviceMonitor (for example, add a suffix)
Re-use an existing service instead of deploying a specific one. In this case, pay attention to the selector.matchLabels. You may hard-code the labels of the existing service instead of compute them with include.

Warning

interval is the scraping interval. You should not specify it but use the default value. Set it if you know what you are doing.

ServiceMonitor result#

This configuration will deploy the following serviceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ""                                    # ServiceMonitor name 
  namespace: ""                               # Namespace target
  labels:
    ## Mandatory label        ##
    ## For Caascad            ##
    caascad.com/prometheus-monitor: ""
    ## For NGOT               ##
    cloudservicesfactory/managed-by: ""
    ## End of mandatory label ##
spec:
  endpoints:
  - port: ""                                  # Service port name 
    scheme: "http"                            # Optional, "http" by default.
    relabelings:                              # Optional, add or replace label to the scraped metrics
    - replacement: ""
      targetLabel: ""
    interval: 15s                             # Optional, scraping interval
  jobLabel:                                   # Optional, name of the job in Prometheus, generated from the specified label of the service
  namespaceSelector:                          # Select which namespaces the Endpoints objects are discovered from
    matchNames:                               #       in most cases, this is the same namespace as the service and the exporter
    - ""
  selector:                                   # Select by labels which service are scraped
    matchLabels:
      ""

Relabeling#

Caascad relabeling#

In order to indicate which prometheus the metric comes from, we add the label cc_prom_source with different value by kind of cluster.

relabelings:
  - targetLabel: cc_prom_source
    replacement: ""

Infra clusters:

replacement value must be one of:

infra-caascad
infra-consumption

Cloud clusters:

replacement value must be one of:

cloud-caascad
cloud-client
cloud-app

Client clusters:

The label cc_prom and cc_prom_source are automatically set. You don't need to set this label by yourself.

However, because Prometheus will not federate all client serviceMonitors, you have to specify it with this label: caascad_com_prometheus_monitor_scope=caascad:

relabelings:
  - targetLabel: caascad_com_prometheus_monitor_scope
    replacement: caascad

NGOT#

In order to simplify serviceMonitor configurations by kind of zone, default values for relabelings and labelSelector are defined in envs-ng/contexts/schemas.cue.

Copy/paste the code below for your serviceMonitor relabelings and labelSelector.

Example:

envs: [string]: {
    zone: _
    configurations: ["<your config>"]: #HelmConfig & {
        helm: {
            values: {
                serviceMonitor: {
                    // relabelings
                    _relabelings: #ServiceMonitorNgotRelabelings[zone.type] & {"\(zone.name)": _}
                    relabelings:  _relabelings[zone.name]

                    // labelSelector
                    labels:       #ServiceMonitorNgotLabels
                }
            }
        }
    }
}

Warning

Keys may be different from the example.

Adding another labels to metrics#

You can add another key=value in the same way as above. Reference relabel config.

Deploy the ServiceMonitor#

Just deploy your ServiceMonitor with helm or Trackbone (as usual). Prometheus-operator runs an automatic discovery of ServiceMonitors and PodMonitor and will detect it soon. Then it will get the informations it needs from the ServiceMonitor or PodMonitor. It will get more information from the Service or Pod you selected with your monitor definition. And it will regenerate the configuration of Prometheus. All of this is automatic.

Disable scraping metrics#

Delete the ServiceMonitor/PodMonitors related to the Service/Pods you do not want to monitor any more. kubectl delete will do it. If you want to disable it permanently, remove it from the helm chart and redeploy.

Prometheus will automatically reload its configuration.

Troubleshooting#

Get the prometheus configuration file#

You can fetch the prometheus configuration file from the kubernetes CLI:

kubectl get -n <prometheus_ns> secret <prometheus_secret> -o jsonpath="{['.data']['prometheus\.yaml\.gz']}" | base64 -d | gunzip

Where <prometheus_secret> naming schema matches prometheus-.*-prometheus.

If you configure your ServiceMonitor/PodMonitor, Prometheus-operator should modify this file almost instantly. Prometheus do reload its configuration file but it's not as fast as the configuration modification and it can take up to few minutes.

Verify prometheus Target (targetDown problem)#

Use Rancher (Caascad only) or port-forward on Prometheus and go to /targets in order to verify if the endpoint target is up:

kubectl port-forward svc/<prometheus_service> -n <prometheus_ns> 9090:9090 &

When the target is down, here are some verifications you should do.

Is the exporter exporting metrics ?#

With Rancher (Caascad only) or port-forward on Prometheus to connect directly to the pod using your browser or use curl to check if the component exposes any metrics.

Example for node-exporter:
```
kubectl port-forward <caascad|ngot>-node-exporter-rj6c8 49100:49100
curl -sL localhost:49100/metrics
```
This should show all the metrics of the exporter.
With a port-forward do the same with the service.

Example for node-exporter:
```
kubectl -n monitoring port-forward svc/<caascad|ngot>-node-exporter 49100:49100
curl -sL localhost:49100/metrics
```
This should show again all the metrics of the exporter.

Control the serviceMonitor#

Get the definition of the serviceMonitor:

kubectl get servicemonitor <your serviceMonitor> -o yaml

Exemples:

Caascad node-exporter serviceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
...
  labels:
    caascad.com/prometheus-monitor: caascad
    ...
spec:
  endpoints:
  - port: metrics
    relabelings:
    - replacement: cloud-caascad
      targetLabel: cc_prom_source
  jobLabel: jobLabel
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app: prometheus-node-exporter
      release: caascad-node-exporter

NGOT node-exporter serviceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: node-exporter
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2023-04-04T09:25:47Z"
  generation: 1
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: node-exporter
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus-node-exporter
    app.kubernetes.io/part-of: prometheus-node-exporter
    app.kubernetes.io/version: 1.5.0
    cloudservicesfactory/managed-by: corp-cluster
    helm.sh/chart: prometheus-node-exporter-4.14.0
    jobLabel: node-exporter
  name: node-exporter
  namespace: monitoring
  resourceVersion: "60739402"
  uid: 1b53c4d7-16c0-4340-bae3-bedbd6e58b85
spec:
  endpoints:
  - port: metrics
    relabelings:
    - action: replace
      replacement: pf
      targetLabel: obs_client
    scheme: http
    scrapeTimeout: 10s
  jobLabel: jobLabel
  selector:
    matchLabels:
      app.kubernetes.io/instance: node-exporter
      app.kubernetes.io/name: prometheus-node-exporter

Control that you added the correct label. Without this label, Prometheus-operator will not detect it and Prometheus will never know about it.
- Caascad: caascad.com/prometheus-monitor: caascad
- NGOT:
  - cluster zone type: cloudservicesfactory/managed-by: corp-cluster
  - service zone type: cloudservicesfactory/managed-by: corp-central
Find the service by copy/pasting the namespace and selector. Here is how to construct the command: kubectl -n <namespace> get svc -l label1=value1,label2=value2,...

For node-exporter:
- get <namespace> from spec.namespaceSelector.matchNames
- get the labels from spec.selector.matchLabels
The command becomes:
```
kubectl -n monitoring get svc -l app=prometheus-node-exporter,release=<caascad|ngot>-node-exporter
```
If you found the service, the serviceMonitor should be correct.
When you don't find the service, there are some rare cases where there is an endpoints but no service. Try the above command and replace svc with endpoints:
```
kubectl -n monitoring get endoints -l app=prometheus-node-exporter,release=<caascad|ngot>-node-exporter
```
Control the port defined in spec.endpoints: the port should be a string and should be the same as the port defined in the service.
Control the path. The default value is /metrics and you should not see it in the serviceMonitor. If your exporter exports metrics somewhere else than /metrics you should specify it with spec.endpoints[].path.

Prometheus behaviour#

Prometheus pod does not reload its configuration immediately. You may wait up to 1 or 2 minutes before its configuration includes your serviceMonitor definition and before it reloads it.

You may check the logs of the containers of the pods of

Prometheus (prometheus and prometheus-config-reloader)
Prometheus-operator

The logs are usually useless for debugging serviceMonitors, but it can show other problems that impact your serviceMonitor. When this happens, first fix the other problems, then go back on your serviceMonitor.

In some very rare cases, Prometheus was running bad and killing (kubectl delete pod) the pod fixed it. Such a behaviour is hard to detect.

Identify the correct Grafana#

First you need to ensure you check on the correct Grafana.

Caascad Grafana types#

On Caascad depending on the zone type, the following Grafana instances are used to check applications metrics:

Grafana infra: grafana-infra.infra (example zone staging)
Grafana cloud: grafana-infra.xxx (example zone cloud)
Grafana client: grafana.xxx (example zone client).

NGOT Grafana types#

On NGOT depending on the zone type, the following Grafana instances are used to check applications metrics:

Grafana for Product Factory team: example staging
Grafana for clients: example client.

Ensure you are checking on the grafana and datasource connected to the Prometheus that collects your metrics. Hint: you can first check on the Prometheus (connect with port-forward).

Duplicate sample for timestamp#

Check the logs and try to find out what serviceMonitor sends duplicate metrics.

Investigate on that serviceMonitor:

check if there is not already another serviceMonitor to scrap the same metrics exporter

Collect metrics from new applications#

Requirements#

Exporter port#

Scraping metrics#

ServiceMonitor vs PodMonitor#

Kubernetes labels#

ServiceMonitor with Helm#

Pre-requisites#

File template/servicemonitor.yaml#

File values.yaml#

ServiceMonitor result#

Relabeling#

Caascad relabeling#

NGOT#

Adding another labels to metrics#

Deploy the ServiceMonitor#

Disable scraping metrics#

Troubleshooting#

Get the prometheus configuration file#

Verify prometheus Target (targetDown problem)#

Is the exporter exporting metrics ?#

Control the serviceMonitor#

Prometheus behaviour#

Identify the correct Grafana#

Caascad Grafana types#

NGOT Grafana types#

Duplicate sample for timestamp#

File `template/servicemonitor.yaml`#

File `values.yaml`#