PrometheusDuplicateTimestamps#

Prometheus scrape metrics twice, that explain the duplicate timestamps.

Known cases :

In most cases, the problem root cause is the presence of 2 ServiceMonitors that link to the same service. This causes a duplicate configuration, duplicate scaping and duplicate timestamps.
The investigation consists in finding the 2 ServiceMonitors and the solution is to remove one of them.
Another case is when 2 distinct exporters expose the same metric (example : kubelet has two distinct metric paths, one for itself, one for cadvisor, and some metrics are duplicate).
The investigation consists in finding the 2 ServiceMonitors, Services and exporter pods, then find a way to add a label on the metrics (2 metrics with distinct labels are considered as distinct).

Troubleshooting hints#

Identify which Prometheus has duplicate timestamps
- notice the namespace label in the alert (the pod does not matter : they usually all have the problem)
Set that Prometheus in debug mode
- kubectl -n <namespace> get prometheus returns the name of the prometheus configuration object
- kubectl -n <namespace> edit prometheus <prometheus configuration object name>
- Search for logLevel. Default value is info. Change this to debug.
- Watch the pods restart : kubectl -n <namespace> get pod |grep prometheus
Warning

Do not forget to set the logLevel back to info after debugging

Find the log lines about Duplicate sample for timestamp. Example of such a line :

level=debug ts=2021-05-21T08:07:39.051Z caller=scrape.go:1250 component="scrape manager" scrape_pool=monitoring-client/mom-caascad-grafana-client/0 target=http://172.16.0.81:3000/metrics msg="Duplicate sample for timestamp" series=go_memstats_mallocs_total

Note

If you search logs with Loki, this request {namespace="monitoring"} |~ "Duplicate" may help (change monitoring with the namespace you found in the alert)

Identify the scaping job in the log line, as scrape_pool value. In the example above, you would identify monitoring-client/mom-caascad-grafana-client/0
- monitoring-client may be the namespace. This is a clue, but you are not sure yet about this information
- mom-caascad-grafana-client may be a ServiceMonitor. This is also a clue, but you are not sure yet about this info either.
Ask for help if you can (fast investigation and fast resolution)
- Usually the clues above help to find some recent work on a component.
- You could find who is working on that component and ask her/him if that was a mistake that could be solved quickly
- The person usually knows what ServiceMonitor to remove and will remove it. Problem solved. End of the story
Warning

Do not forget to set the logLevel back to info after debugging
When asking for help could not help resolve the problem, go on debugging...
Retrieve the Prometheus configuration file
- either as kubectl -n <namespace> exec -it <pod_name> -c prometheus -- cat /etc/prometheus/config_out/prometheus.env.yaml > /tmp/prometheus_config.yaml
- or kubectl -n <namespace> port-forward svc/caascad-prometheus 9090:9090 then get the configuration on http://localhost:9090/config
Tip
CaascadNGOT/serviceNGOT/cluster
<pod_name>: prometheus-caascad-prometheus-0

<svc_name>: caascad-prometheus
<pod_name>: prometheus-obs-corp-stg-prometheus-0

<svc_name>: obs-corp-stg-prometheus
<pod_name>: notice the pod name label in the alert

<svc_name>: notice the service name label in the alert

In the configuration, find the job that was identified earlier. In the example of Caascad zone, we would get this :

...
- job_name: monitoring-client/mom-caascad-grafana-client/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring-client
  metrics_path: /metrics
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app_kubernetes_io_name
    regex: grafana
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: service
...

In the configuration, find another job that would link to the same ServiceMonitor. There is no deterministic way to find one, but here is some help to find it :
- source_labels as __meta_kubernetes_service_label_<label name> with action: keep gives Kubernetes labels information (see above regex: grafana as the value) on the Service
- source_labels as __meta_kubernetes_endpoint_port_name with action: keep gives Kubernetesinformation on the service port name (see above regex: service : the port is strangely named service)
- notice the namespace in the namespaces section
- metrics_path: /metrics will confirm that the other job points to the same metrics path
- With these informations, you could search for other occurrences of them in other jobs.
  - Searching for regex: grafana would return 2 other occurrences
  - one of them does not match because it comes from another namespace
  - the other one is in the job - job_name: monitoring/caascad-grafana/0
  - conclusion (as a new clue) : the other ServiceMonitor may be caascad-grafana in namespace monitoring.
Note

A ServiceMonitor may not be in the same namespace as its target. This is why it is not so easy to find the 2 duplicate ServiceMonitors
When you found another job with a similar configuration, you have clues about the two duplicate ServiceMonitors. Ensure with kubectl -n <namespace> get ServiceMonitor <ServiceMonitor1> -o yaml (and same with the other one) that you found the 2 duplicate ServiceMonitors.

Warning

Do not forget to set the logLevel back to info after debugging
If you definitely don't find 2 duplicate jobs, you can start thinking about the same metric exposed by 2 exporters.
- This case happened only once yet. It was at the beginning of the Caascad Story. If this happens again, this doc may be improved.
- When this case happened, we were working on a ServiceMonitor on the Kubelet exporter. We quickly noticed the problem, and there were not so many ServiceMonitors to investigate.
- We quicky identified that the problem was with the 2 metric paths of the Kubelet (e.g. 2 exporters inside the Kubelet). We exported the metrics of the 2 exporters and we noticed that some were duplicated.
- As a solution, we added a Prometheus label (with relabelings) on the metrics of the 2 exporters.

Solution#

When you know what ServiceMonitors are duplicate, find out why they were deployed.
- if that was a mistake, remove the one that should not exist
- if it is "normal" that the 2 are deployed, re-check the application architecture in order to deploy only one
Warning

Do not forget to set the logLevel back to info after debugging
When there are 2 exporters that expose the same metric, use the relabeling feature of the ServiceMonitors (see the Prometheus configuration documentation) to add distinct labels on each exporter metrics.

Note

2 similar metrics with same name but distinct labels are considered as distinct.