Aller au contenu

Extract raw samples from Prometheus#

Why do you need to extract raw sample from Prometheus ?#

The information visualised in Grafana and Prometheus can be distorted because it works by evaluating PromQL at every step, so it can miss samples if there are more than one sample between steps.

The resulting output will also have the timestamp of the evaluation step rather than the sample.

Sometimes you may need accurate information, and you can access it via the remote read endpoint at api/v1/read.

How to extract raw samples from Prometheus ?#

  1. Connection towards Prometheus

This step is necessary in order to be able to query directly the Prometheus API to extract raw information.

You can use one of the following 2 ways:

  • by passing by Rancher

  • by making a port forward on svc/<prometheus_svc> 9090:9090

  1. Encode the query

Encoding function: 

urlencode() {
  # urlencode <string>

  LC_COLLATE=C

  local length="${#1}"
  for (( i = 0; i < length; i++ )); do
    local c="${1:$i:1}"
    case $c in
      [a-zA-Z0-9.~_-]) printf '%s' "$c" ;;
      *) printf '%%%02X' "'$c" ;;
    esac
  done
}
For the sake of example, we will get the raw information for the following query for an interval of 5 minutes. The query and time interval will depend on your troubleshooting needs.

Metric used in this example: 

container_memory_usage_bytes{cc_prom_source="charlie"}
Create the PromQL query: 
container_memory_usage_bytes{cc_prom_source="charlie"}[5m]

Note

We need to append [<duration>] to the metric in order to get the raw values in the last <duration> interval.

Encode the PromQL query: 

urlencode 'container_memory_usage_bytes{cc_prom_source="charlie"}[5m]'

The output will have the following format: 

container_memory_usage_bytes%7Bcc_prom_source%3D%22charlie%22%7D%5B5m%5D

  1. Query prometheus by using the encoded url obtained at the previous step

Query format: 

curl -G  "http://localhost:9090/api/v1/query?query=<encoded_url>" | jq .
Example: 
curl -G  "http://localhost:9090/api/v1/query?query=container_memory_usage_bytes%7Bcc_prom_source%3D%22charlie%22%7D%5B5m%5D" | jq .
The output will return the last 5 minutes raw data.

How to interpret raw values#

The output provides the raw samples contained by Prometheus in Unix Timestamp and their corresponding values.

Output example: 

          [
            1631548434.047,
            "37523456"
          ],
          [
            1631548446.026,
            "37523456"
          ],
          [
            1631548461.36,
            "37523456"
          ]
Output interpretation:

Unix Timestamp Date Metric value
1631548434.047 13/09/2021 at 15:53:54 37523456
1631548446.026 13/09/2021 at 15:54:06 37523456
1631548461.36 13/09/2021 at 15:54:21 37523456

In order to convert Unix Timestamp to date, use any online converters or one of the following commands:

TZ=UTC date --date='@1631548434' +'%d/%m/%Y at %H:%M:%S'
13/09/2021 at 15:53:54