Thanos authentication

Thanos needs credentials to connect to its long-term storage. Because we use Prometheus-Operator for Thanos-sidecar (in the same pod as Prometheus), there are some differences between Thanos and Thanos-sidecar.

FE/AWS

Principle

The credentials are provided in a file with this format:

[default]
aws_access_key_id = <access key>
aws_secret_access_key = <secret key>

This file is noticed to Thanos or Thanos-sidecar with an environment variable: AWS_SHARED_CREDENTIALS_FILE.

The name of the bucket and the endpoint are not included in the file that contains the secret, they are defined as argument in command container.

Example for ocb-test06:

  args:
    - store # Thanos argument
    - ...
    - |
      --objstore.config=type: s3
      config:
         bucket: thanos-cloud-caascad-ocb-test06
         endpoint: oss.eu-west-0.prod-cloud-ocb.orange-business.com
         list_objects_version: v1
    - ...

Thanos (compactor and store gateway)

The secret is provided with a K8S secret. The secret is mounted in the pod.

The path to that file is specified in an environment variable.

On FE:

env:
  - name: AWS_SHARED_CREDENTIALS_FILE
    value: /etc/creds/fe_access

On AWS:

env:
  - name: AWS_SHARED_CREDENTIALS_FILE
    value: /etc/creds/aws_access

Thanos-sidecar

The secret is injected in Thanos via annotations.

Reference: secret injection via annotations.

The annotations will create a file that contains the S3 credentials.

We had to define the thanos-sidecar container with the environment variable. The full history of this can be read at the beginning of envs-ng/.../kube-prometheus-stack.cue.

The configuration is made in envs-ng/.../kube-prometheus-stack.cue:

containers: [{
        name:            "thanos-sidecar"
        image:           thanosImage
        imagePullPolicy: "IfNotPresent"
      ...
      env: [{
      - name: AWS_SHARED_CREDENTIALS_FILE
        value: /vault/secrets/s3-access-secret-key
      }]
}]