Aller au contenu

Troubleshooting Promtail#

Inspect pipelines stages#

Install binary#

cd /usr/local/bin/
sudo curl -fSL -o promtail.gz "https://github.com/grafana/loki/releases/download/v2.3.0/promtail-linux-amd64.zip"
sudo chmod a+x promtail.gz
sudo gunzip promtail.gz
sudo chmod a+x promtail

Note

Check the latest release.

Create configuration#

Copy the part of the configuration you want to test in scrape_configs

Example :

vim config-promtail.yaml

server:
  http_listen_port: 3101

scrape_configs:
  # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
  - job_name: test
    pipeline_stages:
    - json:
       expressions:
         log: null
    - json:
       expressions:
         level: level
         time: time
       source: log
    - replace:
       expression: .*(\.000Z).*
       replace: Z
       source: time
    - labels:
       level: ""
       time: ""
       test: "level"
    - timestamp:
        format: RFC3339
        location: Europe/Paris
        source: time

Run inspect command#

Example: 

echo '{"log":"{\"time\":\"2021-10-14T09:53:04.000Z\",\"level\":\"FATAL\",\"content\":\"2021-10-14 10:53:04.920 +01:10.0.62.62(38494):root@test234:[7089]:FATAL:  Invalid username/password,login denied.\\n\"}\n","stream":"stdout","time":"2021-10-21T15:12:26.056950769Z"}' | promtail --stdin --dry-run --inspect --client.url http://gateway.logging.svc.cluster.local/api/prom/push -config.file=config-promtail.yaml --log.level=debug

ContainerOutOfMemory#

We have already noticed some slow increase of memory on Promtail. Here is what we noticed :

  • the memory is slowly increasing
  • the cpu is less used while the memory is using more than usual
  • this happened shortly after an operation on Loki (it may also appear after an incident on Loki)

  • in the logs, at the beginning of the increase, we had some log lines like this :

    "level=warn ts=2020-10-15T08:24:13.045655461Z caller=client.go:245 component=client host=gateway:80 msg=\"error sending batch, will retry\" status=500 error=\"server returned HTTP status 500 Internal Server Error (500): at least 2 live replicas required, could only find 1\"\n"

What to check :

  • Check Promtail logs at the beginning of the increase of memory. You may filter on the pod that has a problem ({{namespace="logging", pod="..."}} or {{namespace="logging-client", pod="..."}})
  • If Promtail complains about problems on sending logs to Loki, check Loki global health.
  • If there is nothing interesting, you may kubectl delete pod promtail-xxxxx on Zone-cloud or Infra, or kubectl delete pod caascad-promtail-xxxxx on Zone-Client to remove the alerte. Promtail knows where it was reading the logs and will restart from where it was. Don't worry.