Aller au contenu

PromtailEntryOutOfOrder#

This alert is triggered when the logs are not sent in order in Loki.

It's an alert which was created to see the behavior of Loki/promtail.

Known cases#

  • logs are rejected and lost

  • logs are still stored in Loki (this is the case that happens most often).

Troubleshooting Hints#

Gateway log#

Get log of Gateway component :

kubectl logs -n logging -l app=gateway

Gateway error#

Get at least one error. Example :

level=error ts=2021-12-06T12:33:11.543293885Z caller=client.go:334 component=client host=gateway.logging.svc.cluster.local msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-12-06 12:33:10.589490752 +0000 UTC ignored, reason: 'entry out of order' for stream: {app=\"dashboard\", component=\"front\", container=\"dashboard-front\", filename=\"/var/log/pods/dashboard_dashboard-front-7466b459b7-msv7w_a6449492-6cec-4d8d-a071-8e5ff7a23fbe/dashboard-front/80.log\", hostname=\"10.0.149.162\", job=\"dashboard/dashboard\", namespace=\"dashboard\", node_name=\"10.0.149.162\", pod=\"dashboard-front-7466b459b7-msv7w\", stream=\"stdout\"},"

We see that here the error is for a log with the timestamp : 2021-12-06 12:33:10.589490752

Pod log#

Then, we can go see the pod logs at this moment, here: dashboard-front-7466b459b7-msv7w in namespace dashboard :

kubectl logs dashboard-front-7466b459b7-msv7w -n dashboard --timestamps=true | grep "2021-12-06 12:33:10"

And check that the logs are in order.

If they are out of order, it's not a Loki/Promtail problem but kubernetes problem and we must not do the end of the doc.

Pod log in Loki#

Then, we can go see the pod logs at this moment in Grafana, datasource Loki.

If the log line is present in Loki, we can suppose that we are in the case where we do not lose any log.

Note

There is no way to do an exhaustive check.

Solutions#

Case : log present in Loki#

We can delete the pod(s) for which the problem appeared : in our example dashboard-front-7466b459b7-msv7w :

kubectl delete pod -n dashboard dashboard-front-7466b459b7-msv7w

Case : log not present in Loki#

Contact monitoring team.