Aller au contenu

Loki Ingester and hash Ring#

The ingester service is responsible for writing incoming series to a long-term storage backend on the write path and returning in-memory series samples for queries on the read path.

Ingester service use a hashring to coordinate work-sharing and discovery. This hashring comes from the Cortex project.

In most of the cases, you never need to look at the state of the ring members because members will join, leave and redistribute tokens automatically as components come and go during scaling or failover.

More information about Loki Ingester states and hash ring.

Data loss evaluation#

The Loki Ingesters hold 2 replicas of each time series. When all the nodes are running (missing active ingester is not due to a node reboot for example), the write operation succeed.

In this case the following statements are true:

  • If the Ring still has two ACTIVE Loki Ingesters, no data is lost
  • If the Ring has only one ACTIVE Loki Ingester, some data might be lost, as the impacted Ingesters can be holding the two replicas of the same specific time series
  • If the Ring does not have any ACTIVE Loki Ingester, all in-memory data is lost

Data loss evaluation with unhealthy ingesters#

A Loki Ingester is marked as Unhealthy when it fails to send a heartbeat (timeout set to 1m).

When the missing active ingester is due to a node issue (reboot for example), during a short period of time, there are two active ingesters, and one unhealthy ingester.

Writing operation steps:

  1. two out of three ingesters are chosen, regardless of their status (active or unhealthy).

Note

There are two ingesters chosen because replicationFactor = 2.

  1. the information about the selected ingesters is then passed to a filter function that checks their state.

At this step, there are two possible situations:

  • the two chosen ingesters are active : in this case the write operation succeeds
  • an active and an unhealthy ingester are chosen : in this case the write operation does not work, because the filter function will remove the unhealthy ingester.

Only one ingester remains and data is lost. In this case, the following errors will be present in the distributor pod logs:

POST /api/prom/push (500) 8.978096ms Response: \"at least 2 live replicas required, could only find 1 - unhealthy instances: 172.16.1.32:9095\\n\"

Explanation

Unhealthy state of the ingester in the Ring UI is the hearbeat timeout alone, but this timeout is not used when picking the two ingesters (step 1).

In this case it picks one healthy ingester and the other that is not (may be random depending on the tokens), the filtering will remove the unhealthy one.

Only one active ingester remains and the data is lost as it requires two active ingesters in order to write data.

More information about data loss.

Correction

We use the feature autoforget_unhealthy which automatically removes unhealthy ingesters from the hash ring.

An alert is deployed for this case: LokiUnhealthyIngesterAutomaticallyDeleted.

Periodical data flush#

The Loki Ingesters flush the data chunks on the long-term storage periodically.

The parameters that define this action is made in Loki configuration (file /etc/loki/config.yaml on ingester pod).

It also defines the amount of data that can be lost in worst case scenario.

ingester:
   chunk_idle_period: '15m' # timeout period before flush
   chunk_block_size: 262144 # max block size before flush

Access Ring Health UI#

  1. Port-forward on the distributor 
    kubectl -n <namespace> port-forward svc/distributor 3100:3100 &
  2. Connect to Ring UI: http://localhost:3100/ring

Manage Ring Members#

A Loki Ingester might get stuck in a state other than ACTIVE and they need to be forgotten and restarted in order for the work (allocated through token assignment) to be redistributed to healthy ingesters.

There are two ways that allow us to remove Ingesters (stuck in a state other than ACTIVE) from the Ring:

  1. Delete consul pod to force Ingesters to register themselves again

kubectl -n <namespace> delete pod -l app=consul
2. "Forget" the Ingester in the Ring (http://localhost:3100)

If step 1 does not solve the problem, the faulty Ingesters can be removed from the Ring.

Danger

This is to be done with precaution, because depending on the number of Ingesters still remaining in ACTIVE state, data might be lost.

Connect to Ring UI.

  • If the Ring still has two ACTIVE Loki Ingesters, you can click on "Forget" on the concerned Ingester in the Ring UI

  • If the Ring has only one ACTIVE Loki Ingester, data might be lost. In this case, you should spend more time on investigation before performing the "Forget" action

  • If the Ring has no more ACTIVE Loki Ingesters, data has been lost. In this case, you should spend more time on investigation before performing the "Forget" action

Note

The Loki Ingesters can also be removed from the Ring using a script (equivelent to "Forget action"):

rswitch login <cloud|infra zone>
git clone https://git.corp.caascad.com/caascad/applications/caascad-loki.git
bash caascad/applications/caascad-loki/src/loki_tools/loki_remove_ghosts_ingesters_from_ring.sh <namespace>

To find out more informations regarding on-the-fly hand-over between ingesters, check this Cortex documentation.

  1. If Ingester pod is in a bad state, you can restart it (in-memory data will be lost) 
    kubectl delete pod <ingester> -n <namespace>

When deleting a Loki Ingester pod, this can put it in a non-ACTIVE state in the Ring and the alert may come back. If it's the case, you need to perform the "Forget" action as indicated at step 2.