This is documentation for the next version of Grafana Loki documentation. For the latest stable release, go to the latest version.
Migrate to Kafka-based ingestion
Warning
Kafka-based ingestion is an experimental feature. Engineering and on-call support is not available. No SLA is provided.
This guide explains how to migrate a running Loki cluster from the default gRPC write path to Kafka-based ingestion. Kafka-based ingestion adds Kafka, or a Kafka-protocol-compatible system, as a durable buffer between the distributor and the ingester.
Note
This guide documents running two distinct sets of ingesters side by side during the migration: your existing ingesters, unchanged, and a new set of Kafka-consuming ingesters. This is the only migration path Grafana Labs has tested. Because both sets run at the same time, you need enough extra capacity to run a second, full-sized ingester group for at least
querier.query_ingesters_within(default: 3 hours), and often longer.
Before you begin
- You need a running Kafka cluster, or a Kafka-protocol-compatible system such as WarpStream, reachable from both your distributors and your new Kafka-consuming ingesters.
- Create the Kafka topic you plan to use.
kafka_config.auto_create_topic_enableddefaults totrue, which allows the broker to create a missing topic when Loki first uses it, if the broker permits topic auto-creation. Loki also sets the broker-widenum.partitionsdefault tokafka_config.auto_create_topic_default_partitions(default: 1000), which affects every auto-created topic on the cluster. Setauto_create_topic_enabled: falseif you create the topic yourself or share the cluster with other applications. - Read Kafka-based ingestion to understand the architecture, partition sizing, and authentication limitations before you begin.
- Note your current
querier.query_ingesters_withinsetting (default: 3 hours). You’ll need it in step 4. - Plan for enough compute and storage capacity to run a second, full-sized ingester group alongside your existing ingesters for the duration of the migration.
- Make sure your Kafka topic has at least as many partitions as the number of Kafka-consuming ingester replicas you plan to run. Each ingester owns exactly one partition. Loki doesn’t check the topic’s partition count. An ingester whose partition doesn’t exist keeps retrying the Kafka offset lookup for it, so it never becomes ready and never joins the partition ring. Refer to Partition sizing guidance.
- Plan to give each new Kafka-consuming ingester an instance ID that ends in a hyphen followed by a sequential numeric index, for example
kafka-ingester-0,kafka-ingester-1. Loki derives each ingester’s partition ID from this suffix at startup, and fails to start if it can’t parse one. Kubernetes StatefulSet ordinals satisfy this automatically. Don’t use instance IDs that containlocalor end in.lan: Loki always maps these to partition 0, regardless of any numeric suffix.
Migration steps
1. Configure kafka_config
Set the Kafka broker addresses and topic name. If your Kafka deployment requires authentication, also set sasl_username and sasl_password.
This block is required on both the distributor (which produces records) and the new Kafka-consuming ingesters (which consume records). If your queriers share a base configuration file with your ingesters, no further action is needed for the querier; otherwise, see step 5 for the partition ring settings the querier also needs.
kafka_config:
topic: loki-logs
reader_config:
address: kafka:9092
writer_config:
address: kafka:90922. Deploy Kafka-consuming ingesters
Deploy a new, independently-scaled group of ingesters dedicated to consuming from Kafka. Size this group for your full ingestion volume, because it eventually takes over completely from your existing ingesters.
Apply the following configuration to this new group only:
# Kafka-consuming ingester config (new ingester group)
ingester:
kafka_ingestion:
enabled: true
lifecycler:
ring:
kvstore:
prefix: kafka-ingesters/ # any value different from the default, "collectors/"ingester.kafka_ingestion.enabled: truemakes this group join the partition ring and start consuming from its assigned Kafka partitions.ingester.lifecycler.ring.kvstore.prefixisolates this group’s registration in the classic hash ring from your existing ingesters. Every ingester, including Kafka-consuming ones, always registers in the classic hash ring. Giving the new group a different key-value store prefix keeps the distributor’s gRPC write path from discovering or routing to them, so they don’t receive duplicate gRPC writes. It also keeps the querier’s classic-ring lookups from discovering them until step 5, where you point the querier at the same prefix.
Leave your existing ingesters’ configuration unchanged at this point: kafka_ingestion.enabled stays false (the default), and they keep the default ingester.lifecycler.ring.kvstore.prefix.
Note
Every component that participates in the partition ring, the distributor, the Kafka-consuming ingesters, and the querier, must use the same
ingester.kafka_ingestion.partition_ring.kvstoreconfiguration. Only the classic ring prefix above is meant to differ. Don’t change the otherpartition_ringsettings from their defaults.The partition ring key-value store defaults to
memberlist. If you don’t already run memberlist, configurememberlist.join_memberson these components or set the samepartition_ring.kvstorestore on all of them. If you setcommon.ring, the partition ring uses the key-value store from there.The existing ingesters don’t join the partition ring, so they don’t need this configuration.
Roll out this new ingester group before you enable any Kafka writes in the next step, so you can confirm the new ingesters start cleanly and join the partition ring while receiving no traffic yet.
3. Enable dual-write
On your distributors, set distributor.kafka_writes_enabled: true while keeping distributor.ingester_writes_enabled: true (the default).
Additionally, set ingester.kafka_ingestion.enabled: true while keeping it false on the existing ingester group. This means that the old ingester group and the distributors cannot share the same config file. (Not setting this would cause the error kafka is enabled in distributor but not in ingester on startup.)
# Distributor config
distributor:
kafka_writes_enabled: true
ingester_writes_enabled: true
ingester:
kafka_ingestion:
enabled: trueWith this configuration, the distributor writes every stream to both Kafka, which your new Kafka-consuming ingesters read from, and to your existing ingesters over gRPC, at the same time. A push request succeeds only if both writes succeed. If the Kafka write fails, for example because Kafka is unreachable or the partition ring has no ACTIVE partition, the distributor returns an HTTP 500 error to the client, even though your existing ingesters still received the data.
Before you roll out this change, confirm on the /partition-ring page of a distributor or ingester that all new partitions are ACTIVE. Roll the change out to your distributors gradually, and watch loki_distributor_kafka_appends_total with label status="fail" and your push error rate. To undo this step, set kafka_writes_enabled: false again.
4. Wait for the Kafka path to catch up
Monitor the following metrics to confirm that Kafka writes and reads are healthy:
loki_distributor_kafka_appends_totalwith labelstatus="success"should increase at a steady rate.loki_distributor_kafka_latency_secondsshould stay low and stable.loki_kafka_client_partition_reader_consumption_lag_secondsshould stay near zero after an initial catch-up period.loki_ingester_partition_current_offseton your new Kafka-consuming ingesters should keep advancing.
Beyond checking that lag is low, you also need your dual-write to have been running long enough to hold a full window of recent data. The querier only queries ingesters for data within the last querier.query_ingesters_within (default: 3 hours). If you move queries to the partition ring (the next step) before your Kafka-consuming ingesters have accumulated at least this much data, recent queries can return incomplete results until that window fills in.
Note
Keep both ingester groups running for this entire step. Budget at least
querier.query_ingesters_within(default: 3 hours) for this wait, and often longer, since you want a representative period of healthy metrics in addition to the minimum data-retention window.
Only move to the next step after both of these are true:
- The metrics above have looked healthy for a representative period, for example a few hours or a full day of traffic.
- The time since dual-write was fully rolled out to all distributors is at least as long as your
querier.query_ingesters_withinsetting.
5. Switch the query path to the partition ring
Set querier.query_partition_ingesters: true and roll out your queriers (and rulers, if they evaluate rules locally). This tells the querier to look up ingesters using the partition ring instead of the classic hash ring, so it finds your new Kafka-consuming ingesters instead of your existing ones.
# Querier and ruler config
querier:
query_partition_ingesters: true
ingester:
lifecycler:
ring:
kvstore:
prefix: kafka-ingesters/ # Same as in new ingester configThe querier also needs the same ingester.kafka_ingestion.partition_ring key-value store configuration as your Kafka-consuming ingesters, so it can watch the partition ring and discover them. If your querier already shares a base configuration file with your ingesters, this is already in place; otherwise, add it explicitly.
Note
The partition ring routes each read to the single ingester that owns the partition, with no replication. The classic hash ring instead fans a read out to multiple replicas and uses quorum. If a partition’s owning ingester is temporarily unavailable, recent reads can fail until the ingester recovers. With the default
limits_config.ingestion_partitions_tenant_shard_size: 0, every query covers all partitions, so this affects all tenants. With a non-zero shard size, it affects only the tenants whose shard includes that partition. During dual-write, you can safely revert this setting and the ring prefix to their original values if you need to roll back.
Before continuing, confirm that queries covering recent time ranges, for example the last few minutes, still return the data you expect.
6. Cut over to Kafka-only writes
Once queries are reading from your Kafka-consuming ingesters, you can stop writing directly to your existing ingesters over gRPC. On your distributors, set ingester_writes_enabled: false. At least one of kafka_writes_enabled or ingester_writes_enabled must stay true; Loki fails to start otherwise.
# Distributor config
distributor:
kafka_writes_enabled: true
ingester_writes_enabled: falseYour existing ingesters stop receiving new writes after this change, but keep them running. Don’t delete them yet; you still need them for a safe rollback until you complete the next step.
7. Decommission the original ingesters
After you’ve confirmed that reads and writes are fully and reliably served by your Kafka-consuming ingesters, for example after a day of stable operation with no errors, scale down and remove your original ingester deployment.
Keep your original ingesters’ configuration and deployment definitions available for a buffer period after you remove them, in case you need to roll back.
8. Monitor ongoing health
Continue monitoring your Kafka-based ingestion using the metrics documented in Kafka-based ingestion and Key metrics for monitoring Loki.
Rollback procedure
To revert to the direct gRPC write path:
- If you completed step 6, set
distributor.ingester_writes_enabledback totrueand roll out your distributors. Leavedistributor.kafka_writes_enabledand the querier settings unchanged for now. Then wait at leastquerier.query_ingesters_within(default: 3 hours). During this wait, your existing ingesters build up a full window of recent data while queries still read from the Kafka-consuming ingesters. If you switch queries back earlier, recent queries miss the data that was written only to Kafka. This wait is the slow part of a rollback. If you didn’t complete step 6, skip this item, because your existing ingesters received every write. - If you completed step 5, set
querier.query_partition_ingestersback tofalseand changeingester.lifecycler.ring.kvstore.prefixback to its original value on your queriers (and rulers). Roll out these changes. - Set
distributor.kafka_writes_enabledtofalseand roll out your distributors. Do this after the queriers no longer read from the partition ring, because the Kafka-consuming ingesters receive no new data after this change. - Once you’ve confirmed your existing ingesters are serving all reads and writes correctly again, scale down your Kafka-consuming ingesters. Because they’re isolated from the classic hash ring, you can leave some of them running for a while in case you want to retry the migration, without affecting your existing ingesters. Before you stop each Kafka-consuming ingester that you remove:
- Send
POST /ingester/prepare_partition_downscaleto it. This switches its partition toINACTIVE, so distributors stop writing to it. If you stop an ingester without this call, its partition staysACTIVEwith no consumer, and queries on the partition ring fail if you retry the migration later. - Send
POST /ingester/prepare_shutdownto it, then stop it. This removes the ingester as partition owner. The remaining Kafka-consuming ingesters delete anINACTIVEpartition that has no owners from the partition ring afteringester.kafka_ingestion.partition_ring.delete_inactive_partition_after(default: 13 hours). The ingester writes a shutdown marker file (seeingester.shutdown_marker_path). Before you reuse the ingester, remove the marker withDELETE /ingester/prepare_shutdown. Refer to Prepare ingester shutdown.
- Send
Because your existing ingesters never stopped processing gRPC writes during dual-write, rolling back doesn’t cause data loss as long as you haven’t completed step 6. If you had already cut over to Kafka-only writes before deciding to roll back, any data already buffered in Kafka but not yet consumed is read with the normal Kafka consumer catch-up process described in Offset management and restart behavior, so make sure your Kafka-consuming ingesters have caught up on Kafka lag before you disable Kafka consumption on them.
Warning
Rollback is straightforward only while your original ingesters are still running. Once you’ve decommissioned them in step 7, rolling back requires redeploying them from scratch. Freshly redeployed ingesters start with no recent data in memory, so queries for recent time ranges return incomplete results until they rebuild state, either by replaying their write-ahead log (WAL) or by accumulating new data over another
querier.query_ingesters_withinwindow. Don’t decommission your original ingesters until you’re confident you won’t need to roll back.


