Migrate to Grafana Mimir on Grafana Labs

Migrate from Cortex to Grafana Mimir

Wed, 03 Jun 2026 09:01:40 +0200

Migrate from Cortex to Grafana Mimir

As an operator, you can migrate a Jsonnet deployment of Cortex to Grafana Mimir. The overview includes the steps required for any environment. To migrate deployment environments with Jsonnet, see Migrate to Grafana Mimir using Jsonnet.

Warning
This document was tested with Cortex versions 1.10 and 1.11.

It might work with more recent versions of Cortex, but it’s not guaranteed.

To migrate a Helm deployment of Cortex refer to Migrate from Cortex.

Grafana Mimir includes significant changes that simplify the deployment and continued operation of a horizontally scalable, multi-tenant time series database with long-term storage.

The changes make Grafana Mimir easier to run out of the box:

Removed configuration parameters that don’t require tuning
Renamed some parameters so that they’re more easily understood
Updated the default values of some existing parameters

The mimirtool automates configuration conversion. It provides a simple migration by generating Mimir configuration from Cortex configuration.

Before you begin:

Ensure that you are running either Cortex 1.10.X or Cortex 1.11.X.

If you are running an older version of Cortex, upgrade to Cortex 1.11.1 before proceeding with the migration.
Ensure you have installed Cortex alerting and recording rules as well as Cortex dashboards.

Using the monitoring mixin, you need to install both alerting and recording rules in either Prometheus or Cortex. You also need to install dashboards in Grafana. To download a prebuilt ZIP file that contains the alerting and recording rules, refer to Release Cortex-jsonnet 1.11.0.

To upload rules to the ruler using mimirtool, refer to mimirtool rules. To import the dashboards into Grafana, refer to Import dashboard.

Notable changes

Note
For the full list of changes, refer to Mimir’s CHANGELOG.

The Grafana Mimir HTTP server defaults to listening on port 8080; Cortex defaults to listening on port 80. To maintain port 80 as the listening port, set -server.http-listen-port=80.
Grafana Mimir uses anonymous as the default tenant ID when -auth.multitenancy=false. Cortex uses fake as the default tenant ID when -auth.enabled=false. Use -auth.no-auth-tenant=fake when -auth.multitenancy=false to match the Cortex default tenant ID.

Grafana Mimir removes the legacy HTTP prefixes deprecated in Cortex.

Query endpoints

Legacy	Current
`/<legacy-http-prefix>/api/v1/query`	`<prometheus-http-prefix>/api/v1/query`
`/<legacy-http-prefix>/api/v1/query_range`	`<prometheus-http-prefix>/api/v1/query_range`
`/<legacy-http-prefix>/api/v1/query_exemplars`	`<prometheus-http-prefix>/api/v1/query_exemplars`
`/<legacy-http-prefix>/api/v1/series`	`<prometheus-http-prefix>/api/v1/series`
`/<legacy-http-prefix>/api/v1/labels`	`<prometheus-http-prefix>/api/v1/labels`
`/<legacy-http-prefix>/api/v1/label/{name}/values`	`<prometheus-http-prefix>/api/v1/label/{name}/values`
`/<legacy-http-prefix>/api/v1/metadata`	`<prometheus-http-prefix>/api/v1/metadata`
`/<legacy-http-prefix>/api/v1/read`	`<prometheus-http-prefix>/api/v1/read`
`/<legacy-http-prefix>/api/v1/cardinality/label_names`	`<prometheus-http-prefix>/api/v1/cardinality/label_names`
`/<legacy-http-prefix>/api/v1/cardinality/label_values`	`<prometheus-http-prefix>/api/v1/cardinality/label_values`
`/api/prom/user_stats`	`/api/v1/user_stats`

Distributor endpoints

Legacy endpoint	Current
`/<legacy-http-prefix>/push`	`/api/v1/push`
`/all_user_stats`	`/distributor/all_user_stats`
`/ha-tracker`	`/distributor/ha_tracker`

Ingester endpoints

Legacy	Current
`/ring`	`/ingester/ring`
`/shutdown`	`/ingester/shutdown`
`/flush`	`/ingester/flush`
`/push`	`/ingester/push`

Ruler endpoints

Legacy	Current
`/<legacy-http-prefix>/api/v1/rules`	`<prometheus-http-prefix>/api/v1/rules`
`/<legacy-http-prefix>/api/v1/alerts`	`<prometheus-http-prefix>/api/v1/alerts`
`/<legacy-http-prefix>/rules`	`<prometheus-http-prefix>/config/v1/rules`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/<legacy-http-prefix>/rules/{namespace}/{groupName}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/<legacy-http-prefix>/rules/{namespace}/{groupName}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/ruler_ring`	`/ruler/ring`

Alertmanager endpoints

Legacy	Current
`/<legacy-http-prefix>`	`/alertmanager`
`/status`	`/multitenant_alertmanager/status`

Generate the configuration for Grafana Mimir

The mimirtool config convert command converts Cortex configuration to Mimir configuration. You can use it to update both flags and configuration files.

Install mimirtool

To install Mimirtool, download the appropriate binary from the latest release for your operating system and architecture and make it executable.

Alternatively, use a command line tool such as curl to download mimirtool. For example, for Linux with the AMD64 architecture, use the following command:

curl -fLo mimirtool https://github.com/grafana/mimir/releases/latest/download/mimirtool-linux-amd64
chmod +x mimirtool

Use mimirtool

The mimirtool config convert command converts Cortex 1.11 configuration files to Grafana Mimir configuration files. It removes any configuration parameters that are no longer available in Grafana Mimir, and it renames configuration parameters that have a new name. If you have explicitly set configuration parameters to a value matching the Cortex default, by default, mimirtool config convert doesn’t update the value. To have mimirtool config convert update explicitly set values from the Cortex defaults to the new Grafana Mimir defaults, provide the --update-defaults flag. Refer to convert for more information on using mimirtool for configuration conversion.

Migrate to Grafana Mimir using Jsonnet

Grafana Mimir has a Jsonnet library that replaces the existing Cortex Jsonnet library and updated monitoring mixin.

Migrate to Grafana Mimir video

The following video shows you how to migrate to Grafana Mimir using Jsonnet.

Migrate to Grafana Mimir instructions

The following instructions describe how to migrate to Grafana Mimir using Jsonnet.

To install the updated libraries using jsonnet-bundler, run the following commands:

jb install github.com/grafana/mimir/operations/mimir@main
jb install github.com/grafana/mimir/operations/mimir-mixin@main

To deploy the updated Jsonnet:

Install the updated monitoring mixin.

a. Add the dashboards to Grafana. The dashboards replace your Cortex dashboards and continue to work for monitoring Cortex deployments.

Note
Resource dashboards are enabled by default and require additional metrics sources. To understand the required metrics sources, refer to Additional resources metrics.

b. Install the recording and alerting rules into the ruler or a Prometheus server.
Replace the import of the Cortex Jsonnet library with the Mimir Jsonnet library. For example:
jsonnet
```
import 'github.com/grafana/mimir/operations/mimir/mimir.libsonnet'
```
Remove the cortex_ prefix from any member keys of the <MIMIR>._config object. For example, cortex_compactor_disk_data_size becomes compactor_disk_data_size.
If you are using the Cortex defaults, set the server HTTP port to 80. The new Mimir default is 8080. For example:
jsonnet
```
(import 'github.com/grafana/mimir/operations/mimir/mimir.libsonnet') {
  _config+: {
    server_http_port: 80,
  },
}
```
For each component, use mimirtool to update the configured arguments. To extract the flags for each component, refer to Extracting flags from Jsonnet.
Apply the updated Jsonnet

To verify that the cluster is operating correctly, use the monitoring mixin dashboards.

Migrate from Thanos or Prometheus to Grafana Mimir

Wed, 03 Jun 2026 09:01:40 +0200

Migrate from Thanos or Prometheus to Grafana Mimir

This document guides an operator through the process of migrating a deployment of Thanos or Prometheus to Grafana Mimir.

Overview

Grafana Mimir stores series in TSDB blocks uploaded in an object storage bucket. These blocks are the same as those used by Prometheus and Thanos. Each project stores blocks in different places and uses slightly different block metadata files.

Configuring remote write to Grafana Mimir

For configuration of remote write to Grafana Mimir, refer to Configuring Prometheus remote write.

Uploading historic TSDB blocks to Grafana Mimir

Grafana Mimir supports uploading of historic TSDB blocks, notably from Prometheus. In order to enable this functionality, either for all tenants or a specific one, refer to Configure TSDB block upload.

Prometheus stores TSDB blocks in the path specified in the --storage.tsdb.path flag.

To find all block directories in the TSDB <STORAGE TSDB PATH>, run the following command:

find <STORAGE TSDB PATH> -name chunks -exec dirname {} \;

Grafana Mimir supports multiple tenants and stores blocks per tenant. With multi-tenancy disabled, there is a single tenant called anonymous.

Use Grafana mimirtool to upload each block, such as those identified by the previous command, to Grafana Mimir:

mimirtool backfill --address=http://<mimir-hostname> --id=<tenant> <block1> <block2>...

Note
If you need to authenticate against Grafana Mimir, you can provide an API key via the --key flag, for example --key=$(cat token.txt).

Grafana Mimir performs some sanitization and validation of each block’s metadata. As a result, it rejects Thanos blocks due to unsupported labels. As a workaround, if you need to upload Thanos blocks, upload the blocks directly to the Grafana Mimir blocks bucket, prefixed by <tenant>/<block ID>/.

Block metadata

Each block has a meta.json metadata file that is used by Grafana Mimir, Prometheus, and Thanos to identify the block contents. Each project has its own metadata conventions.

In the Grafana Mimir 2.1 (or earlier) release, the ingesters added an external label to the meta.json file to identify the tenant that owns the block.

In the Grafana Mimir 2.2 (or later) release, blocks no longer have a label that identifies the tenant.

Note
Blocks from Prometheus don’t have any external labels stored in them. Only blocks from Thanos use labels.

Note
If you encounter the HTTP error 413 “Request Entity Too Large” when uploading blocks using mimirtool, and if Nginx is being used as a reverse proxy, the uploaded block size may be exceeding Nginx’s default maximum allowed request body size. To resolve this issue:

Determine the current size of the blocks you are trying to upload by running the following command(by default the blocks are sized around 500MB, but it can vary depending on the configuration of Prometheus or Thanos.)
Bash
find <path/to/blocks> -name 'chunks' -printf '%s\n' | numfmt --to=iec-i
This shows the size of each block’s chunks directory in a human-readable format. Increase the client_max_body_size directive in the Nginx configuration:

For manual Nginx deployments, open the Nginx configuration file (e.g., /etc/nginx/nginx.conf) and set the client_max_body_size directive inside the server block for the Mimir endpoint to a value about 5% larger than the maximum size of the blocks you are uploading. For example:
server {
    ...
    client_max_body_size 540M;
    ...
}
location / {
    ...
    client_max_body_size 540M;
    ...
}
For Helm deployments of Mimir, you can set the gateway.nginx.config.clientMaxBodySize value in your Helm values file to a higher value, e.g.:
YAML
gateway:
  nginx:
    config:
      clientMaxBodySize: 540M
Apply the configuration changes:

For manual Nginx deployments, save the configuration file and reload Nginx:
Bash
sudo nginx -s reload
For Helm deployments, upgrade your Mimir release with the updated values file:
Bash
helm upgrade <release-name> <chart-name> -f values.yaml
After increasing the client_max_body_size setting, you can upload the blocks without encountering the 413 error.

Considerations on Thanos specific features

Thanos requires that Prometheus is configured with external labels. When the Thanos sidecar uploads blocks, it includes the external labels from Prometheus in the meta.json file inside the block. When you query the block, Thanos injects Prometheus’ external labels in the series returned in the query result. Thanos also uses labels for the deduplication of replicated data.

If you want to use existing blocks from Thanos by Grafana Mimir, there are some considerations:

Grafana Mimir doesn’t inject external labels into query results. This means that blocks that were originally created by Thanos will not include their external labels in the results when queried by Grafana Mimir. If you need to have external labels in your query results, this is currently not possible to achieve in Grafana Mimir.

Grafana Mimir will not respect deduplication labels configured in Thanos when querying the blocks. For best query performance, only upload Thanos blocks from a single Prometheus replica from each HA pair. If you upload blocks from both replicas, the query results returned by Mimir will include samples from both replicas.

Grafana Mimir does not support Thanos’ downsampling feature. To guarantee query results correctness please only upload original (raw) Thanos blocks into Mimir’s storage. If you also upload blocks with downsampled data (ie. blocks with non-zero Resolution field in meta.json file), Grafana Mimir will merge raw samples and downsampled samples together at the query time. This may cause that incorrect results are returned for the query.

Migrate historic TSDB blocks from Thanos to Grafana Mimir

Copy the blocks from Thanos’s bucket to an Intermediate bucket.

Create an intermediate object storage bucket (such as Amazon S3 or GCS) within your cloud provider, where you can copy the historical blocks and work on them before uploading them to the Mimir bucket.

Tip
Run the commands within a screen or tmux session, to avoid any interruptions because the steps might take some time depending on the amount of data.

For Amazon S3, use the aws tool:
Bash
```
aws s3 cp --recursive s3://<THANOS-BUCKET> s3://<INTERMEDIATE-MIMIR-BUCKET>/
```
For Google Cloud Storage (GCS), use the gsutil tool:
Bash
```
gsutil -m cp -r gs://<THANOS-BUCKET>/* gs://<INTERMEDIATE-MIMIR-BUCKET>/
```
After the copy process completes, inspect the blocks in the bucket to make sure that they are valid from a Thanos perspective.
Bash
```
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml
```
Remove the downsampled blocks.

Mimir doesn’t understand the downsampled blocks from Thanos, such as blocks with a non-zero Resolution field in the meta.json file. Therefore, you need to remove the 5m and 1h downsampled blocks from this bucket.

Mark the downsampled blocks for deletion:
Bash
```
thanos tools bucket retention \
    --objstore.config-file bucket.yaml \
    --retention.resolution-1h=1s \
    --retention.resolution-5m=1s \
    --retention.resolution-raw=0s
```
Cleanup the blocks marked for deletion.
Bash
```
thanos tools bucket cleanup \
    --objstore.config-file bucket.yaml \
    --delete-delay=0
```

Remove the duplicated blocks.

If two replicas of Prometheus instances are deployed for high-availability, then only upload the blocks from one of the replicas and drop from the other replica.

# Get list of all blocks in the bucket
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml \
    --output=tsv > blocks.tsv

# Find blocks from replica that we will drop
cat blocks.tsv| grep prometheus_replica=<PROMETHEUS-REPLICA-TO-DROP> \
    | awk '{print $1}' > blocks_to_drop.tsv

# Mark found blocks for deletion
for ID in $(cat blocks_to_drop.tsv)
do
    thanos tools bucket mark \
       --marker="deletion-mark.json" \
       --objstore.config-file bucket.yaml \
       --details="Removed as duplicate" \
       --id $ID
done

Note
You must replace prometheus_replica with the unique label that would differentiate Prometheus replicas in your setup.

Clean up the duplicate blocks marked for deletion again:

thanos tools bucket cleanup \
    --objstore.config-file bucket.yaml \
    --delete-delay=0

Tip
If you want to visualize exactly what’s happening in the blocks, with respect to the source of blocks, external labels, compaction levels, and more, you can use the following command to get the output as CSV and import it into a spreadsheet:
Bash
thanos tools bucket inspect \
    --objstore.config-file bucket-prod.yaml \
    --output=csv > thanos-blocks.csv

Relabel the blocks with external labels.

Mimir doesn’t inject external labels from the meta.json file into query results. Therefore, you need to relabel the blocks with the required external labels in the meta.json file.

Tip
You can get the external labels in the meta.json file of each block from the CSV file that’s imported, and build the rewrite configuration accordingly.

Tip
Add the __tenant_id__ label with the value of your target tenant. This is important for Mimir to successfully query the migrated data when you enable tenant federation and configure the Grafana data source to query multiple tenants.

Create a rewrite configuration that is similar to this:

# relabel-config.yaml
- action: replace
  target_label: "<LABEL-KEY>"
  replacement: "<LABEL-VALUE>"

Perform the rewrite dry run to confirm all works well.

# Get list of all blocks in the bucket after removing the depuplicate and downsampled blocks.
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml \
    --output=tsv > blocks-to-rewrite.tsv

# Check if rewrite of the blocks with external labels is working as expected.
for ID in $(cat blocks-to-rewrite.tsv)
do
    thanos tools bucket rewrite \
        --objstore.config-file bucket.yaml \
        --rewrite.to-relabel-config-file relabel-config.yaml \
        --dry-run \
        --id $ID
done

After you confirm that the rewrite is working as expected via --dry-run, apply the changes with the --no-dry-run flag. Remember to include --delete-blocks, otherwise the original blocks will not be marked for deletion.

# Rewrite the blocks with external labels and mark the original blocks for deletion.
for ID in $(cat blocks-to-rewrite.tsv)
do
    thanos tools bucket rewrite  \
        --objstore.config-file bucket.yaml \
        --rewrite.to-relabel-config-file relabel-config.yaml \
        --no-dry-run \
        --delete-blocks \
        --id $ID
done

The output of relabelling of every block would look like something below.

level=info ts=2022-10-10T13:03:32.032820262Z caller=factory.go:50 msg="loading bucket configuration"
level=info ts=2022-10-10T13:03:32.516953867Z caller=tools_bucket.go:1160 msg="downloading block" source=01GEGWPME2187SVFH63G8DH7KH
level=info ts=2022-10-10T13:03:35.825009556Z caller=tools_bucket.go:1197 msg="changelog will be available" file=/tmp/thanos-  rewrite/01GF0ZWPWGEPHG5NV79NH9KMPV/change.log
level=info ts=2022-10-10T13:03:35.836953593Z caller=tools_bucket.go:1212 msg="starting rewrite for block" source=01GEGWPME2187SVFH63G8DH7KH  new=01GF0ZWPWGEPHG5NV79NH9KMPV toDelete= toRelabel="- action: replace\n  target_label: \"cluster\"\n  replacement: \"prod-cluster\"\n"
level=info ts=2022-10-10T13:04:47.57624244Z caller=compactor.go:42 msg="processed 10.00% of 701243 series"
level=info ts=2022-10-10T13:04:53.4046885Z caller=compactor.go:42 msg="processed 20.00% of 701243 series"
level=info ts=2022-10-10T13:04:59.649337602Z caller=compactor.go:42 msg="processed 30.00% of 701243 series"
level=info ts=2022-10-10T13:05:02.986219042Z caller=compactor.go:42 msg="processed 40.00% of 701243 series"
level=info ts=2022-10-10T13:05:05.990498497Z caller=compactor.go:42 msg="processed 50.00% of 701243 series"
level=info ts=2022-10-10T13:05:09.349918024Z caller=compactor.go:42 msg="processed 60.00% of 701243 series"
level=info ts=2022-10-10T13:05:12.040895624Z caller=compactor.go:42 msg="processed 70.00% of 701243 series"
level=info ts=2022-10-10T13:05:15.253899238Z caller=compactor.go:42 msg="processed 80.00% of 701243 series"
level=info ts=2022-10-10T13:05:18.471471014Z caller=compactor.go:42 msg="processed 90.00% of 701243 series"
level=info ts=2022-10-10T13:05:21.536267363Z caller=compactor.go:42 msg="processed 100.00% of 701243 series"
level=info ts=2022-10-10T13:05:21.536466158Z caller=tools_bucket.go:1222 msg="wrote new block after modifications; flushing" source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV
level=info ts=2022-10-10T13:05:28.675240198Z caller=tools_bucket.go:1231 msg="uploading new block" source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV
level=info ts=2022-10-10T13:05:38.922348564Z caller=tools_bucket.go:1241 msg=uploaded source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV
level=info ts=2022-10-10T13:05:38.979696873Z caller=block.go:203 msg="block has been marked for deletion" block=01GEGWPME2187SVFH63G8DH7KH
level=info ts=2022-10-10T13:05:38.979832767Z caller=tools_bucket.go:1249 msg="rewrite done" IDs=01GEGWPME2187SVFH63G8DH7KH
level=info ts=2022-10-10T13:05:38.980197796Z caller=main.go:161 msg=exiting

Cleanup the original blocks which are marked for deletion.

thanos tools bucket cleanup \
    --objstore.config-file bucket.yaml \
    --delete-delay=0

Note
If there are multiple Prometheus clusters, then relabelling each of them in parallel may speed up the entire process.

Get the list of blocks that for each cluster, and process it separately.
Bash
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml \
    --output=tsv \
    | grep <PROMETHEUS-CLUSTER-NAME> \
    | awk '{print $1}' > prod-blocks.tsv
Bash
for ID in `cat prod-blocks.tsv`
do
    thanos tools bucket rewrite  \
       --objstore.config-file bucket.yaml \
       --rewrite.to-relabel-config-file relabel-config.yaml \
       --delete-blocks \
       --no-dry-run \
       --id $ID
done

Remove external labels from meta.json.

Mimir compactor will not be able to compact the blocks having external labels with Mimir’s own blocks that don’t have any such labels in their meta.json. Therefore these external labels have to be removed before copying them to the Mimir bucket.

Use the below script to remove the labels from the meta.json.

For Amazon S3, use the aws tool:

#!/bin/bash

BUCKET="XXX"

echo "Fetching list of meta.json files (this can take a while if there are many blocks)"
aws s3 ls $BUCKET --recursive | awk '{print $4}' | grep meta.json | grep -v meta.json.orig > meta-files.txt

echo "Processing meta.json files"
for FILE in $(cat meta-files.txt); do
   echo "Removing Thanos labels from $FILE"
   ORIG_META_JSON=$(aws s3 cp s3://$BUCKET/$FILE -)
   UPDATED_META_JSON=$(echo "$ORIG_META_JSON" | jq "del(.thanos.labels)")

   if ! diff -u <( echo "$ORIG_META_JSON" | jq . ) <( echo "$UPDATED_META_JSON" | jq .) > /dev/null; then
     echo "Backing up $FILE to $FILE.orig"
     aws s3 cp "s3://$BUCKET/$FILE" "s3://$BUCKET/$FILE.orig"
     echo "Uploading modified $FILE"
     echo "$UPDATED_META_JSON" | aws s3 cp - "s3://$BUCKET/$FILE"
   else
     echo "No diff for $FILE"
   fi
done

For Google Cloud Storage (GCS), use the gsutil tool:

#!/bin/bash

BUCKET="GCS Bucket name"

echo "Fetching list of meta.json files (this can take a while if there are many blocks)"
gsutil ls "gs://$BUCKET/*/meta.json" > meta-files.txt

echo "Processing meta.json files"
for FILE in $(cat meta-files.txt); do
   echo "Removing Thanos labels from $FILE"
   ORIG_META_JSON=$(gsutil cat "$FILE")
   UPDATED_META_JSON=$(echo "$ORIG_META_JSON" | jq "del(.thanos.labels)")

   if ! diff -u <( echo "$ORIG_META_JSON" | jq . ) <( echo "$UPDATED_META_JSON" | jq .) > /dev/null; then
      echo "Backing up $FILE to $FILE.orig"
      gsutil cp "$FILE" "$FILE.orig"
      echo "Uploading modified $FILE"
      echo "$UPDATED_META_JSON" | gsutil cp - "$FILE"
   else
      echo "No diff for $FILE"
   fi
done

Copy the blocks from the intermediate bucket to the Mimir bucket.

For Amazon S3, use the aws tool:
Bash
```
aws s3 cp --recursive s3://<INTERMEDIATE-MIMIR-BUCKET> s3://<MIMIR-AWS-BUCKET>/<TENANT>/
```
For Google Cloud Storage (GCS), use the gsutil tool:
Bash
```
gsutil -m cp -r gs://<INTERMEDIATE-MIMIR-BUCKET> gs://<MIMIR-GCS-BUCKET>/<TENANT>/
```
Historical blocks are not available for querying immediately after they are uploaded because the bucket index with the list of all available blocks first needs to be updated by the compactor. The compactor typically perform such an update every 15 minutes. After an update completes, other components such as the querier or store-gateway are able to work with the historical blocks, and the blocks are available for querying through Grafana.
Check the store-gateway HTTP endpoint at http://<STORE-GATEWAY-ENDPOINT>/store-gateway/tenant/<TENANT-NAME>/blocks to verify that the uploaded blocks are there.

Migrate from Thanos to Mimir using Thanos sidecar

Wed, 03 Jun 2026 09:01:40 +0200

Migrate from Thanos to Mimir using Thanos sidecar

As an operator, you can migrate a deployment of Thanos to Grafana Mimir by using Thanos sidecar to move metrics over with a stepped workflow.

Overview

An option when migrating is to allow Thanos to query Mimir. This way you retain historical data via your existing Thanos deployment while pointing all of your Prometheus servers (or Grafana Alloy and other metric sources) to Mimir.

The overall setup consists of setting up Thanos Sidecar alongside Mimir and then pointing Thanos Query to the sidecar as if it was just a normal sidecar.

This is not so much of a guide as it is a collection of configurations that have been used with success prior.

Warning: This setup is unsupported, experimental and has not been battle tested. Your mileage may vary, ensure you have tested this for your use case and have made appropriate backups (and tested your procedures for recovery)

Technical details

There are very few obstacles to get this working properly. Thanos Sidecar is designed to work with the Prometheus API which Mimir (almost) implements. There are only 2 endpoints that are not implemented that we need to “spoof” to get Thanos sidecar to believe it is connecting to a Prometheus server: /api/v1/status/buildinfo and /api/v1/status/config. We spoof these endpoints using NGINX (more details later).

In addition, you need to handle the /metrics endpoint of the Prometheus server through NGINX configuration. Since Thanos version v0.37.0, the sidecar evaluates the prometheus_tsdb_lowest_timestamp_seconds to get the lowest timestamp in the TSDB database. Refer to https://github.com/thanos-io/thanos/pull/7820.

The only other roadblock is the requirement for requests to Mimir to contain the X-Scope-Org-Id header to identify the Tenant. We inject this header using another NGINX container sitting in between Thanos Sidecar and Mimir.

In our case, everything was being deployed in Kubernetes. Mimir was deployed using the mimir-distributed helm chart. Therefore configurations shown will be Kubernetes manifests that will need to be modified for your environment. If you are not using Kubernetes then you will need to set up the appropriate configuration for NGINX to implement the technical setup described above. You may use the configurations below for inspiration but they will not work out of the box.

Deploy Thanos sidecar

We deployed the sidecar using Kubernetes using the below manifest:

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-sidecar
  namespace: mimir
  labels:
    app: thanos-sidecar
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-sidecar
  template:
    metadata:
      labels:
        app: thanos-sidecar
    spec:
      volumes:
        - name: config
          configMap:
            name: sidecar-nginx
            defaultMode: 420
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.19-alpine
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          volumeMounts:
            - name: config
              mountPath: /etc/nginx
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
        - name: thanos-sidecar
          image: quay.io/thanos/thanos:v0.26.0
          args:
            - sidecar
            - "--prometheus.url=http://localhost:8080/prometheus"
            - "--grpc-address=:10901"
            - "--http-address=:10902"
            - "--log.level=info"
            - "--log.format=logfmt"
          ports:
            - name: http
              containerPort: 10902
              protocol: TCP
            - name: grpc
              containerPort: 10901
              protocol: TCP
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

Service

apiVersion: v1
kind: Service
metadata:
  name: thanos-sidecar
  namespace: mimir
  labels:
    app: thanos-sidecar
spec:
  ports:
    - name: 10901-10901
      protocol: TCP
      port: 10901
      targetPort: 10901
  selector:
    app: thanos-sidecar
  type: ClusterIP

NGINX configuration for Thanos sidecar

Notice that there is an NGINX container within the sidecar deployment definition. This NGINX instance sits between the sidecar and Mimir (ie. sidecar --> nginx --> mimir) it is responsible for injecting the tenant ID into the requests from the sidecar to Mimir. The configuration that this NGINX instance uses is as follows:

worker_processes  5;  ## Default: 1
error_log  /dev/stderr;
pid        /tmp/nginx.pid;
worker_rlimit_nofile 8192;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;

  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
        '"$request" $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';
  access_log   /dev/stderr  main;

  sendfile     on;
  tcp_nopush   on;
  resolver kube-dns.kube-system.svc.cluster.local;

  server {
    listen 8080;

    # Distributor endpoints
    location / {
      proxy_set_header X-Scope-OrgID 1;
      proxy_pass      http://mimir-distributed-nginx.mimir.svc.cluster.local:80$request_uri;
    }
  }
}

NGINX configuration for Mimir

Now we modified the Mimir Distributed NGINX configuration to allow the sidecar to fetch the “external labels” and the “version” of our “Prometheus” server. You should be careful with the external labels to make sure that you don’t override any existing labels (in our case we used “source=‘mimir’”). You should also consider existing dashboards and make sure that they will continue to work as expected. Make sure to test your setup first.

In particular, these location blocks were added (full config further down):

location /prometheus/api/v1/status/config {
        add_header Content-Type application/json;
        return 200 "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n  external_labels:\\n    source: mimir\"}}";
    }

location /prometheus/api/v1/status/buildinfo {
        add_header Content-Type application/json;
        return 200 "{\"status\":\"success\",\"data\":{\"version\":\"2.35.0\",\"revision\":\"6656cd29fe6ac92bab91ecec0fe162ef0f187654\",\"branch\":\"HEAD\",\"buildUser\":\"root@cf6852b14d68\",\"buildDate\":\"20220421-09:53:42\",\"goVersion\":\"go1.18.1\"}}";
    }

location /prometheus/metrics {
        gzip on;
        gzip_types text/plain
        add_header "Content-Encoding" "gzip";
        add_header "Content-Type" "text/plain; version=1.0.0; escaping=underscores";
        return 200 "# HELP prometheus_tsdb_lowest_timestamp_seconds Lowest timestamp value stored in the database.\n# TYPE prometheus_tsdb_lowest_timestamp_seconds gauge\nprometheus_tsdb_lowest_timestamp_seconds 0\n";
    }

You need to modify the first one to set your external labels. The second one just allows Thanos to detect the version of Prometheus running. The third one fakes the metrics endpoint of the Prometheus server and returns the prometheus_tsdb_lowest_timestamp_seconds metric with a value of 0 which disables setting the min time within the Thanos sidecar.

To modify the external labels alter this string: "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n external_labels:\\n source: mimir\"}}". You will notice it is escaped JSON. Simply add elements to the data.external_labels field to add more external labels as required or alter the existing key to modify them.

Full Mimir distributed NGINX configuration

worker_processes  5;  ## Default: 1
error_log  /dev/stderr;
pid        /tmp/nginx.pid;
worker_rlimit_nofile 8192;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;

  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
        '"$request" $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';
  access_log   /dev/stderr  main;

  sendfile     on;
  tcp_nopush   on;
  resolver kube-dns.kube-system.svc.cluster.local;

  server {
    listen 8080;

    location = / {
      return 200 'OK';
      auth_basic off;
    }

    # Distributor endpoints
    location /distributor {
      proxy_pass      http://mimir-distributed-distributor.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /api/v1/push {
      proxy_pass      http://mimir-distributed-distributor.mimir.svc.cluster.local:8080$request_uri;
    }

    # Alertmanager endpoints
    location /alertmanager {
      proxy_pass      http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /multitenant_alertmanager/status {
      proxy_pass      http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /api/v1/alerts {
      proxy_pass      http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$request_uri;
    }

    # Ruler endpoints
    location /prometheus/config/v1/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location /prometheus/api/v1/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }

    location /api/v1/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location /prometheus/api/v1/alerts {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location /prometheus/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /ruler/ring {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }

    location /prometheus/api/v1/status/config {
      add_header Content-Type application/json;
      return 200 "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n  external_labels:\\n    source: mimir\"}}";
    }

    location /prometheus/api/v1/status/buildinfo {
      add_header Content-Type application/json;
      return 200 "{\"status\":\"success\",\"data\":{\"version\":\"2.35.0\",\"revision\":\"6656cd29fe6ac92bab91ecec0fe162ef0f187654\",\"branch\":\"HEAD\",\"buildUser\":\"root@cf6852b14d68\",\"buildDate\":\"20220421-09:53:42\",\"goVersion\":\"go1.18.1\"}}";
    }

    location /prometheus/metrics {
      gzip on;
      gzip_types text/plain
      add_header "Content-Encoding" "gzip";
      add_header "Content-Type" "text/plain; version=1.0.0; escaping=underscores";
      return 200 "# HELP prometheus_tsdb_lowest_timestamp_seconds Lowest timestamp value stored in the database.\n# TYPE prometheus_tsdb_lowest_timestamp_seconds gauge\nprometheus_tsdb_lowest_timestamp_seconds 0\n";
    }

    # Rest of /prometheus goes to the query frontend
    location /prometheus {
      proxy_pass      http://mimir-distributed-query-frontend.mimir.svc.cluster.local:8080$request_uri;
    }

    # Buildinfo endpoint can go to any component
    location = /api/v1/status/buildinfo {
      proxy_pass      http://mimir-distributed-query-frontend.mimir.svc.cluster.local:8080$request_uri;
    }
  }
}

Migrate from classic to ingest storage architecture

Wed, 03 Jun 2026 09:01:40 +0200

Migrate from classic to ingest storage architecture

Migrate from a Grafana Mimir cluster running with classic architecture to one running with ingest storage architecture. This process allows you to migrate without downtime by temporarily running two clusters in parallel and duplicating writes. For background on the different architectures, refer to Grafana Mimir architecture.

Note
This procedure temporarily doubles your ingestion and storage costs, because both clusters run in parallel and receive duplicated writes.

Follow these steps to migrate your Grafana Mimir cluster from classic architecture to ingest storage architecture.

Before you begin

Before you begin, ensure you have the following:

An existing Mimir cluster running with classic architecture.
Access to deploy and configure a second Mimir cluster.
Write clients, for example, Prometheus or OpenTelemetry Collector, that you can configure to send data to multiple endpoints.
Read clients, for example, Grafana, that you can reconfigure to point to a new endpoint.
The ability to modify Helm or Jsonnet configurations for both clusters.

Prepare your existing Grafana Mimir cluster for migration

Before creating a new Grafana Mimir cluster, temporarily scale out compactors in your existing cluster. This step helps handle the increased number of blocks created while both clusters are writing to the same object storage bucket.

Set up a new Grafana Mimir cluster with ingest storage

Deploy a new Grafana Mimir cluster that uses ingest storage. Configure the new cluster to use the same object storage bucket as the old cluster.

To prevent conflicts, disable the compactor, as shown in the following Helm values file sample:

compactor:
  replicas: 0

This configuration lets the new cluster use existing blocks from the old cluster while avoiding block overwrites.

Optionally, to avoid duplicate alerts, disable alert sending from the ruler. For example, if you’re using the Helm chart:

mimir:
structuredConfig:
  limits:
    ruler_alerting_rules_evaluation_enabled: false

# or

ruler:
extraArgs:
  ruler.alerting-rules-evaluation-enabled: false

If your alert delivery system, for example, Grafana OnCall or PagerDuty, already deduplicates alerts, you can skip this step.

For setup instructions, refer to the existing Helm chart or Jsonnet configuration documentation for ingest storage.

After completing this step, open the Mimir overview dashboards for both clusters and verify that all components are healthy. Check that ingestion and querying are succeeding and that there are no gaps or errors.

Configure write clients to duplicate writes

Configure all write clients, such as Prometheus servers and OpenTelemetry Collectors, to send metrics to both clusters simultaneously.

This ensures that the new cluster receives all recent samples while it accesses historical blocks from object storage.

Keep read clients, such as Grafana, configured to use the old cluster during this step.

Wait until both clusters have a complete view of the data

Keep write duplication running long enough for both clusters to have a complete view of recent data.

Ingesters in the old cluster continuously upload new blocks to object storage and create new blocks on disk every 2 hours. Because ingesters typically hold about 12–13 hours of recent data in memory and on local disk, continue double writing for at least that duration. This ensures that the new cluster receives all data written during the overlap period and that queries return a complete dataset.

You don’t need to disable block uploads from ingesters in the new cluster. Both clusters can safely upload blocks to the shared object storage because compactors merge them correctly.

After the migration is complete, scale the compactors in the new cluster back down to their normal count.

Switch read clients to the new cluster

Update read clients, like Grafana, to query the new cluster.

Verify that the old cluster is no longer receiving queries by checking for msg="query stats" entries in the query-frontend logs.

Scale down compactors in the old cluster

Reduce the number of compactor replicas in the old cluster to zero. For example:

compactor:
  replicas: 0

Run this step immediately before scaling up the compactors in the new cluster, ideally within 15 minutes. If the bucket index isn’t updated by the new cluster’s compactors within about an hour of stopping the old compactors, read queries can fail.

Scale out compactors in the new cluster

Increase the number of compactor replicas in the new cluster to match the old configuration. This activates compaction in the new cluster and completes the transition for background storage management.

Update ruler alerting configuration

If you disabled alert sending in the new cluster:

Disable rule evaluations in the old cluster.
Enable them in the new cluster by removing the override you previously set.

Tip
Complete these actions in quick succession to prevent duplicate alerts.

Stop writing to the old cluster

Reconfigure write clients to only send data to the new cluster.

Decommission the old cluster

After the new cluster is fully operational, decommission the old cluster. For example, if you’re using Helm, run the following command:

helm uninstall <OLD_RELEASE_NAME>

Your system runs entirely with ingest storage architecture.

Scale down compactors in the new cluster

After the migration is complete and the old cluster is decommissioned, reduce the number of compactor replicas in the new cluster to your normal steady-state configuration. This lowers resource use after the temporary load from the dual-cluster operation is gone.