Grafana Mimir tools on Grafana Labs

Grafana Mimir listblocks

Wed, 03 Jun 2026 09:01:40 +0200

Grafana Mimir listblocks

The listblocks tool lists blocks and shows the block details of a tenant.

Download and configure listblocks

Download listblocks as part of the Mimir binary. To download the binary, refer to Grafana Mimir releases on GitHub.

To configure listblocks, you need to provide access to the object storage bucket and specify the tenant ID. For example:

mimir listblocks --config.file=/path/to/mimir.yaml --tenant-id=<your-tenant-id>

You can also specify bucket details directly if they aren’t in your configuration file. For example:

mimir listblocks --tenant-id=<your-tenant-id> --blocks-storage.backend=s3 --blocks-storage.s3.bucket-name=<your-bucket> --blocks-storage.s3.endpoint=<your-s3-endpoint>

Use listblocks

Listblocks doesn’t use the bucket index; instead, it downloads the meta.json file of every block in the tenant. This means that listblocks has an up-to-date view of the blocks in the bucket.

$ ./listblocks -backend=gcs -gcs.bucket-name=bucket-with-blocks -user=10428
Block ID                     Min Time               Max Time               Duration
01E0HMK47RGAAKZJBMG8B8QXGP   2020-02-07T07:49:46Z   2020-02-08T00:00:00Z   16h10m13.89s
01E0M0VK2KEDZC5AK1PX8K00EX   2020-02-08T00:00:00Z   2020-02-09T00:00:00Z   24h0m0s
01E0PK9B84XJ9KQ0DHZDQECNH6   2020-02-09T00:00:00Z   2020-02-10T00:00:00Z   24h0m0s
01E0S8VAKJ0H41N41GBKQN4G1N   2020-02-10T00:00:00Z   2020-02-11T00:00:00Z   24h0m0s
01E0VTN88859KW1KTDVBS14E7A   2020-02-11T00:00:00Z   2020-02-12T00:00:00Z   24h0m0s
01E0YCZKFG2ME5GZ60AYCQ39V4   2020-02-12T00:00:00Z   2020-02-13T00:00:00Z   24h0m0s
01E111CX17BXFZD97AKSYKX0A5   2020-02-13T00:00:00Z   2020-02-14T00:00:00Z   24h0m0s
01E13JCZK9A5SJMAY6QSSEB0XX   2020-02-14T00:00:00Z   2020-02-15T00:00:00Z   24h0m0s
01E164EJFPT8ZCY6QWEKNJ0VYX   2020-02-15T00:00:00Z   2020-02-16T00:00:00Z   24h0m0s
...

Listblocks has many options you can use to modify the output. The following list contains the most important listblocks options:

  -format string
    	The format of the output. Must be one of "tabbed", "json", or "yaml" (default "tabbed")
  -max-time value
    	If set, only blocks with MaxTime <= this value is printed
  -min-time value
    	If set, only blocks with MinTime >= this value is printed
  -show-block-size
    	Show size of block based on details in meta.json, if available
  -show-compaction-level
    	Show compaction level
  -show-deleted
    	Show deleted blocks
  -show-labels
    	Show block labels
  -show-parents
    	Show parent blocks
  -show-sources
    	Show compaction sources
  -show-ulid-time
    	Show time from ULID

Example

$ ./listblocks -backend=gcs -gcs.bucket-name=bucket-with-blocks -user=10428 -min-time=2022-02-01T00:00:00Z -max-time=2022-02-04T00:00:00Z -show-labels -show-block-size
Block ID                     Min Time               Max Time               Duration   Size     Labels (excl. __org_id__)
01FTWJ3V2TP7N4D7FCSSBJXQ9Z   2022-02-01T00:00:00Z   2022-02-02T00:00:00Z   24h0m0s    69 GiB   {__compactor_shard_id__="1_of_4"}
01FTWJZ3FD4QX4T1FMJJNP7XR1   2022-02-01T00:00:00Z   2022-02-02T00:00:00Z   24h0m0s    69 GiB   {__compactor_shard_id__="2_of_4"}
01FTWMN7AQBPMXWBHVC61ENPT7   2022-02-01T00:00:00Z   2022-02-02T00:00:00Z   24h0m0s    69 GiB   {__compactor_shard_id__="3_of_4"}
01FTWQ5Y87AWVKXH44T2N23BHW   2022-02-01T00:00:00Z   2022-02-02T00:00:00Z   24h0m0s    69 GiB   {__compactor_shard_id__="4_of_4"}
01FTZ4QWE2PNK69ZJGTK2NCWFB   2022-02-02T00:00:00Z   2022-02-03T00:00:00Z   24h0m0s    73 GiB   {__compactor_shard_id__="1_of_4"}
01FTZ55XAZCVHWP9K5AAR5BVHF   2022-02-02T00:00:00Z   2022-02-03T00:00:00Z   24h0m0s    73 GiB   {__compactor_shard_id__="2_of_4"}
01FTZ7AQBCSBB8T6P2Q5QZ416W   2022-02-02T00:00:00Z   2022-02-03T00:00:00Z   24h0m0s    73 GiB   {__compactor_shard_id__="3_of_4"}
01FTYW42TNTZ44QMM9YTFDE6Y4   2022-02-02T00:00:00Z   2022-02-03T00:00:00Z   24h0m0s    73 GiB   {__compactor_shard_id__="4_of_4"}
01FV1S5GQDAFTQ4M9CTN1CD1E4   2022-02-03T00:00:00Z   2022-02-04T00:00:00Z   24h0m0s    77 GiB   {__compactor_shard_id__="1_of_4"}
01FV1JKPH2VFXA4K6XNETC8FBR   2022-02-03T00:00:00Z   2022-02-04T00:00:00Z   24h0m0s    77 GiB   {__compactor_shard_id__="2_of_4"}
01FV1VQQTAJVA287ZY8DC435HD   2022-02-03T00:00:00Z   2022-02-04T00:00:00Z   24h0m0s    77 GiB   {__compactor_shard_id__="3_of_4"}
01FV1FRX39NC1J64D6H6W9VVZ9   2022-02-03T00:00:00Z   2022-02-04T00:00:00Z   24h0m0s    77 GiB   {__compactor_shard_id__="4_of_4"}

Grafana Mimir tenant injector

Wed, 03 Jun 2026 09:01:40 +0200

Grafana Mimir tenant injector

The tenant injector is a standalone HTTP proxy that injects the X-Scope-OrgID header with the value that you specify in the -tenant-id flag into incoming HTTP requests. It then forwards the modified requests to the URL you specify in the -remote-address flag.

Download the tenant injector

Download the tenant injector as a separate binary from Grafana Mimir. Refer to Grafana Mimir releases on GitHub.

Use the tenant injector

You can use the tenant injector to query data for a tenant during development or troubleshooting.

Usage of tenant-injector:
  -local-address string
    	Local address to listen on (host:port or :port). (default ":8080")
  -remote-address string
    	URL of target to forward requests to to (eg. http://domain.com:80).
  -tenant-id string
    	Tenant ID to inject to proxied requests.

Grafana Mimir query-tee

Wed, 03 Jun 2026 09:01:40 +0200

Grafana Mimir query-tee

The query-tee is a standalone tool that you can use for testing purposes when comparing the query results and performance of two Grafana Mimir clusters. The two Mimir clusters compared by the query-tee must ingest the same series and samples.

The query-tee exposes Prometheus-compatible read API endpoints and acts as a proxy. When the query-tee receives a request, it performs the same request against the two backend Grafana Mimir clusters and tracks the response time of each backend, and compares the query results.

Download the query-tee

Using Docker:

docker pull "grafana/query-tee:latest"

Using a local binary:

Download the appropriate release asset for your operating system and architecture and make it executable.

For Linux with the AMD64 architecture, execute the following command:

curl -Lo query-tee https://github.com/grafana/mimir/releases/latest/download/query-tee-linux-amd64
chmod +x query-tee

Configure the query-tee

The query-tee requires the endpoints of the backend Grafana Mimir clusters. You can configure the backend endpoints by setting the -backend.endpoints flag to a comma-separated list of HTTP or HTTPS URLs.

For each incoming request, the query-tee clones the request and sends it to each configured backend.

Note
You can configure the query-tee proxy listening ports via the -server.http-service-port flag for the HTTP port and server.grpc-service-port flag for the gRPC port.

How the query-tee works

This section describes how the query-tee tool works.

API endpoints

Query-tee accepts two types of requests:

HTTP requests on the configured -server.http-service-port flag (default port 80)
HTTP over gRPC requests on the configured -server.grpc-service-port flag (default port: 9095)

The following Prometheus API endpoints are supported by query-tee:

GET <prefix>/api/v1/query
GET <prefix>/api/v1/query_range
GET <prefix>/api/v1/query_exemplars
GET <prefix>/api/v1/labels
GET <prefix>/api/v1/label/{name}/values
GET <prefix>/api/v1/series
GET <prefix>/api/v1/metadata
GET <prefix>/api/v1/alerts
GET <prefix>/prometheus/config/v1/rules

You can configure the <prefix> by setting the -server.path-prefix flag, which defaults to an empty string.

Pass-through requests

The query-tee can optionally act as a transparent proxy for requests to routes not matching any of the supported API endpoints. You can enable the pass-through support setting -proxy.passthrough-non-registered-routes=true and configuring a preferred backend using the -backend.preferred flag. When pass-through is enabled, a request for an unsupported API endpoint is transparently proxied to the configured preferred backend.

Authentication

The query-tee supports HTTP basic authentication. The query-tee can merge the HTTP basic authentication in the received request with the username and configured in a backend URL.

A request sent from the query-tee to a backend includes HTTP basic authentication when one of the following conditions is met:

If the backend endpoint URL is configured with both a username and password, then query-tee uses it.
If the backend endpoint URL is configured only with a username, then query-tee keeps the configured username and injects the password received in the incoming request.
If the backend endpoint URL is configured without a username and password, then query-tee forwards the authentication credentials found in the incoming request.

Configure the backend

You can configure individual backend behavior using the -backend.config-file flag to specify a YAML or JSON configuration file. Each key in the configuration file corresponds to a backend hostname, and the value contains the configuration for that backend.

The following configuration options are available for each backend:

request_headers: Additional HTTP headers to send to this backend
request_proportion: Proportion of requests to send to this backend. Set between 0.0 -1.0. This value overrides the global -proxy.secondary-backends-request-proportion setting for this backend.
min_data_queried_age: Minimum time threshold for time-based query routing (Go duration format like “24h”, “168h”, “1h30m”). Default is “0s”, which means to serve all queries.

Backend configuration examples

JSON configuration example:

{
  "prometheus-main": {
    "request_headers": {
      "X-Storage-Tier": ["main"]
    }
  },
  "prometheus-hot": {
    "request_headers": {
      "X-Storage-Tier": ["hot"],
      "Cache-Control": ["no-store"]
    },
    "request_proportion": 1.0,
    "min_data_queried_age": "0s"
  },
  "prometheus-cold": {
    "request_headers": {
      "X-Storage-Tier": ["warm"],
      "Cache-Control": ["no-store"]
    },
    "request_proportion": 1.0,
    "min_data_queried_age": "6h"
  }
}

YAML configuration example:

prometheus-main:
  request_headers:
    X-Storage-Tier: ["main"]
prometheus-hot:
  request_headers:
    X-Storage-Tier: ["hot"]
    Cache-Control: ["no-store"]
  request_proportion: 1.0
  min_data_queried_age: "0s" # serves all queries
prometheus-cold:
  request_headers:
    X-Storage-Tier: ["warm"]
    Cache-Control: ["no-store"]
  request_proportion: 1.0
  min_data_queried_age: "6h" # 6 hours

Select backends

You can use the query-tee to either send requests to all backends, or to send a proportion of requests to all backends and the remaining requests to only the preferred backend.

Configure request proportion

You can configure request proportions in two ways:

Global setting: Use the -proxy.secondary-backends-request-proportion CLI flag to set the default proportion for all secondary backends.
Per-backend setting: Use the request_proportion field in the backend configuration file to override the global setting for individual backends.

For example, if you set the -proxy.secondary-backends-request-proportion CLI flag to 1.0, then all requests are sent to all backends. Alternatively, if you set the -proxy.secondary-backends-request-proportion CLI flag to 0.2, then 20% of requests are sent to all backends, and the remaining 80% of requests are sent only to your preferred backend.

Per-backend request proportions take precedence over the global setting. In the previous configuration example, prometheus-warm would receive 80% of requests and prometheus-cold would receive 50% of requests, regardless of the global setting.

Configure time-based routing

You can configure backends to only serve queries based on the time range of the requested data using the min_data_queried_age setting. This is useful for implementing tiered storage architectures where different backends store data for different time periods.

How time-based routing works:

A backend with min_data_queried_age: "24h" only serves queries where the minimum query time is within the last 24 hours.
A backend with min_data_queried_age: "0s", the default, serves all queries regardless of their time range.
The preferred backend is always included regardless of its time threshold.
Range queries, meaning/api/v1/query_range, use earliest time between the start and end parameters.
Instant queries, meaning /api/v1/query use, the time parameter or, if not specified, the current time.

Example:

With the preceding configuration example:

Recent queries (< 6 hours old) are sent to prometheus-main and prometheus-hot only. This excludes prometheus-cold.
Old queries (> 6 hours old) are sent to all backends, meaning prometheus-main, prometheus-hot, and prometheus-cold.

This allows you to route queries to appropriate storage tiers based on data age, optimizing both performance and cost.

Note
The min_data_queried_age field supports Go duration format. Valid time units are ns, us (or µs), ms, s, m, h. Examples: "30s", "15m", "24h", "168h" (7 days), "1h30m". Days are not supported directly; use hours instead (e.g., "168h" for 7 days).

Backend response selection

The query-tee enables you to configure a preferred backend that selects the response to send back to the client. The query-tee returns the Content-Type header, HTTP status code, and body of the response from the preferred backend. The preferred backend can be configured via -backend.preferred=<hostname>. The value of the preferred backend configuration option must be the hostname of one of the configured backends.

When a preferred backend is configured, the query-tee always returns the response from the preferred backend.

When a preferred backend is not configured, the query-tee uses the following algorithm to select the backend response to send back to the client:

If at least one backend response status code is 2xx or 4xx, the query-tee selects the first received response whose status code is 2xx or 4xx.
If no backend response status code is 2xx or 4xx, the query-tee selects the first received response regardless of the status code.

Note
The query-tee considers a 4xx response as a valid response to select because a 4xx status code is generally an invalid request and not a server-side issue.

Backend results comparison

You can use the query-tee to compare query results received from multiple backends. The query results comparison can be enabled setting the flag -proxy.compare-responses=true and requires that:

You’ve configured at least two backends by setting -backend.endpoints.
You’ve configured a preferred backend by setting -backend.preferred.

When you enable the query results comparison, the query-tee compares the response received from the preferred backend against each secondary backend individually and logs a message for each query whose results don’t match. Query-tee keeps track of the number of successful and failed comparison through the metric cortex_querytee_responses_compared_total, with separate metrics for each secondary backend.

By default, query-tee considers equivalent error messages as matching, even if they are not exactly the same. This ensures that comparison does not fail for known situations where error messages are non-deterministic. Set -proxy.compare-exact-error-matching=true to require that error messages match exactly.

Note
Query-tee compares floating point sample values with a tolerance that you can configure with the -proxy.value-comparison-tolerance option.

The configured tolerance prevents false positives due to differences in floating point values rounding introduced by the non-deterministic series ordering within the Prometheus PromQL engine.

Note
The default value of -proxy.compare-skip-recent-samples is two minutes. This means points within results with a timestamp within two minutes of the current time aren’t compared. This prevents false positives due to racing with ingestion, and, if the query selects the output of recording rules, rule evaluation.

If either Mimir cluster is running with a non-default value of -ruler.evaluation-delay-duration, you should set -proxy.compare-skip-recent-samples to one minute more than the value of -ruler.evaluation-delay-duration.

Slow query log

You can configure query-tee to log requests that take longer than the fastest backend by setting the flag -proxy.log-slow-query-response-threshold.

The default value is 10s which logs requests that are ten seconds slower than the fastest backend.

To disable slow query logging, set -proxy.log-slow-query-response-threshold to 0.

Exported metrics

The query-tee exposes the following Prometheus metrics at the /metrics endpoint listening on the port configured via the flag -server.metrics-port:

# HELP cortex_querytee_backend_request_duration_seconds Time (in seconds) spent serving requests.
# TYPE cortex_querytee_backend_request_duration_seconds histogram
cortex_querytee_backend_request_duration_seconds_bucket{backend="<hostname>",method="<method>",route="<route>",status_code="<status>",le="<bucket>"}
cortex_querytee_backend_request_duration_seconds_sum{backend="<hostname>",method="<method>",route="<route>",status_code="<status>"}
cortex_querytee_backend_request_duration_seconds_count{backend="<hostname>",method="<method>",route="<route>",status_code="<status>"}

# HELP cortex_querytee_responses_total Total number of responses sent back to the client by the selected backend.
# TYPE cortex_querytee_responses_total counter
cortex_querytee_responses_total{backend="<hostname>",method="<method>",route="<route>"}

# HELP cortex_querytee_responses_compared_total Total number of responses compared per route name by result.
# TYPE cortex_querytee_responses_compared_total counter
cortex_querytee_responses_compared_total{route="<route>",secondary_backend="<hostname>",result="<success|fail|skip>"}

Additionally, if backend results comparison is configured, two native histograms are available:

cortex_querytee_backend_response_relative_duration_seconds{route="<route>",secondary_backend="<hostname>"}: Time (in seconds) of the secondary backend minus the preferred backend, for each secondary backend.
cortex_querytee_backend_response_relative_duration_proportional{route="<route>",secondary_backend="<hostname>"}: Response time of the secondary backend minus the preferred backend, as a proportion of the preferred backend response time.

Ruler remote operational mode test

When the ruler is configured with the remote evaluation mode you can use the query-tee to compare rule evaluations too. To test ruler evaluations with query-tee, set the -ruler.query-frontend.address CLI flag or its respective YAML configuration parameter for the ruler with query-tee’s gRPC address:

ruler:
  query_frontend:
    address: "dns://query-tee:9095"

When the ruler evaluates a rule, the test flow is the following:

ruler sends gRPC request to query-tee
query-tee forwards the request to the query-frontend backends configured setting the -backend.endpoints CLI flag
query-tee receives the response from the query-frontend and forwards the result (based on the preferred backend) to the ruler

Grafana mimir-continuous-test

Wed, 03 Jun 2026 09:01:40 +0200

Grafana mimir-continuous-test

As a developer, you can use the mimir-continuous-test tool to run smoke tests on live Grafana Mimir clusters. This tool identifies a class of bugs that could be difficult to spot during development. Two operating modes are supported:

As a continuously running deployment in your environment, mimir-continuous-test can be used to detect issues on a live Grafana Mimir cluster over time.
As an ad-hoc smoke test tool, mimir-continuous-test can be used to validate basic functionality after configuration changes are made to a Grafana Mimir cluster.

Download and run mimir-continuous-test

Using Docker:

docker pull "grafana/mimir:latest"
docker run --rm -ti grafana/mimir -target=continuous-test

Using a local binary:

Download the appropriate mimir binary for your operating system and architecture, and make it executable.

For Linux with the AMD64 architecture, execute the following command:

curl -Lo mimir https://github.com/grafana/mimir/releases/latest/download/mimir-linux-amd64
chmod +x mimir
mimir -target=continuous-test

Note
In Grafana Mimir 2.13 the mimir-continuous-test became a part of Mimir as its own target.

The standalone grafana/mimir-continuous-test Docker image and mimir-continuous-test binary were deprecated in 2.13 and removed in 3.0.

Configure mimir-continuous-test

Mimir-continuous-test requires the endpoints of the backend Grafana Mimir clusters and the authentication for writing and querying testing metrics:

Set -tests.write-endpoint to the base endpoint on the write path. Remove any trailing slash from the URL. The tool appends the specific API path to the URL, for example /api/v1/push for the remote-write API.
Set -tests.read-endpoint to the base endpoint on the read path. Remove any trailing slash from the URL. The tool appends the specific API path to the URL, for example /api/v1/query_range for the range-query API.
Set the authentication means to use to write and read metrics in tests. By priority order:
- -tests.bearer-token for bearer token authentication.
- -tests.basic-auth-user and -tests.basic-auth-password for a basic authentication.
- -tests.tenant-id to the tenant ID, default to anonymous.
Set -tests.smoke-test to run the test once and immediately exit. In this mode, the process exit code is non-zero when the test fails.

Note
You can run mimir -help to list all available configuration options. All configuration options for mimir-continuous-test begin with tests.

How it works

Mimir-continuous-test periodically runs a suite of tests, writes data to Mimir, queries that data back, and checks if the query results match what is expected. The tool exposes metrics that you can use to alert on test failures. The tool logs the details about the failed tests.

Exported metrics

Mimir-continuous-test exposes the following Prometheus metrics at the /metrics endpoint listening on the port that you configured via the flag -server.metrics-port:

# HELP mimir_continuous_test_writes_total Total number of attempted write requests.
# TYPE mimir_continuous_test_writes_total counter
mimir_continuous_test_writes_total{test="<name>"}
{test="<name>"}

# HELP mimir_continuous_test_writes_failed_total Total number of failed write requests.
# TYPE mimir_continuous_test_writes_failed_total counter
mimir_continuous_test_writes_failed_total{test="<name>",status_code="<code>"}

# HELP mimir_continuous_test_writes_request_duration_seconds Duration of the requests
# TYPE mimir_continuous_test_writes_request_duration_seconds histogram
mimir_continuous_test_writes_request_duration_seconds{test="<name>"}

# HELP mimir_continuous_test_queries_total Total number of attempted query requests.
# TYPE mimir_continuous_test_queries_total counter
mimir_continuous_test_queries_total{test="<name>"}

# HELP mimir_continuous_test_queries_failed_total Total number of failed query requests.
# TYPE mimir_continuous_test_queries_failed_total counter
mimir_continuous_test_queries_failed_total{test="<name>"}

# HELP mimir_continuous_test_query_result_checks_total Total number of query results checked for correctness.
# TYPE mimir_continuous_test_query_result_checks_total counter
mimir_continuous_test_query_result_checks_total{test="<name>"}

# HELP mimir_continuous_test_query_result_checks_failed_total Total number of query results failed when checking for correctness.
# TYPE mimir_continuous_test_query_result_checks_failed_total counter
mimir_continuous_test_query_result_checks_failed_total{test="<name>"}

# HELP mimir_continuous_test_queries_request_duration_seconds Duration of the requests
# TYPE mimir_continuous_test_queries_request_duration_seconds histogram
mimir_continuous_test_queries_request_duration_seconds{results_cache="false",test="<name>"}
mimir_continuous_test_queries_request_duration_seconds{results_cache="true",test="<name>"}

Note
The metrics mimir_continuous_test_writes_request_duration_seconds and mimir_continuous_test_queries_request_duration_seconds are exposed as native histograms and don’t have proper textual presentation on the /metrics endpoint.

Alerts

Grafana Mimir alerts include checks on failures that mimir-continuous-test tracks. When running mimir-continuous-test, use the provided alerts.

Grafana Mimirtool

Wed, 03 Jun 2026 09:01:40 +0200

Grafana Mimirtool

Mimirtool is a command-line tool that operators and tenants can use to execute a number of common tasks that involve Grafana Mimir or Grafana Cloud Metrics.

The alertmanager command enables you to create, update, and delete tenant configurations in Grafana Mimir Alertmanager or Grafana Cloud Metrics.

For more information about the alertmanager command, refer to Alertmanager.
The rules command enables you to validate and lint Prometheus rule files and convert them for use in Grafana Mimir. You can also create, update, and delete rulegroups in Grafana Mimir or Grafana Cloud Metrics.

For more information about the rules command, refer to Rules.
The remote-read subcommand enables you to fetch statistics and series from remote-read APIs. You can write series from a remote-read API to a local TSDB file that you load into Prometheus.

For more information about the remote-read command, refer to Remote-read.
The analyze command extracts statistics about metric usage from Grafana or Hosted Grafana instances. You can also extract the same metrics from Grafana dashboard JSON files or Prometheus rule YAML files.

For more information about the analyze command, refer to Analyze.
The bucket-validation command verifies that an object storage bucket is suitable as a backend storage for Grafana Mimir.

For more information about the bucket-validation command, refer to Bucket validation.
The acl command generates the label-based access control header used in Grafana Enterprise Metrics and Grafana Cloud Metrics.

For more information about the acl command, refer to ACL.
The config command helps convert configuration files from Cortex to Grafana Mimir.

For more information about the config command, refer to Config
The backfill command uploads existing Prometheus TSDB blocks into Grafana Mimir.

For more information about the backfill command, refer to Backfill
The partition-ring commands allow for forcefully modifying the ingest storage partition ring via memberlist.

For more information about the partition-ring command, refer to Partition ring

Mimirtool interacts with:

User-facing APIs provided by Grafana Mimir.
Backend storage components containing Grafana Mimir data.

Installation

To install Mimirtool with Homebrew on macOS, run brew install mimirtool.

To install Mimirtool in other environments, download the appropriate binary from the latest release for your operating system and architecture and make it executable.

Alternatively, use a command line tool such as curl to download mimirtool. For example, for Linux with the AMD64 architecture, use the following command:

curl -fLo mimirtool https://github.com/grafana/mimir/releases/latest/download/mimirtool-linux-amd64
chmod +x mimirtool

Configuration options

For Mimirtools to interact with Grafana Mimir, Grafana Enterprise Metrics, Prometheus, or Grafana, set the following environment variables or CLI flags.

Environment variable	Flag	Description
`MIMIR_ADDRESS`	`--address`	Sets the address of the API of the Grafana Mimir cluster.
`MIMIR_API_USER`	`--user`	Sets the basic authentication username. If this variable is empty and `MIMIR_API_KEY` is set, the system uses `MIMIR_TENANT_ID` instead. If you’re using Grafana Cloud, this variable is your instance ID.
`MIMIR_API_KEY`	`--key`	Sets the basic authentication password. If you’re using Grafana Cloud, this variable is your API key.
`MIMIR_TENANT_ID`	`--id`	Sets the tenant ID of the Grafana Mimir instance that Mimirtool interacts with.
`MIMIR_EXTRA_HEADERS`	`--extra-headers`	Extra headers to add to the requests in header=value format. You can specify this flag multiple times. You must newline separate environment values.

It is also possible to set TLS-related options with the following environment variables or CLI flags:

Environment variable	Flag	Description
`MIMIR_TLS_CA_PATH`	`--tls-ca-path`	Sets the path to the CA certificate to use to verify the connection to the Grafana Mimir cluster.
`MIMIR_TLS_CERT_PATH`	`--tls-cert-path`	Sets the path to the client certificate to use to authenticate to the Grafana Mimir cluster.
`MIMIR_TLS_KEY_PATH`	`--tls-key-path`	Sets the path to the private key to use to authenticate to the Grafana Mimir cluster.
`MIMIR_TLS_INSECURE_SKIP_VERIFY`	`--tls-insecure-skip-verify`	If `true`, disables verification of the Grafana Mimir cluster’s TLS certificate. This is insecure and not recommended.

Commands

The following sections outline the commands that you can run against Grafana Mimir and Grafana Cloud Metrics.

Alertmanager

The following commands interact with Grafana Mimir Alertmanager configuration and alert template files.

Note
The Mimir Alertmanager exposes the configuration API based on the path that you set via the -server.path-prefix flag; it doesn’t use the path of the -http.alertmanager-http-prefix flag.

If you run Mimir with the default configuration, -server.path-prefix, where the default value is /, then only set the hostname for the --address flag of the mimirtool command; do not set a path-specific address.

For example, / is correct, and /alertmanager is incorrect.

Get Alertmanager configuration

The following command shows the current Alertmanager configuration.

mimirtool alertmanager get

Alternatively, you can output the config and template files to a folder which can then be loaded back into alert manager at a later date. For example, the following command outputs the files to a folder called am

mimirtool alertmanager get --output-dir="am"

The config file is named config.yaml and the template files ends in .tpl, where each template is written out to its own file. Note that using the --output-dir flag only writes the output to files and no longer print the config to the console.

Load Alertmanager configuration

The following command loads an Alertmanager configuration to the Alertmanager instance.

mimirtool alertmanager load <config_file>
mimirtool alertmanager load <config_file> <template_files>...

Example

mimirtool alertmanager load ./example_alertmanager_config.yaml ./example_alertmanager_template.tpl

./example_alertmanager_config.yaml:

route:
  receiver: "example_receiver"
  group_by: ["example_groupby"]
receivers:
  - name: "example_receiver"
templates:
  - example_template.tpl

./example_alertmanager_template.tpl:

{{ define "alert_customer_env_message" }}
  [{{ .CommonLabels.alertname }} | {{ .CommonLabels.customer }} | {{ .CommonLabels.environment }}]
{{ end }}

The input to [<template_files>...] accepts wildcard, i.e. *.tpl will include all the template files ending in .tpl. If we have used the previous command of exporting the config and templates to a directory, they can be loaded

# assuming we have written the files out to the folder am
# mimirtool alertmanager get --output-dir="am"
mimirtool alertmanager load  am/config.yaml am/*.tpl

Delete Alertmanager configuration

The following command deletes the Alertmanager configuration in the Grafana Mimir Alertmanager.

mimirtool alertmanager delete

Migrate Alertmanager configuration for UTF-8 in Mimir 2.12 and later

This requires mimirtool version 2.12 or later. To check your version of mimirtool, run mimirtool version.

In accordance with prometheus/prometheus#13095 and prometheus/alertmanager#3486, Mimir is adding support for UTF-8. To support UTF-8 in alerts, routes, silences, and inhibition rules, Alertmanager has added a new parser for matchers that has a number of backwards incompatible changes. More information about these changes can be found here.

The migrate-utf8 command migrates an existing Alertmanager configuration in preparation for when UTF-8 is enabled in a Mimir installation. To do this, it translates matchers that are incompatible with UTF-8 into equivalent matchers that are compatible. This translation is backwards compatible, does not change the behavior of existing matchers, and works even in Mimir installations that do not have UTF-8 enabled.

The need to migrate an Alertmanager configuration depends on whether it contains matchers incompatible with UTF-8. You can verify if an existing Alertmanager configuration needs to be migrated for UTF-8 by running the verify command:

mimirtool alertmanager verify <config_file> [template_files...]

If the command succeeds without any warnings, the configuration is compatible with UTF-8 and you don’t need to migrate it.

However, if the command prints the following warning, then you need to migrate the Alertmanager configuration with the migrate-utf8 command.

Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible. Alertmanager has instead parsed the input using the classic matchers parser as a fallback. To make this input compatible with the UTF-8 matchers parser please make sure all regular expressions and values are double-quoted. If you are still seeing this message please open an issue.

The command takes as input an existing configuration and template files, and prints as output the migrated configuration and template files:

mimirtool alertmanager migrate-utf8 <config_file> [template_files...]

You can also output the migrated configuration and template files to a folder which can be reviewed before being loaded back into the Alertmanager at a later time. We recommended you output the migrated files to a different folder than the original files so you always have the original files as a backup. For example, the following command outputs the migrated configuration and template files to a folder called migrated:

mimirtool alertmanager migrate-utf8 <config_file> [template_files...] --output-dir="migrated"

Within the output dir, the configuration file is named config.yaml and the template files ends in .tpl, where each template is written out to its own file.

Note
When you use the --output-dir flag, the command only writes the output to files and doesn’t print the configuration to the console.

After you migrate an Alertmanager configuration, use the verify command to verify it:

mimirtool alertmanager verify <config_file> [template_files...]

It should succeed without warnings.

You can also run the verify command with the --utf8-strict-mode flag to verify if an Alertmanager configuration is compatible with UTF-8:

mimirtool alertmanager verify <config_file> [template_files...] --utf8-strict-mode

When using this flag, the verify command prints a warning with the text UTF-8 mode enabled to let you know the command is using UTF-8 strict mode to validate the Alertmanager configuration. Unlike when the flag is omitted, it doesn’t print warnings if a configuration is incompatible with UTF-8. Instead, the command exits with an error. For example:

level=warn msg="UTF-8 strict mode enabled"
mimirtool: error: end of input: expected label value, try --help

If the command exits without error, and you’re satisfied with the changes made to the migrated configuration and template files, reload it using the load command.

Validate Alertmanager configuration

The following command validates an Alertmanager configuration file. It does not load the configuration to the Alertmanager instance.

mimirtool alertmanager verify <config_file> [template_files...]

Alert verification

The following command verifies if alerts in an Alertmanager cluster are deduplicated. This command is useful for verifying the correct configuration when transferring from Prometheus to Grafana Mimir alert evaluation.

mimirtool alerts verify

Render template

You can render your Alertmanager template with the render command.

The argument to the --template.glob option is a glob that expands to the alert template paths.

The argument to the --template.data option is a JSON file containing Alertmanager template data.

The argument to the --template.text option is a Go template. The command renders this template with the templates found be expanding --template-glob.

The following command render a template and prints it to the terminal. It assumes you have written the templates to the directory templates.

mimirtool alertmanager render --template.glob 'templates/*' --template.data alert_data1.json --template.text '{{ template "my_message" . }}'

Rules

The rules command features sub-commands for working with Prometheus rule files and with the APIs in the Grafana Mimir ruler.

The commands in this section enable you to perform the following actions:

Load and show Prometheus rule files
Interact with individual rule groups in the Mimir ruler
Manipulate local rule files

Some of the functionality that mimirtool rules offers is also available as a GitHub Action. For more information, refer to the documentation of Mimirtool Github Action.

Configuration options relevant to rules commands:

Flag	Description
`--auth-token`	Authentication token for bearer token or JWT authentication.
`--enable-experimental-functions`	If set, enables parsing experimental PromQL functions.

List rules

The following command retrieves the names of all rule groups in the Grafana Mimir instance and prints them to the terminal.

mimirtool rules list

Print rules

The following command retrieves all rule groups in the Grafana Mimir instance and prints them to the terminal.

mimirtool rules print

To save all the rules for editing and re-upload to Mimir, use the --output-dir option. The default output directory is the current directory. The output file has the format required by mimirtool rules load or mimirtool rules sync.

For example, to save the file in the rules subdirectory:

mimirtool rules print --output-dir=rules

Get rule group

The following command retrieves a single rule group and prints it to the terminal.

mimirtool rules get <namespace> <rule_group_name>

To save the rule group for editing and re-upload to Mimir, use the --output-dir option. The default output directory is the current directory. The output file has the format required by mimirtool rules load or mimirtool rules sync.

For example, to save the file in the rules subdirectory:

mimirtool rules get <namespace> <rule_group_name> --output-dir=rules

Delete rule group

The following command deletes a rule group.

mimirtool rules delete <namespace> <rule_group_name>

Load rule group

The following command loads each rule group from the files into Grafana Mimir. This command overwrites all existing rule groups with the same name.

mimirtool rules load <file_path>...

Example

mimirtool rules load ./example_rules_one.yaml

./example_rules_one.yaml:

namespace: my_namespace
groups:
  - name: example
    interval: 5m
    rules:
      - record: job:http_inprogress_requests:sum
        expr: sum by (job) (http_inprogress_requests)

This command, like the other rules subcommands, can load multiple rule groups at once:

mimirtool rules load ./example_rules_one.yaml ./example_rules_two.yaml

Delete a namespace

The following command deletes all of the rule groups in a namespace, including the namespace itself:

mimirtool rules delete-namespace <namespace>

Lint

The lint command provides YAML and PromQL expression formatting within the rule file.

This command edits the rule file in place. To perform a trial run that does not make changes, you can use the dry run flag (-n).

Note
This command doesn’t verify if a query is correct and doesn’t interact with your Grafana Mimir cluster.

mimirtool rules lint <file_path>...

The format of the file is the same format as shown in rules load.

Prepare

This prepare command prepares a rules file that you upload to Grafana Mimir. It lints all PromQL expressions and adds a label to your PromQL query aggregations in the file. The format of the file is the same format as shown in rules load.

Note
This command doesn’t interact with your Grafana Mimir cluster.

mimirtool rules prepare <file_path>...

Configuration

Flag	Description
`-i`, `--in-place`	Edits the file in place. If not set, the system generates a new file with the extension `.result` that contains the results.
`-l`, `--label="cluster"`	Specifies the label for aggregations. By default, the label is set to `cluster`.
`--label-excluded-rule-groups`	Comma separated list of rule group names to exclude when including the configured label to aggregations.
`--rule-dirs`	Comma separated list of paths to directories containing rules yaml files. Each file in a directory with a .yml or .yaml suffix will be parsed.

Example

mimirtool rules prepare ./example_rules_one.yaml

./example_rules_one.yaml:

namespace: my_namespace
groups:
  - name: example
    interval: 5m
    rules:
      - record: job:http_inprogress_requests:sum
        expr: sum by (job) (http_inprogress_requests)

./example_rules_one.yaml.result:

namespace: my_namespace
groups:
  - name: example
    interval: 5m
    rules:
      - record: job:http_inprogress_requests:sum
        # note the added cluster label
        expr: sum by(job, cluster) (http_inprogress_requests)

After the command runs, an output message indicates if the operation was successful:

INFO[0000] SUCCESS: 1 rules found, 0 modified expressions

Check

The check command checks rules against the recommended best practices for rules. This command does not interact with your Grafana Mimir cluster.

mimirtool rules check <file_path>...

Example

mimirtool rules check rules.yaml

rules.yaml

namespace: my_namespace
groups:
  - name: example
    interval: 5m
    rules:
      - record: job_http_inprogress_requests_sum
        expr: sum by (job) (http_inprogress_requests)

ERRO[0000] bad recording rule name error="recording rule name does not match level:metric:operation format, must contain at least one colon" file=rules.yaml rule=job_http_inprogress_requests_sum ruleGroup=example

The format of the file is the same format as shown in rules load.

Diff

The following command compares rules against the rules in your Grafana Mimir cluster.

mimirtool rules diff <file_path>...

The format of the file is the same format as shown in rules load.

To restrict the affected namespaces, use the --namespaces and --ignore-namespaces parameters as well as their RegEx variants --namespaces-regex and --ignore-namespaces-regex.

Configuration

Flag	Description
`--namespaces`	comma-separated list of namespaces to check during a diff
`--ignored-namespaces`	comma-separated list of namespaces to ignore during a diff
`--namespaces-regex`	regex matching namespaces to check during a diff
`--ignored-namespaces-regex`	regex matching namespaces to ignore during a diff

Only one of the namespace selection flags can be specified.

Sync

The sync command compares rules against the rules in your Grafana Mimir cluster. The command applies any differences to your Grafana Mimir cluster.

mimirtool rules sync <file_path>...

The format of the file is the same format as shown in rules load.

Configuration

Flag	Description
`--namespaces`	comma-separated list of namespaces to check during a sync
`--ignored-namespaces`	comma-separated list of namespaces to ignore during a sync
`--namespaces-regex`	regex matching namespaces to check during a sync
`--ignored-namespaces-regex`	regex matching namespaces to ignore during a sync

Only one of the namespace selection flags can be specified.

Remote-read

Grafana Mimir exposes a remote read API which allows the system to access the stored series. The remote-read subcommand mimirtool enables you to interact with its API, and to determine which series are stored.

The remote-read commands support multiple --selector flags to query multiple series selectors in a single request, leveraging the remote read protocol’s native batching capabilities for improved performance. When multiple selectors are specified, their results are combined into a single unified output.

Additionally, you can control the response format using the --use-chunks flag:

--use-chunks=true (default): Requests chunked streaming response for better performance with large datasets
--use-chunks=false: Requests traditional sampled response format

Stats

The remote-read stats command summarizes statistics of the stored series that match the specified selector(s).

Example

# Single selector
mimirtool remote-read stats --selector '{job="node"}' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read

# Multiple selectors in a single request
mimirtool remote-read stats --selector '{job="node"}' --selector 'up' --selector 'go_memstats_alloc_bytes' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read

Running the command results in the following output:

INFO[0000] Create remote read client using endpoint 'http://demo.robustperception.io:9090/api/v1/read'
INFO[0000] Querying time from=2020-12-30T14:00:00Z to=2020-12-30T15:00:00Z with selector={job="node"}
INFO[0000] MIN TIME                           MAX TIME                           DURATION     NUM SAMPLES  NUM SERIES   NUM STALE NAN VALUES  NUM NAN VALUES
INFO[0000] 2020-12-30 14:00:00.629 +0000 UTC  2020-12-30 14:59:59.629 +0000 UTC  59m59s       159480       425          0                     0

Dump

The remote-read dump command prints all series and samples that match the specified selector(s).

Example

# Single selector
mimirtool remote-read dump --selector 'up{job="node"}' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read

# Multiple selectors
mimirtool remote-read dump --selector 'up{job="node"}' --selector 'go_memstats_alloc_bytes' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read

Running the command results in the following output:

{__name__="up", instance="demo.robustperception.io:9100", job="node"} 1 1609336914711
{__name__="up", instance="demo.robustperception.io:9100", job="node"} NaN 1609336924709 # StaleNaN
...

Export

The remote-read export command exports all series and samples that match the specified selector(s) into a local TSDB. You can use local tooling such as prometheus and promtool to further analyze the TSDB.

# Use Remote Read API to download all metrics with label job=node into local tsdb
mimirtool remote-read export --selector '{job="node"}' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read --tsdb-path ./local-tsdb

# Download multiple metric families in a single request
mimirtool remote-read export --selector '{job="node"}' --selector 'up' --selector 'prometheus_build_info' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read --tsdb-path ./local-tsdb

Running the command results in the following output:

INFO[0000] Create remote read client using endpoint 'http://demo.robustperception.io:9090/api/v1/read'
INFO[0000] Created TSDB in path './local-tsdb'
INFO[0000] Using existing TSDB in path './local-tsdb'
INFO[0000] Querying time from=2020-12-30T13:53:59Z to=2020-12-30T14:53:59Z with selector={job="node"}
INFO[0001] Store TSDB blocks in './local-tsdb'
INFO[0001] BLOCK ULID                  MIN TIME                       MAX TIME                       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
INFO[0001] 01ETT28D6B8948J87NZXY8VYD9  2020-12-30 13:53:59 +0000 UTC  2020-12-30 13:59:59 +0000 UTC  6m0.001s     15950        429          425          105KiB867B
INFO[0001] 01ETT28D91Z9SVRYF3DY0KNV41  2020-12-30 14:00:00 +0000 UTC  2020-12-30 14:53:58 +0000 UTC  53m58.001s   143530       1325         425          509KiB679B

Local TSDB examples

The following command uses promtool to analyze file contents.

promtool tsdb analyze ./local-tsdb

The following command dumps all values of the TSDB.

promtool tsdb dump ./local-tsdb

The following command runs a local Prometheus with the local TSDB.

prometheus --storage.tsdb.path ./local-tsdb --config.file=<(echo "")

ACL

The acl command generates the label-based access control header used in Grafana Enterprise Metrics and Grafana Cloud Metrics.

Generate header

The following command enables you to generate a header that you can use to enforce access control rules in Grafana Enterprise Metrics or Grafana Cloud.

Note
Grafana Mimir doesn’t support ACLs.

mimirtool acl generate-header --id=<tenant_id> --rule=<promql_selector>

Example

mimirtool acl generate-header --id=1234 --rule='{namespace="A"}'

Example output:

The header to set:
X-Prom-Label-Policy: 1234:%7Bnamespace=%22A%22%7D

Analyze

You can analyze your Grafana or Hosted Grafana instance to determine which metrics are used and exported. You can also extract metrics from dashboard JSON files and rules YAML files.

Configuration options relevant to analyze commands:

Environment variable	Flag	Description
-	`--enable-experimental-functions`	Enables parsing PromQL queries containing experimental functions.

Grafana

The following command runs against your Grafana instance, downloads its dashboards, and extracts the Prometheus metrics used in its queries. The output is a JSON file. You can use this file with analyze prometheus --grafana-metrics-file.

mimirtool analyze grafana --address=<url>

Configuration

Environment variable	Flag	Description
`GRAFANA_ADDRESS`	`--address`	Sets the address of the Grafana instance.
`GRAFANA_API_KEY`	`--key`	Sets the API Key for the Grafana instance. To create a key, refer to Authentication API.
-	`--output`	Sets the output file path, which by default is `metrics-in-grafana.json`.
-	`--folder-title`	Sets the folder filter to limit dashboards analysis for unused metrics based on their exact folder title. When repeated any of the matching folders will be analyzed.

Example output file

{
  "metricsUsed": [
    "apiserver_request:availability30d",
    "workqueue_depth",
    "workqueue_queue_duration_seconds_bucket"
  ],
  "dashboards": [
    {
      "slug": "",
      "uid": "09ec8aa1e996d6ffcd6817bbaff4db1b",
      "title": "Kubernetes / API server",
      "metrics": [
        "apiserver_request:availability30d",
        "apiserver_request_total",
        "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile",
        "workqueue_depth",
        "workqueue_queue_duration_seconds_bucket"
      ],
      "parse_errors": ["unsupported panel type: \"news\""]
    }
  ]
}

Ruler

The following command runs against your Grafana Mimir, Grafana Enterprise Metrics, or Grafana Cloud Prometheus instance. The command fetches the rule groups and extracts the Prometheus metrics used in the rule queries. The output is a JSON file. You can use this file with analyze prometheus --ruler-metrics-file.

mimirtool analyze ruler --address=<url> --id=<tenant_id>

Configuration

Environment variable	Flag	Description
`MIMIR_ADDRESS`	`--address`	Sets the address of the Prometheus instance.
`MIMIR_TENANT_ID`	`--id`	Sets the basic authentication username. If you’re using Grafana Cloud, this variable is your instance ID.
`MIMIR_API_KEY`	`--key`	Sets the basic authentication password. If you’re using Grafana Cloud, this variable is your API key.
`MIMIR_AUTH_TOKEN`	`--auth-token`	Sets the bearer or JWT token that is required for Mimir clusters authenticating with this method.
-	`--output`	Sets the output file path, which by default is `metrics-in-ruler.json`.

Example output file

{
  "metricsUsed": [
    "apiserver_request_duration_seconds_bucket",
    "container_cpu_usage_seconds_total",
    "scheduler_scheduling_algorithm_duration_seconds_bucket"
  ],
  "ruleGroups": [
    {
      "namspace": "prometheus_rules",
      "name": "kube-apiserver.rules",
      "metrics": [
        "apiserver_request_duration_seconds_bucket",
        "apiserver_request_duration_seconds_count",
        "apiserver_request_total"
      ],
      "parse_errors": null
    }
  ]
}

Dashboard

The following command accepts Grafana dashboard JSON files as input and extracts Prometheus metrics used in the queries. The output is a JSON file. You can use the output file with analyze prometheus --grafana-metrics-file.

mimirtool analyze dashboard <file>...

Configuration

Environment variable	Flag	Description
-	`--output`	Sets the output file path, which by default is `metrics-in-grafana.json`.

Rule-file

The following command accepts Prometheus rule YAML files as input and extracts Prometheus metrics used in the queries. The output is a JSON file. You can use the output file with analyze prometheus --ruler-metrics-file.

mimirtool analyze rule-file <file>

Configuration

Environment variable	Flag	Description
-	`--output`	Sets the output file path, which by default is `metrics-in-ruler.json`.

Prometheus

The following command runs against your Grafana Mimir, Grafana Metrics Enterprise, Prometheus, or Cloud Prometheus instance. The command uses the output from a previous run of analyze grafana, analyze dashboard, analyze ruler or analyze rule-file to show the number of series in the Prometheus instance that are used in dashboards or rules, or both. This command also shows which metrics exist in Grafana Cloud that are not in dashboards or rules. The output is a JSON file.

Note
The command makes a request for every active series in the Prometheus instance. For Prometheus instances with a large number of active series, this command might take time to complete.

mimirtool analyze prometheus --address=<url> --id=<tenant_id>

Configuration

Environment variable	Flag	Description
`MIMIR_ADDRESS`	`--address`	Sets the address of the Prometheus instance.
`MIMIR_TENANT_ID`	`--id`	Sets the basic authentication username. If you’re using Grafana Cloud this variable is your instance ID, also set as tenant ID.
`MIMIR_API_KEY`	`--key`	Sets the basic authentication password. If you’re using Grafana Cloud, this variable is your API key.
`MIMIR_AUTH_TOKEN`	`--auth-token`	Sets the bearer or JWT token that is required for Mimir clusters authenticating with this method.
-	`--grafana-metrics-file`	`mimirtool analyze grafana` or `mimirtool analyze dashboard` output file, which by default is `metrics-in-grafana.json`.
-	`--ruler-metrics-file`	`mimirtool analyze ruler` or `mimirtool analyze rule-file` output file, which by default is `metrics-in-ruler.json`.
-	`--output`	Sets the output file path, which by default is `prometheus-metrics.json`.
-	`--prometheus-http-prefix`	Sets the HTTP URL path under which the Prometheus api will be served.

Example output

{
  "total_active_series": 38184,
  "in_use_active_series": 14047,
  "additional_active_series": 24137,
  "in_use_metric_counts": [
    {
      "metric": "apiserver_request_duration_seconds_bucket",
      "count": 11400,
      "job_counts": [
        {
          "job": "apiserver",
          "count": 11400
        }
      ]
    },
    {
      "metric": "apiserver_request_total",
      "count": 684,
      "job_counts": [
        {
          "job": "apiserver",
          "count": 684
        }
      ]
    }
  ],
  "additional_metric_counts": [
    {
      "metric": "etcd_request_duration_seconds_bucket",
      "count": 2688,
      "job_counts": [
        {
          "job": "apiserver",
          "count": 2688
        }
      ]
    }
  ]
}

Bucket validation

The following command validates that the object store bucket works correctly.

mimirtool bucket-validation

Flag	Description
`--object-count`	Sets the number of objects to create and delete. By default, the value is 2000.
`--report-every`	Sets the number operations afterwhich an operations progress report is printed. By default, the value is 100.
`--test-runs`	Sets the number of times to run the test. By default, the value is 1.
`--prefix`	Sets the path prefix to use for test objects in the object store.
`--retries-on-error`	Sets the number of times to retry if the object store returns an error.
`--bucket-config`	Sets the CLI arguments to configure a storage bucket.
`--bucket-config-help`	Displays help text that explains how to use the -bucket-config parameter.

Config

Convert

The config convert command converts configuration parameters that work with Cortex v1.10.0 and above to parameters that work with Grafana Mimir v2.0.0. It supports converting both CLI flags and YAML configuration files.

Configuration

Flag	Description
`--yaml-file`	The YAML configuration file to convert.
`--flags-file`	A file containing a newline-delimited list of CLI flags to convert.
`--yaml-out`	File to use for the converted YAML configuration. If not set, output to `stdout`.
`--flags-out`	File to use for list of converted CLI flags. If not set, output to `stdout`.
`--update-defaults`	If you set this flag and you set a configuration parameter to a default value that has changed in Mimir 2.0, the parameter updates to the new default value.
`--include-defaults`	If you set this flag, all default values are included in the output YAML, regardless of whether you explicitly set the values in the input files.
`-v`, `--verbose`	If you set this flag, the CLI flags and YAML paths from the old configuration that do not exist in the new configuration are printed to `stderr`. This flag also prints default values that have changed between the old and the new configuration.
`--gem`	If you set this flag, the tool will convert from Grafana Metrics Enterprise (GEM) v1.7.x to v2.0.0.

Changes to default values

mimirtool config convert helps you migrate from Cortex to Grafana Mimir. There are changes to the default values of some configuration parameters in Mimir v2.0.0 that you might not want to use as part of this migration:

Parameter	Cortex/GEM 1.7 default	Mimir/GEM 2.0 default
`blocks_storage.backend`	`s3`	`filesystem`
`ruler_storage.backend`	`s3`	`filesystem`
`alertmanager_storage.backend`	`s3`	`filesystem`
`server.http_listen_port`	`80`	`8080`
`activity_tracker.filepath`	`./active-query-tracker`	`./metrics-activity.log`
`alertmanager.data_dir`	`data/`	`./data-alertmanager/`
`blocks_storage.filesystem.dir`	`<empty>`	`blocks`
`compactor.data_dir`	`./data`	`./data-compactor/`
`ruler.rule_path`	`/rules`	`./data-ruler/`
`ruler_storage.filesystem.dir`	`<empty>`	`ruler`
(GEM only) `auth.type`	`trust`	`enterprise`
(GEM only) `graphite.querier.schemas.backend`	`s3`	`filesystem`

For these parameters mimirtool config convert will output the Cortex default value even when the configuration parameter is not explicitly set in the input configuration. If in the input configuration you explicitly set the Cortex default value, and you have provided the --update-defaults flag, mimirtool config convert will not update the value to the Mimir default.

Example

The following example shows a command that converts Cortex query-frontend YAML configuration file and CLI flag to a Mimir-compatible YAML and CLI flag.

mimirtool config convert --yaml-file=cortex.yaml --flags-file=cortex.flags --yaml-out=mimir.yaml --flags-out=mimir.flags

cortex.yaml input file:

query_range:
  results_cache:
    cache:
      memcached:
        expiration: 10s # Expiration was removed in Grafana Mimir, so this parameter will be missing from the output YAML
        batch_size: 2048
        parallelism: 10
      memcached_client:
        max_idle_conns: 32

cortex.flags input file:

-frontend.background.write-back-concurrency=45

After you run the command, the converted output should be:

mimir.yaml converted output file:

frontend:
  results_cache:
    memcached:
      max_get_multi_batch_size: 2048
      max_get_multi_concurrency: 10
      max_idle_connections: 32

server:
  http_listen_port: 80

Note
As a precaution,the output file includes server.http_listen_port. The default value in Grafana Mimir changed from 80 to 8080.

Unless you explicitly set the port in the input configuration, the tool outputs the original default value.

mimir.flags converted output file:

-query-frontend.results-cache.memcached.max-async-concurrency=45

Verbose output

When you set the --verbose flag, the output explains which configuration parameters were removed and which default values were changed. The verbose output is printed to stderr.

The output includes the following entries:

field is no longer supported: <yaml_path>

This parameter was used in the input Cortex YAML file and removed from the output configuration.
flag is no longer supported: <flag_name>

This parameter was used in the input Cortex CLI flags file, but the parameter was removed in Grafana Mimir. The tool removed this CLI flag from the output configuration.
using a new default for <yaml_path>: <new_value> (used to be <old_value>)

The default value for a configuration parameter changed in Grafana Mimir. This parameter was not explicitly set in the input configuration files. When you run Grafana Mimir with the output configuration from mimirtool config convert Grafana Mimir uses the new default.
default value for <yaml_path> changed: <new_value> (used to be <old_value>); not updating

The default value for a configuration parameter that was set in the input configuration file has changed in Grafana Mimir. The tool has not converted the old default value to the new default value. To automatically update the default value to the new default value, pass the --update-defaults flag.

Extracting flags from Jsonnet

When using the Grafana Mimir Jsonnet library, all configuration uses flags set as object member key-value pairs. To perform conversion with mimirtool, you first need to extract the flags from the JSON manifested from the Jsonnet evaluation.

Use the following bash script to extract the arguments from a specific component:

 #!/usr/bin/env bash

 set -euf -o pipefail

 function usage {
   cat <<EOF
 Extract the CLI flags from individual components.

 Usage:
   $0 <resources JSON> <component>

 Examples:
   $0 resources.json ingester
   $0 <(tk eval environments/default) distributor
   $0 <(jsonnet environments/default/main.jsonnet) query-frontend
 EOF
 }

 if ! command -v jq &>/dev/null; then
   echo "jq command not found in PATH"
   echo "To download jq, refer to https://stedolan.github.io/jq/download/."
 fi

 if [[ $# -ne 2 ]]; then
   usage
   exit 1
 fi

 jq -rf /dev/stdin -- "$1" <<EOF
 ..
 | if type == "object" and .metadata.name == "$2" then .spec.template.spec.containers[]?.args[] else null end
 | select(. != null)
 EOF

The first parameter of the script is a JSON file containing Kubernetes resources. The second parameter of the script is the name of a container.

To retrieve the arguments from the distributor for a Tanka environment:

<PATH TO SCRIPT> <(tk eval environments/default) distributor

The script outputs results that are similar to the following:

-consul.hostname=consul.cortex-to-mimir.svc.cluster.local:8500
-distributor.ha-tracker.enable=false
-distributor.ha-tracker.enable-for-all-users=true
-distributor.ha-tracker.etcd.endpoints=etcd-client.cortex-to-mimir.svc.cluster.local.:2379
-distributor.ha-tracker.prefix=prom_ha/
-distributor.ha-tracker.store=etcd
-distributor.health-check-ingesters=true
-distributor.ingestion-burst-size=200000
-distributor.ingestion-rate-limit=10000
-distributor.ingestion-rate-limit-strategy=global
-distributor.remote-timeout=20s
-distributor.replication-factor=3
-distributor.ring.consul.hostname=consul.cortex-to-mimir.svc.cluster.local:8500
-distributor.ring.prefix=
-distributor.shard-by-all-labels=true
-mem-ballast-size-bytes=1073741824
-ring.heartbeat-timeout=10m
-ring.prefix=
-runtime-config.file=/etc/cortex/overrides.yaml
-server.grpc.keepalive.max-connection-age=2m
-server.grpc.keepalive.max-connection-age-grace=5m
-server.grpc.keepalive.max-connection-idle=1m
-server.grpc.keepalive.min-time-between-pings=10s
-server.grpc.keepalive.ping-without-stream-allowed=true
-target=distributor
-validation.reject-old-samples=true
-validation.reject-old-samples.max-age=12h

Use the output of the script as input to run the mimirtool configuration conversion.

After conversion, you can use the following script to transform the converted flags back into JSON:

#!/usr/bin/env bash

set -euf -o pipefail

function usage {
 cat <<EOF
Transform Go flags into JSON key value pairs

Usage:
 $0 <flags file>

Examples:
 $0 flags.flags
EOF
}

if [[ $# -ne 1 ]]; then
 usage
 exit 1
fi

key_values=$(sed -E -e 's/^-*(.*)=(.*)$/  "\1": "\2",/' "$1")
printf "{\n%s\n}" "${key_values::-1}"

The only parameter of the script is a file containing the flags, with each flag on its own line.

Backfill

The backfill command uploads Prometheus TSDB blocks into Grafana Mimir, by using the block-upload API that is exposed by the compactor component.

If the command is interrupted, you can restart it. Mimirtool detects which blocks are already uploaded, and will only upload unfinished or new blocks.

The block-upload feature is disabled by default. To enable the block-upload feature for a user or an entire system, refer to Configure TSDB block upload. If block upload is not enabled for the user, mimirtool backfill will fail.

Example

mimirtool backfill --address=http://mimir-compactor/ --id=anonymous /var/prometheus/{01G803NFXZ0MVKN71GT91HMV3Z,01G8BQ8PRR4TAP7EXZVBNTRBZ4,01G8CB7GTTC5ZXY23WTXHSYQXQ}

The results of the backfill command are as follows:

INFO[0000] Backfilling                              blocks="/var/prometheus/01G803NFXZ0MVKN71GT91HMV3Z,/var/prometheus/01G8BQ8PRR4TAP7EXZVBNTRBZ4,/var/prometheus/01G8CB7GTTC5ZXY23WTXHSYQXQ" user=anonymous
INFO[0000] making request to start block upload     block=01G803NFXZ0MVKN71GT91HMV3Z file=meta.json path=/var/prometheus/01G803NFXZ0MVKN71GT91HMV3Z
WARN[0000] block already exists on the server       path=/var/prometheus/01G803NFXZ0MVKN71GT91HMV3Z
INFO[0000] making request to start block upload     block=01G8BQ8PRR4TAP7EXZVBNTRBZ4 file=meta.json path=/var/prometheus/01G8BQ8PRR4TAP7EXZVBNTRBZ4
INFO[0000] uploading block file                     block=01G8BQ8PRR4TAP7EXZVBNTRBZ4 file=index path=/var/prometheus/01G8BQ8PRR4TAP7EXZVBNTRBZ4 size=259867
INFO[0000] uploading block file                     block=01G8BQ8PRR4TAP7EXZVBNTRBZ4 file=chunks/000001 path=/var/prometheus/01G8BQ8PRR4TAP7EXZVBNTRBZ4 size=5024391
INFO[0000] block uploaded successfully              block=01G8BQ8PRR4TAP7EXZVBNTRBZ4 path=/var/prometheus/01G8BQ8PRR4TAP7EXZVBNTRBZ4
INFO[0000] making request to start block upload     block=01G8CB7GTTC5ZXY23WTXHSYQXQ file=meta.json path=/var/prometheus/01G8CB7GTTC5ZXY23WTXHSYQXQ
INFO[0000] uploading block file                     block=01G8CB7GTTC5ZXY23WTXHSYQXQ file=index path=/var/prometheus/01G8CB7GTTC5ZXY23WTXHSYQXQ size=151181
INFO[0000] uploading block file                     block=01G8CB7GTTC5ZXY23WTXHSYQXQ file=chunks/000001 path=/var/prometheus/01G8CB7GTTC5ZXY23WTXHSYQXQ size=1986792
INFO[0000] block uploaded successfully              block=01G8CB7GTTC5ZXY23WTXHSYQXQ path=/var/prometheus/01G8CB7GTTC5ZXY23WTXHSYQXQ
INFO[0001] finished uploading blocks                already_exists=1 failed=0 succeeded=2

Note
Backfilled data visibility depends on the sample timestamps in the uploaded blocks. Sometimes data can appear more than 12 hours after upload.

Partition ring

The partition-ring command provides subcommands for forcefully modifying the ingest storage partition ring via memberlist. These commands are intended for emergency use only and bypass normal ingester lifecycle management.

Warning
These are dangerous operations not intended for production systems. Modifying the partition ring directly can cause data loss or query failures if used incorrectly.

All partition-ring subcommands require joining the memberlist cluster of the Grafana Mimir deployment. The tool must be able to reach at least one memberlist node via TCP and UDP.

For further information on how to run commands within a cell see Debugging distroless containers in Kubernetes.

An example of running mimirtool from within a debug container attached to a cell pod is as follows;

# Attach a debug container to an existing pod and shell into it
kubectl -n <namespace> debug -it pod/<pod> --image=alpine:latest --target=ingester -c debug-container -ti -- sh

# Copy over the `mimirtool` binary (make sure you have it compiled for the correct architecture)
kubectl -n <namespace> cp ./mimirtool <pod>:/tmp/mimirtool -c debug-container

# Find an ingester IP address and memberlist cluster label
kubectl -n <namespace> get pods ingester-zone-a-0 -o yaml | grep podIP
kubectl -n <namespace> get pods ingester-zone-a-0 -o yaml | grep memberlist.cluster

# Run mimirtool (in the shell from the first step)
cd /tmp
chmod u+x mimirtool
./mimirtool partition-ring remove-owner --owner.id=ingester-zone-c-40 --memberlist.join=<ingester-ip>:7946 --memberlist.cluster-label=<memberlist-cluster-label> --memberlist.bind-port=0

See Cleanup and Limitations after finishing with the debug container.

`add-partition`

Forcefully adds one or more partitions to the partition ring.

Flag	Description
`--partition.id`	Required. Comma-separated list of partition IDs to add (must be >= 0).
`--partition.state`	Required. State for the new partition(s). Must be one of: `pending`, `active`, `inactive`.
`--memberlist.join`	Required. Address of a memberlist node to join. Can be specified multiple times.
`--memberlist.cluster-label`	The cluster label to use when joining the memberlist cluster.
`--memberlist.bind-port`	Port to listen on for memberlist gossip messages. Default: `7946`. Use `0` to pick a random available port.
`--verbose`	Enable verbose logging.

Example

mimirtool partition-ring add-partition \
  --partition.id=39 \
  --partition.state=active \
  --memberlist.join=<ingester-pod-ip>:7946 \
  --memberlist.cluster-label=<cluster>.<namespace> \
  --memberlist.bind-port=0

`remove-partition`

Forcefully removes one or more partitions from the partition ring. A partition cannot be removed if it still has owners; use remove-owner first.

Flag	Description
`--partition.id`	Required. Comma-separated list of partition IDs to remove (must be >= 0).
`--memberlist.join`	Required. Address of a memberlist node to join. Can be specified multiple times.
`--memberlist.cluster-label`	The cluster label to use when joining the memberlist cluster.
`--memberlist.bind-port`	Port to listen on for memberlist gossip messages. Default: `7946`. Use `0` to pick a random available port.
`--verbose`	Enable verbose logging.

Example

mimirtool partition-ring remove-partition \
  --partition.id=39 \
  --memberlist.join=<ingester-pod-ip>:7946 \
  --memberlist.cluster-label=<cluster>.<namespace> \
  --memberlist.bind-port=0

`add-owner`

Forcefully adds one or more owners (ingester instances) to the partition ring, associating them with the specified partition.

Flag	Description
`--owner.id`	Required. Comma-separated list of owner IDs (ingester instance names) to add.
`--partition.id`	Required. The partition ID that the owner(s) will own (must be >= 0). Exactly one ID may be specified.
`--memberlist.join`	Required. Address of a memberlist node to join. Can be specified multiple times.
`--memberlist.cluster-label`	The cluster label to use when joining the memberlist cluster.
`--memberlist.bind-port`	Port to listen on for memberlist gossip messages. Default: `7946`. Use `0` to pick a random available port.
`--verbose`	Enable verbose logging.

Example

mimirtool partition-ring add-owner \
  --owner.id=ingester-zone-a-0 \
  --partition.id=0 \
  --memberlist.join=<ingester-pod-ip>:7946 \
  --memberlist.cluster-label=<cluster>.<namespace> \
  --memberlist.bind-port=0

`remove-owner`

Forcefully removes one or more owners (ingester instances) from the partition ring. This is useful for cleaning up stale owner entries left behind when an ingester is removed without going through a normal shutdown.

Flag	Description
`--owner.id`	Required. Comma-separated list of owner IDs (ingester instance names) to remove.
`--memberlist.join`	Required. Address of a memberlist node to join. Can be specified multiple times.
`--memberlist.cluster-label`	The cluster label to use when joining the memberlist cluster.
`--memberlist.bind-port`	Port to listen on for memberlist gossip messages. Default: `7946`. Use `0` to pick a random available port.
`--verbose`	Enable verbose logging.

Example

mimirtool partition-ring remove-owner \
  --owner.id=ingester-zone-c-38 \
  --memberlist.join=<ingester-pod-ip>:7946 \
  --memberlist.cluster-label=<cluster>.<namespace> \
  --memberlist.bind-port=0

To remove multiple owners in one operation, pass a comma-separated list:

mimirtool partition-ring remove-owner \
  --owner.id=ingester-zone-c-38,ingester-zone-c-39,ingester-zone-c-40 \
  --memberlist.join=<ingester-pod-ip>:7946 \
  --memberlist.cluster-label=<cluster>.<namespace> \
  --memberlist.bind-port=0

License

This software is licensed as AGPLv3. For more information, see LICENSE.

Grafana Markblocks

Wed, 03 Jun 2026 09:01:40 +0200

Grafana Markblocks

You can use the markblocks tool in Grafana Mimir to manage the state of time series data blocks in object storage.

Use markblocks for the following purposes:

Mark blocks for non-compaction. This is useful for corrupted blocks, preventing them from being processed by the compactor and potentially causing issues. Newer Mimir versions might automatically mark corrupted blocks.
Mark blocks for deletion. Use the tool to explicitly mark specific blocks for deletion from the object storage.

The markblocks tool helps to maintain data integrity and to manage storage effectively, especially in scenarios involving data corruption or manual cleanup.

Download markblocks

Download markblocks as part of the Mimir binary. To download the binary, refer to Grafana Mimir releases on GitHub.

Use markblocks

To use markblocks, you need to specify the tenant ID, bucket configuration, and block IDs you’d like to mark. For example:

# Mark a block for non-compaction (example for a corrupted block)
mimir markblocks --tenant-id=<your-tenant-id> --blocks-storage.backend=s3 --blocks-storage.s3.bucket-name=<your-bucket> --blocks-storage.s3.endpoint=<your-s3-endpoint> --mark-no-compact=<block-id>

# Mark a block for deletion
mimir markblocks --tenant-id=<your-tenant-id> --blocks-storage.backend=s3 --blocks-storage.s3.bucket-name=<your-bucket> --blocks-storage.s3.endpoint=<your-s3-endpoint> --mark-for-deletion=<block-id>

Note
Use caution when using markblocks, as it affects data management within your Mimir storage.