Monitor Grafana Mimir on Grafana Labs

Monitor the health of your system

Wed, 03 Jun 2026 09:01:40 +0200

Monitor the health of your system

To monitor and understand the health of your Grafana Mimir or Grafana Enterprise Metrics (GEM) database, you should collect and store the metrics and logs it exposes. This process is called meta-monitoring. You’re “monitoring” your “monitoring database.”

Collect metrics and logs from Grafana Mimir

Note
The Grafana Mimir Helm chart contains built-in configurations for meta-monitoring that use the Grafana Agent, which is now deprecated. Therefore, we no longer recommend using this approach. Instead we recommend an approach that uses the Kubernetes Monitoring Helm chart.

To collect metrics, logs and traces from Grafana Mimir, use the built-in Mimir integration provided by the grafana/k8s-monitoring-helm chart. This configures Grafana Alloy to handle all scraping and log collection automatically, no ServiceMonitors are needed. You should not enable any metaMonitoring settings in the Mimir Helm chart (see note above).

Refer to the meta-monitoring example for guidance. Update the destinations section to specify where to send the metrics, logs, and traces collected from Grafana Mimir or GEM. Metrics must be sent to Prometheus or a Prometheus-compatible TSDB (such as Thanos, another Mimir instance, or Grafana Cloud Metrics). You can configure multiple metrics, logs, and/or traces destinations to forward data to several backends simultaneously.

Visualize metrics and logs from Grafana Mimir

Once you’ve collected metrics, logs, and traces from Grafana Mimir or GEM and stored them in their respective databases, you can use this data to build dashboards and define alerts.

Start by configuring Grafana with datasources for metrics, logs, and traces. This typically involves adding Prometheus, Loki, and Tempo datasources that point to the respective backends.

From there, refer to Installing Grafana Mimir dashboards and alerts for instructions on how to set up our best practice dashboards and alerts for observing Grafana Mimir.

About Grafana Mimir dashboards and alerts requirements

Wed, 03 Jun 2026 09:01:40 +0200

About Grafana Mimir dashboards and alerts requirements

Grafana Mimir dashboards and alerts require certain labels to exist on metrics scraped from Grafana Mimir.

The mimir-distributed Helm chart provides metamonitoring support, which takes care of these labels. For more information about Helm chart metamonitoring, refer to Collect metrics and logs via the Helm chart. If you are using Helm chart metamonitoring, go to Installing Grafana Mimir dashboards and alerts.

If you’re not using the Helm chart, you must configure your Prometheus or Grafana Alloy instance to add these labels for the dashboards and alerts to function. The following table shows the required label names and whether they can be customized when compiling dashboards or alerts from sources.

Label name	Configurable?	Description
`cluster`	Yes	The Kubernetes cluster or datacenter where the Mimir cluster is running. You can configure the cluster label via the `per_cluster_label` field in the mixin configuration.
`namespace`	Yes	The Kubernetes namespace where the Mimir cluster is running. You can configure the namespace label via the `per_namespace_label` field in the mixin configuration.
`job`	Yes	A prefix (by default `<namespace>/`) followed by the Mimir component. When running in monolithic mode, make sure that the `<component>` is `mimir`. When running in microservices mode, make sure that the `<component>` is the name of the specific Mimir component (singular), such as `distributor`, `ingester`, or `store-gateway`. In the mixin configuration, you can configure the prefix via the `job_prefix` field, the label name via the `per_job_label` field, and the regular expressions that are used to match components via the `job_names` field.
`pod`	Yes	The unique identifier of a Mimir replica, for example the Pod ID when running on Kubernetes. You can configure the instance label via the `per_instance_label` field in the mixin configuration.
`instance`	Yes	The unique identifier of the node or machine where the Mimir replica is running, for example the node when running on Kubernetes. You can configure the node label via the `per_node_label` field in the mixin configuration.

For rules and alerts to function, you must configure your Prometheus or Grafana Alloy instance to scrape metrics from Grafana Mimir at an interval of 15s or shorter.

Deployment type

By default, Grafana Mimir dashboards assume Mimir is deployed in containers orchestrated by Kubernetes. If you’re running Mimir on baremetal, set the configuration field deployment_type: 'baremetal' and re-compile the dashboards.

Job selection

A metric could be exposed by multiple Grafana Mimir components, or even different applications running in the same namespace. To provide accurate dashboards and alerts, the job label (by default job) selects a metric from specific components. A job is a combination of a prefix and component. The default prefix is the Kubernetes namespace followed by a slash, for example <namespace>/ in <namespace>/ingester.

Pre-compiled dashboards and alerts are shipped with a default configuration. If you compile dashboards and alerts from source, you have the option to customize a) the label used for the job selection via the per_job_label field, b) the prefix expected in the label value, which you can omit by setting it to '' via the job_prefix field, and c) the regular expression used to select each Mimir component via the job_names field in the mixin configuration.

Default `job` selection in monolithic mode

When running Grafana Mimir in monolithic mode and using the pre-compiled dashboards and alerts, the job label should be set to <namespace>/mimir.

Default `job` selection in microservices mode

When running Grafana Mimir in microservices mode and using the pre-compiled dashboards and alerts, the job label should be set according to the following table.

Mimir service	Expected `job` label
Distributor	`<namespace>/distributor`
Ingester	`<namespace>/ingester`
Querier	`<namespace>/querier`
Ruler	`<namespace>/ruler`
Query-frontend	`<namespace>/query-frontend`
Query-scheduler	`<namespace>/query-scheduler`
Store-gateway	`<namespace>/store-gateway`
Compactor	`<namespace>/compactor`

Additional resources metrics

The Grafana Mimir dashboards displaying CPU, memory, disk, and network resources utilization require Prometheus metrics scraped from the following endpoints:

For more information about the kubelet metrics and cAdvisor metrics exported by the kubelet, refer to Metrics For Kubernetes System Components.

Metrics from kubelet, kube-state-metrics, and cAdvisor must all have a cluster label with the same value as in the Mimir metrics.

Metrics from Node Exporter and cAdvisor must all have an instance label on them that has the same value as the instance label on Mimir metrics.

Log labels

The Slow queries dashboard uses a Loki data source with the logs from Grafana Mimir to visualize slow queries. The query-frontend component logs query statistics when the -query-frontend.query-stats-enabled parameter is set to true. These logs need to have specific labels in order for the dashboard to work.

Label name	Configurable?	Description
`cluster`	Yes	The Kubernetes cluster or datacenter where the Mimir cluster is running. You can configure the cluster label via the `per_cluster_label` field in the mixin configuration.
`namespace`	No	The Kubernetes namespace where the Mimir cluster is running.
`name`	Yes	Name of the component. For example, `query-frontend`. You can configure the cluster label via the `per_component_loki_label` field in the mixin configuration.

Deploying the Grafana Mimir monitoring mixin

Wed, 03 Jun 2026 09:01:40 +0200

Deploying the Grafana Mimir monitoring mixin

Grafana Mimir exposes a /metrics endpoint returning Prometheus metrics. You can configure your Prometheus server to scrape Grafana Mimir or you can use the built-in functionality of the Helm chart to automatically send these metrics to a remote. The endpoint is exposed on the Mimir HTTP server address / port which can be customized through -server.http-listen-address and -server.http-listen-port CLI flags or their respective YAML config options.

Dashboards and alerts

Grafana Mimir is shipped with a comprehensive set of production-ready Grafana dashboards and alerts to monitor the state and health of a Mimir cluster.

Dashboards provide both a high-level and in-depth view of every aspect of a Grafana Mimir cluster. You can take a look at all the available dashboards in this overview.

Alerts allow you to monitor the health of a Mimir cluster. For each alert, we provide detailed runbooks to further investigate and fix the issue.

The requirements documentation lists prerequisites for using the Grafana Mimir dashboards and alerts.

The installation instructions show available options to install Grafana Mimir dashboards and alerts.

Installing Grafana Mimir dashboards and alerts

Wed, 03 Jun 2026 09:01:40 +0200

Installing Grafana Mimir dashboards and alerts

Grafana Mimir is shipped with a comprehensive set of production-ready Grafana dashboards and alerts to monitor the state and health of a Mimir cluster.

About HTTP status codes in alerting

Mimir’s default alerting mixins focus on actionable server-side issues. They include standard HTTP error codes, such as 500, 502, 503, that indicate service problems. They exclude error codes such as 529 and 598, that indicate client-side issues rather than server failures. You can create custom alerts for codes 529 and 598 if needed for your specific use case.

Requirements

Grafana Mimir dashboards and alerts require specific labels to be set by Prometheus or Grafana Alloy when scraping your Mimir cluster metrics
Some dashboards require recording rules that you should install in your Prometheus.

Install from package

Grafana Mimir provides ready to use Grafana dashboards in the .json format and Prometheus alerts in the .yaml format, that you can directly import into your Grafana installation and Prometheus config.

The packaged dashboards and alerts have been compiled from the sources using a default configuration and don’t allow you to customize the required metrics label names. If you need to customize the required metrics label names please choose one of the other installation options.

Download dashboards, recording rules and alerts from Grafana Mimir repository
Import dashboards in Grafana
Install recording rules and alerts in your Prometheus

Install from sources

Grafana Mimir dashboards and alerts are built using Jsonnet language and you can compile them from sources. If you choose this option, you can change the configuration to match your deployment, like customizing the required label names.

Checkout Mimir source code

git clone https://github.com/grafana/mimir.git

Review the mixin configuration at operations/mimir-mixin/config.libsonnet, and apply your changes if necessary.
Compile the mixin
Bash
```
make build-mixin
```
Import the dashboards saved at operations/mimir-mixin-compiled/dashboards/ in Grafana
Install the recording rules saved at operations/mimir-mixin-compiled/rules.yaml in your Prometheus
Install the alerts saved at operations/mimir-mixin-compiled/alerts.yaml in your Prometheus

Install dashboards from Jsonnet mixin

In case you’re already using Jsonnet to define your infrastructure as a code, you can vendor the Grafana Mimir mixin directly into your infrastructure repository and configure it overriding the _config fields. Given the exact setup really depends on a case-by-case basis, the following instructions are not meant to be prescriptive but just show the main steps required to vendor the mixin.

Initialize Jsonnet
Bash
```
jb init
```

Install Grafana Mimir mixin

jb install github.com/grafana/mimir/operations/mimir-mixin@main

Import and configure it

(import 'github.com/grafana/mimir/operations/mimir-mixin/mixin.libsonnet') + {
  _config+:: {
    // Override the Grafana Mimir mixin config here.
  },
}

Deploy mixin with Terraform

Based on Jsonnet configuration file, you can use Terraform to deploy the mixin in both Grafana and Grafana Mimir. To do so, you can use grafana/grafana, ovh/mixtool, and ovh/mimirtool providers.

Create a Terraform main.tf file:

# Specify which providers to use
terraform {
  required_version = ">= 1.3.5"
  required_providers {
    mixtool = {
      source  = "ovh/mixtool"
      version = "~> 0.1.1"
    }
    mimirtool = {
      source  = "ovh/mimirtool"
      version = "~> 0.1.1"
    }
    grafana = {
      source  = "grafana/grafana"
      version = "~> 1.32.0"
    }
  }
}

# Configure providers if needed
provider "grafana" {
  url  = "http://localhost:9000"
  auth = "admin:admin"
}

provider "mimirtool" {
  address   = "http://localhost:9009"
  tenant_id = "anonymous"
}

locals {
  mixin_source   = "custom.libsonnet"
  jsonnet_path   = "vendor"
}

# Build alerts
data "mixtool_alerts" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Build rules
data "mixtool_rules" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Build dashboards
data "mixtool_dashboards" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Deploy rules
resource "mimirtool_ruler_namespace" "rules" {
  namespace   = "rules_community"
  config_yaml = data.mixtool_rules.mimir.rules
}

# Deploy alerts
resource "mimirtool_ruler_namespace" "alerts" {
  namespace   = "alerts_community"
  config_yaml = data.mixtool_alerts.mimir.alerts
}

# Deploy dashboards
resource "grafana_dashboard" "mimir" {
  for_each    = data.mixtool_dashboards.mimir.dashboards
  config_json = each.value
}

Initialize Terraform:
Bash
```
terraform init
```
Review the changes that Terraform would apply to your infrastructure:
Bash
```
terraform plan
```
Deploy the changes:
Bash
```
terraform apply
```

Viewing Grafana Mimir dashboards

Wed, 03 Jun 2026 09:01:40 +0200

Viewing Grafana Mimir dashboards

Grafana Mimir provides the following production-ready dashboards.

Monitor Grafana Mimir on Grafana Labs

Monitor the health of your system

Monitor the health of your system

Collect metrics and logs from Grafana Mimir

Visualize metrics and logs from Grafana Mimir

About Grafana Mimir dashboards and alerts requirements

About Grafana Mimir dashboards and alerts requirements

Deployment type

Job selection

Default job selection in monolithic mode

Default job selection in microservices mode

Additional resources metrics

Log labels

Deploying the Grafana Mimir monitoring mixin

Deploying the Grafana Mimir monitoring mixin

Dashboards and alerts

Installing Grafana Mimir dashboards and alerts

Installing Grafana Mimir dashboards and alerts

About HTTP status codes in alerting

Requirements

Install from package

Install from sources

Install dashboards from Jsonnet mixin

Deploy mixin with Terraform

Viewing Grafana Mimir dashboards

Viewing Grafana Mimir dashboards

Default `job` selection in monolithic mode

Default `job` selection in microservices mode