Additional configuration on Grafana Labs

Configure roles and permissions

Fri, 03 Apr 2026 12:35:46 -0500

Configure roles and permissions

This guide explains how to configure roles and permissions for Grafana Alerting for Grafana OSS users. You’ll learn how to manage access using roles, folder permissions, and contact point permissions.

A user is any individual who can log in to Grafana. Each user is associated with a role that includes permissions. Permissions determine the tasks a user can perform in the system. For example, the Admin role includes permissions for an administrator to create and delete users.

For more information, refer to Organization roles.

Manage access using roles

Grafana OSS has three roles: Admin, Editor, and Viewer.

The following table describes the access each role provides for Grafana Alerting.

Role	Access
Viewer	Read access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences).
Editor	Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and provisioning.
Admin	Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and provisioning, as well as assign roles.

Assign roles

To assign roles, an admin needs to complete the following steps.

Navigate to Administration > Users and access > Users, Teams, or Service Accounts.
Search for the user, team or service account you want to add a role for.
Add the role you want to assign.

Manage access using folder permissions

You can extend the access provided by a role to alert rules and rule-specific silences by assigning permissions to individual folders.

This allows different users, teams, or service accounts to have customized access to modify or silence alert rules in specific folders.

Refer to the following table for details on the additional access provided by folder permissions:

Folder permission	Additional Access
View	No additional access: all permissions already contained in Viewer role.
Edit	Write access to alert rules and their rule-specific silences only in the given folder and subfolders.
Admin	Same additional access as Edit.

Note
You can’t use folders to customize access to notification resources.

To manage folder permissions, complete the following steps:

In the left-side menu, click Dashboards.
Hover your mouse cursor over a folder and click Go to folder.
Click Manage permissions from the Folder actions menu.
Update or add permissions as required.

Manage access to contact points

Extend or limit the access provided by a role to contact points by assigning permissions to individual contact points.

This allows different users, teams, or service accounts to have customized access to read or modify specific contact points.

Refer to the following table for details on the additional access provided by contact point permissions.

Contact point permission	Additional Access
View	View and export contact point as well as select it on the Alert rule edit page
Edit	Update or delete the contact point
Admin	Same additional access as Edit and manage permissions for the contact point. User should have additional permissions to read users and teams.

Assign contact point permissions

To manage contact point permissions, complete the following steps:

In the left-side menu, click Contact points.
Hover your mouse cursor over a contact point and click More.
Click Manage permissions from the actions menu.
Update or add permissions as required.

Configure RBAC

Fri, 03 Apr 2026 12:35:46 -0500

Configure RBAC

Role-based access control (RBAC) for Grafana Enterprise and Grafana Cloud provides a standardized way of granting, changing, and revoking access, so that users can view and modify Grafana resources.

A user is any individual who can log in to Grafana. Each user has a role that includes permissions. Permissions determine the tasks a user can perform in the system.

Each permission contains one or more actions and a scope.

Role types

Grafana has three types of roles for managing access:

Basic roles: Admin, Editor, Viewer, and No basic role. These are assigned to users and provide default access levels.
Fixed roles: Predefined groups of permissions for specific use cases. Basic roles automatically include certain fixed roles.
Custom roles: User-defined roles that combine specific permissions for granular access control.

Basic role permissions

The following table summarizes the default alerting permissions for each basic role.

Capability	Admin	Editor	Viewer
View alert rules	✓	✓	✓
Create, edit, and delete alert rules	✓	✓
View silences	✓	✓	✓
Create, edit, and expire silences	✓	✓
View contact points and templates	✓	✓	✓
Create, edit, and delete contact points	✓	✓
View notification policies	✓	✓	✓
Create, edit, and delete policies	✓	✓
View mute timings	✓	✓	✓
Create, edit, and delete timing intervals	✓	✓
View alert enrichments	✓	✓	✓
Create, edit, and delete enrichments	✓	✓
Access provisioning API	✓	✓
Export with decrypted secrets	✓

Note
Access to alert rules also requires permission to read the folder containing the rules and permission to query the data sources used in the rules.

Permissions

Grafana Alerting has the following permissions organized by resource type.

Alert rules

Permissions for managing Grafana-managed alert rules.

Action	Applicable scope	Description
`alert.rules:create`	`folders:` `folders:uid:`	Create Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder and `datasources:query` in the scope of data sources the user can query.
`alert.rules:read`	`folders:` `folders:uid:`	Read Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder.
`alert.rules:write`	`folders:` `folders:uid:`	Update Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder. To allow query modifications add `datasources:query` in the scope of data sources the user can query.
`alert.rules:delete`	`folders:` `folders:uid:`	Delete Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder.

External alert rules

Permissions for managing alert rules in external data sources that support alerting.

Action	Applicable scope	Description
`alert.rules.external:read`	`datasources:` `datasources:uid:`	Read alert rules in data sources that support alerting (Prometheus, Mimir, and Loki).
`alert.rules.external:write`	`datasources:` `datasources:uid:`	Create, update, and delete alert rules in data sources that support alerting (Mimir and Loki).

Alert instances and silences

Permissions for managing alert instances and silences in Grafana.

Action	Applicable scope	Description
`alert.instances:read`	n/a	Read alerts and silences in the current organization.
`alert.instances:create`	n/a	Create silences in the current organization.
`alert.instances:write`	n/a	Update and expire silences in the current organization.
`alert.silences:read`	`folders:` `folders:uid:`	Read all general silences and rule-specific silences in a folder and its subfolders.
`alert.silences:create`	`folders:` `folders:uid:`	Create rule-specific silences in a folder and its subfolders.
`alert.silences:write`	`folders:` `folders:uid:`	Update and expire rule-specific silences in a folder and its subfolders.

External alert instances

Permissions for managing alert instances in external data sources.

Action	Applicable scope	Description
`alert.instances.external:read`	`datasources:` `datasources:uid:`	Read alerts and silences in data sources that support alerting.
`alert.instances.external:write`	`datasources:` `datasources:uid:`	Manage alerts and silences in data sources that support alerting.

Contact points

Permissions for managing contact points (notification receivers).

Action	Applicable scope	Description
`alert.notifications.receivers:list`	n/a	List contact points in the current organization.
`alert.notifications.receivers:read`	`receivers:` `receivers:uid:`	Read contact points.
`alert.notifications.receivers.secrets:read`	`receivers:` `receivers:uid:`	Export contact points with decrypted secrets.
`alert.notifications.receivers:create`	n/a	Create a new contact points. The creator is automatically granted full access to the created contact point.
`alert.notifications.receivers:write`	`receivers:` `receivers:uid:`	Update existing contact points.
`alert.notifications.receivers:delete`	`receivers:` `receivers:uid:`	Update and delete existing contact points.
`alert.notifications.receivers:test`	n/a	Test contact points to verify their configuration. Deprecated. Use “alert.notifications.receivers.test:create”
`alert.notifications.receivers.test:create`	`receivers:` `receivers:uid:` `receivers:uid:-`	Test contact points to verify their configuration. Use scope `receivers:uid:-` to grant permission to test new integrations
`receivers.permissions:read`	`receivers:` `receivers:uid:`	Read permissions for contact points.
`receivers.permissions:write`	`receivers:` `receivers:uid:`	Manage permissions for contact points.

Notification policies

Permissions for managing notification policies (routing rules).

Action	Applicable scope	Description
`alert.notifications.routes:read`	n/a	Read notification policies.
`alert.notifications.routes:write`	n/a	Create new, update, and delete notification policies.

Time intervals

Permissions for managing mute time intervals.

Action	Applicable scope	Description
`alert.notifications.time-intervals:read`	n/a	Read mute time intervals.
`alert.notifications.time-intervals:write`	n/a	Create new or update existing mute time intervals.
`alert.notifications.time-intervals:delete`	n/a	Delete existing time intervals.

Templates

Permissions for managing notification templates.

Action	Applicable scope	Description
`alert.notifications.templates:read`	n/a	Read templates.
`alert.notifications.templates:write`	n/a	Create new or update existing templates.
`alert.notifications.templates:delete`	n/a	Delete existing templates.
`alert.notifications.templates.test:write`	n/a	Test templates with custom payloads (preview and payload editor functionality).

General notifications

Legacy permissions for managing all notification resources.

Action	Applicable scope	Description
`alert.notifications:read`	n/a	Read all templates, contact points, notification policies, and mute timings in the current organization.
`alert.notifications:write`	n/a	Manage templates, contact points, notification policies, and mute timings in the current organization.

External notifications

Permissions for managing notification resources in external data sources.

Action	Applicable scope	Description
`alert.notifications.external:read`	`datasources:` `datasources:uid:`	Read templates, contact points, notification policies, and mute timings in data sources that support alerting.
`alert.notifications.external:write`	`datasources:` `datasources:uid:`	Manage templates, contact points, notification policies, and mute timings in data sources that support alerting.

Provisioning

Permissions for managing alerting resources via the provisioning API.

Action	Applicable scope	Description
`alert.provisioning:read`	n/a	Read all Grafana alert rules, notification policies, etc via provisioning API. Permissions to folders and data source are not required.
`alert.provisioning.secrets:read`	n/a	Same as `alert.provisioning:read` plus ability to export resources with decrypted secrets.
`alert.provisioning:write`	n/a	Update all Grafana alert rules, notification policies, etc via provisioning API. Permissions to folders and data source are not required.
`alert.rules.provisioning:read`	n/a	Read Grafana alert rules via provisioning API. More specific than `alert.provisioning:read`.
`alert.rules.provisioning:write`	n/a	Create, update, and delete Grafana alert rules via provisioning API. More specific than `alert.provisioning:write`.
`alert.notifications.provisioning:read`	n/a	Read notification resources (contact points, notification policies, templates, time intervals) via provisioning API. More specific than `alert.provisioning:read`.
`alert.notifications.provisioning:write`	n/a	Create, update, and delete notification resources via provisioning API. More specific than `alert.provisioning:write`.
`alert.provisioning.provenance:write`	n/a	Set provisioning status for alerting resources. Cannot be used alone. Requires user to have permissions to access resources.

Alert enrichments

Permissions for managing alert enrichments.

Action	Applicable scope	Description
`alert.enrichments:read`	n/a	Read alert enrichment configurations in the current organization.
`alert.enrichments:write`	n/a	Create, update, and delete alert enrichment configurations in the current organization.

To help plan your RBAC rollout strategy, refer to Plan your RBAC rollout strategy.

Configure Alertmanagers

Fri, 03 Apr 2026 12:35:46 -0500

Configure Alertmanagers

Grafana Alerting is based on the architecture of the Prometheus alerting system. Grafana sends firing and resolved alerts to an Alertmanager, which is responsible for handling notifications. This architecture decouples alert rule evaluation from notification handling, improving scalability.

Grafana includes a built-in Grafana Alertmanager to handle notifications. This guide shows you how to:

Use different types of Alertmanagers with Grafana
Add other Alertmanager and enable it to receive all Grafana-managed alerts
Use an Alertmanager as a contact point to route specific alerts

Alertmanager resources

It’s important to note that each Alertmanager manages its own independent alerting resources, such as:

Contact points and notification templates
Notification policies and mute timings
Silences
Active notifications

Use the Choose Alertmanager dropdown on these pages to switch between Alertmanagers and view or manage their resources.

Types of Alertmanagers in Grafana

Grafana can be configured to handle alert notifications using various Alertmanagers.

Grafana Alertmanager: Grafana includes a built-in Alertmanager that extends the Prometheus Alertmanager. This is the default Alertmanager and is referred to as “Grafana” in the user interface. It can only handle Grafana-managed alerts.
Cloud Alertmanager: Each Grafana Cloud instance comes preconfigured with an additional Alertmanager (grafanacloud-STACK_NAME-ngalertmanager) from the Mimir (Prometheus) instance running in the Grafana Cloud Stack.

The Cloud Alertmanager is available exclusively in Grafana Cloud and can handle both Grafana-managed and data source-managed alerts.
Other Alertmanagers: Grafana Alerting also supports sending alerts to other Alertmanagers, such as the Prometheus Alertmanager, which can handle both Grafana-managed and data source-managed alerts.

Grafana Alerting supports using a combination of Alertmanagers and can enable other Alertmanagers to receive Grafana-managed alerts. The decision often depends on your alerting setup and where your alerts are generated.

For example, if you already have an Alertmanager running in your on-premises or cloud infrastructure to handle Prometheus alerts, you can forward Grafana-managed alerts to the same Alertmanager for unified notification handling.

Add an Alertmanager

Alertmanagers should be configured as data sources using Grafana Configuration from the main Grafana navigation menu. To add an Alertmanager, complete the following steps.

Click Connections in the left-side menu.
Under Your connections, click Data sources.
Enter Alertmanager in the search bar.
Click Alertmanager.

The Settings tab of the data source is displayed.

Set the data source’s basic configuration options:

Name	Description
Name	Sets the name you use to refer to the data source
Default	Sets whether the data source is pre-selected for new panels and queries
Alertmanager Implementation	Alertmanager implementation. Mimir, Cortex, and Prometheus are supported
Receive Grafana Alerts	When enabled, the Alertmanager can receive Grafana-managed alerts. Important: This works only if receiving alerts is enabled for the Alertmanager in the Grafana Alerting Settings page
HTTP URL	Sets the HTTP protocol, IP, and port of your Alertmanager instance, such as `https://alertmanager.example.org:9093`
Access	Only Server access mode is functional

For provisioning instructions, refer to the Alertmanager data source documentation.

After adding an Alertmanager, you can use the Grafana Alerting UI to manage notification policies, contact points, silences, and other alerting resources from within Grafana.

Note
When using Prometheus, you can manage silences in the Grafana Alerting UI. However, other Alertmanager resources such as contact points, notification policies, and templates are read-only because the Prometheus Alertmanager HTTP API does not support updates for these resources.

When using multiple Alertmanagers, use the Choose Alertmanager dropdown to switch between Alertmanagers.

Enable an Alertmanager to receive Grafana-managed alerts

After enabling Receive Grafana Alerts in the Data Source Settings, you must also configure the Alertmanager in the Alerting Settings page. Grafana supports enabling one or multiple Alertmanagers to receive all generated Grafana-managed alerts.

In the left-side menu, click Alerts & IRM and then Alerting.
Click Settings to view the list of configured Alertmanagers.
For the selected Alertmanager, click the Enable/Disable button to toggle receiving Grafana-managed alerts. When activated, the Alertmanager displays Receiving Grafana-managed alerts.

All Grafana-managed alerts are forwarded to Alertmanagers marked as Receiving Grafana-managed alerts.

Note
Grafana Alerting does not support forwarding Grafana-managed alerts to the AlertManager in Amazon Managed Service for Prometheus. For more details, refer to this GitHub issue.

Use an Alertmanager as a contact point to receive specific alerts

The previous instructions enable sending all Grafana-managed alerts to an Alertmanager.

To send specific alerts to an Alertmanager, configure the Alertmanager as a contact point. You can then assign this contact point to notification policies or individual alert rules.

For detailed instructions, refer to:

Manage Alertmanager configurations

On the Settings page, you can also manage your Alertmanager configurations.

Manage version snapshots for the built-in Alertmanager, which allows administrators to roll back unintentional changes or mistakes in the Alertmanager configuration.
Compare the historical snapshot with the latest configuration to see which changes were made.

Configure alert state history

Fri, 03 Apr 2026 12:35:46 -0500

Configure alert state history

Alerting can record all alert rule state changes for your Grafana managed alert rules in a Loki or Prometheus instance, or in both.

With Prometheus, you can query the GRAFANA_ALERTS metric for alert state changes in Grafana Explore.
With Loki, you can query and view alert state changes in Grafana Explore and the Grafana Alerting History views.

Configure Loki for alert state

The following steps describe a basic configuration:

Configure Loki

The default Loki settings might need some tweaking as the state history view might query up to 30 days of data.

The following change to the default configuration should work for most instances, but look at the full Loki configuration settings and adjust according to your needs.
YAML
```
limits_config:
  split_queries_by_interval: '24h'
  max_query_parallelism: 32
```
As this might impact the performances of an existing Loki instance, use a separate Loki instance for the alert state history.

Configure Grafana

The following Grafana configuration instructs Alerting to write alert state history to a Loki instance:

[unified_alerting.state_history]
enabled = true
backend = loki

# The URL of the Loki server
loki_remote_url = http://localhost:3100

[feature_toggles]
enable = alertingCentralAlertHistory

Configure the Loki data source in Grafana

Add the Loki data source to Grafana.

If everything is set up correctly, you can access the History view and History page to view and filter alert state history. You can also use Grafana Explore to query the Loki instance, see Alerting Meta monitoring for details.

Configure Prometheus for alert state (GRAFANA_ALERTS metric)

You can also configure a Prometheus instance to store alert state changes for your Grafana-managed alert rules. However, this setup does not enable the Grafana Alerting History views, as Loki does.

Instead, Grafana Alerting writes alert state data to the GRAFANA_ALERTS metric-similar to how Prometheus Alerting writes to the ALERTS metric.

GRAFANA_ALERTS{alertname="", alertstate="", grafana_alertstate="", grafana_rule_uid="", <additional alert labels>}

The following steps describe a basic configuration:

Configure Prometheus

Enable the remote write receiver in your Prometheus instance by setting the --web.enable-remote-write-receiver command-line flag. This enables the endpoint to receive alert state data from Grafana Alerting.
Configure the Prometheus data source in Grafana

Add the Prometheus data source to Grafana.

In the Prometheus data source configuration options, set the Prometheus type to match your Prometheus instance type. Grafana Alerting uses this option to identify the remote write endpoint.

Configure Grafana

The following Grafana configuration instructs Alerting to write alert state history to a Prometheus instance:

[unified_alerting.state_history]
enabled = true
backend = prometheus
# Target data source UID for writing alert state changes.
prometheus_target_datasource_uid = <DATA_SOURCE_UID>

# (Optional) Metric name for the alert state metric. Default is "GRAFANA_ALERTS".
# prometheus_metric_name = GRAFANA_ALERTS
# (Optional)  Timeout for writing alert state data to the target data source. Default is 10s.
# prometheus_write_timeout = 10s

You can then use Grafana Explore to query the alert state metric. For details, refer to Alerting Meta monitoring.

GRAFANA_ALERTS{alertstate='firing'}

Configure Loki and Prometheus for alert state

You can also configure both Loki and Prometheus to record alert state changes for your Grafana-managed alert rules.

Start with the same setup steps as shown in the previous Loki and Prometheus sections. Then, adjust your Grafana configuration as follows:

[unified_alerting.state_history]
enabled = true
backend = multiple

primary = loki
# URL of the Loki server.
loki_remote_url = http://localhost:3100

secondaries = prometheus
# Target data source UID for writing alert state changes.
prometheus_target_datasource_uid = <DATA_SOURCE_UID>

Provision Alerting resources

Fri, 03 Apr 2026 12:35:46 -0500

Provision Alerting resources

Alerting infrastructure is often complex, with many pieces of the pipeline that often live in different places. Scaling this across multiple teams and organizations is an especially challenging task. Importing and exporting (or provisioning) your alerting resources in Grafana Alerting makes this process easier by enabling you to create, manage, and maintain your alerting data in a way that best suits your organization.

You can import alert rules, contact points, notification policies, mute timings, and templates.

You cannot edit imported alerting resources in the Grafana UI in the same way as alerting resources that were not imported. You can only edit imported contact points, notification policies, templates, and mute timings in the source where they were created. For example, if you manage your alerting resources using files from disk, you cannot edit the data in Terraform or from within Grafana.

Import alerting resources

Choose from the options below to import (or provision) your Grafana Alerting resources.

Use configuration files to provision your alerting resources, such as alert rules and contact points, through files on disk.
Note
- You cannot edit provisioned resources from files in the Grafana UI.
- Provisioning with configuration files is not available in Grafana Cloud.
Use Terraform to provision alerting resources.
Use the Alerting provisioning HTTP API to manage alerting resources.

Note
The Alerting provisioning HTTP API can be used to create, modify, and delete resources for Grafana-managed alerts.

To manage resources related to data source-managed alerts, including recording rules, use the Mimir or Cortex tool.

The JSON output from the majority of Alerting HTTP endpoints isn’t compatible for provisioning via configuration files.

If you need the alerting resources for file provisioning, use Export Alerting endpoints to return or download them in provisioning format.

Export alerting resources

You can export both manually created and provisioned alerting resources. You can also edit and export an alert rule without applying the changes.

For detailed instructions on the various export options, refer to Export alerting resources.

View provisioned alerting resources

To view your provisioned resources in Grafana, complete the following steps.

Open your Grafana instance.
Navigate to Alerting.
Click an alerting resource folder, for example, Alert rules.

Provisioned resources are labeled Provisioned, so that it is clear that they were not created manually.

Configure high availability

Fri, 03 Apr 2026 12:35:46 -0500

Configure high availability

Grafana Alerting uses the Prometheus model of separating the evaluation of alert rules from the delivering of notifications. In this model, the evaluation of alert rules is done in the alert generator and the delivering of notifications is done in the alert receiver. In Grafana Alerting, the alert generator is the Scheduler and the receiver is the Alertmanager.

When running multiple instances of Grafana, all alert rules are evaluated on all instances. You can think of the evaluation of alert rules as being duplicated by the number of running Grafana instances. This is how Grafana Alerting makes sure that as long as at least one Grafana instance is working, alert rules are still be evaluated and notifications for alerts are still sent.

You can find this duplication in state history and it is a good way to verify your high availability setup.

While the alert generator evaluates all alert rules on all instances, the alert receiver makes a best-effort attempt to avoid duplicate notifications. The alertmanagers use a gossip protocol to share information between them to prevent sending duplicated notifications.

Alertmanager chooses availability over consistency, which may result in occasional duplicated or out-of-order notifications. It takes the opinion that duplicate or out-of-order notifications are better than no notifications.

Alertmanagers also gossip silences, which means a silence created on one Grafana instance is replicated to all other Grafana instances. Both notifications and silences are persisted to the database periodically, and during graceful shut down.

Enable alerting high availability using Memberlist

Before you begin

Since gossiping of notifications and silences uses both TCP and UDP port 9094, ensure that each Grafana instance is able to accept incoming connections on these ports.

To enable high availability support:

In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set [ha_peers] to the number of hosts for each Grafana instance in the cluster (using a format of host:port), for example, ha_peers=10.0.0.5:9094,10.0.0.6:9094,10.0.0.7:9094. You must have at least one (1) Grafana instance added to the ha_peers section.
Set [ha_listen_address] to the instance IP address using a format of host:port (or the Pod’s IP in the case of using Kubernetes). By default, it is set to listen to all interfaces (0.0.0.0).
Set [ha_advertise_address] to the instance’s hostname or IP address in the format “host:port”. Use this setting when the instance is behind NAT (Network Address Translation), such as in Docker Swarm or Kubernetes service, where external and internal addresses differ. This address helps other cluster instances communicate with it. The setting is optional.
Set [ha_peer_timeout] in the [unified_alerting] section of the custom.ini to specify the time to wait for an instance to send a notification via the Alertmanager. The default value is 15s, but it may increase if Grafana servers are located in different geographic regions or if the network latency between them is high.

For a demo, see this example using Docker Compose.

Enable alerting high availability using Redis

As an alternative to Memberlist, you can configure Redis to enable high availability. Redis standalone, Redis Cluster and Redis Sentinel modes are supported.

Note
Memberlist is the preferred option for high availability. Use Redis only in environments where direct communication between Grafana servers is not possible, such as when TCP or UDP ports are blocked.

Make sure you have a Redis server that supports pub/sub. If you use a proxy in front of your Redis cluster, make sure the proxy supports pub/sub.
In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set ha_redis_address to the Redis server address or addresses Grafana should connect to. It can be a single Redis address if using Redis standalone, or a list of comma-separated addresses if using Redis Cluster or Sentinel.
Optional: Set ha_redis_cluster_mode_enabled to true if you are using Redis Cluster.
Optional: Set ha_redis_sentinel_mode_enabled to true if you are using Redis Sentinel. Also set ha_redis_sentinel_master_name to the Redis Sentinel master name.
Optional: Set the username and password if authentication is enabled on the Redis server using ha_redis_username and ha_redis_password.
Optional: Set the username and password if authentication is enabled on Redis Sentinel using ha_redis_sentinel_username and ha_redis_sentinel_password.
Optional: Set ha_redis_prefix to something unique if you plan to share the Redis server with multiple Grafana instances.
Optional: Set ha_redis_tls_enabled to true and configure the corresponding ha_redis_tls_* fields to secure communications between Grafana and Redis with Transport Layer Security (TLS).
Set [ha_advertise_address] to ha_advertise_address = "${POD_IP}:9094" This is required if the instance doesn’t have an IP address that is part of RFC 6890 with a default route.

For a demo, see this example using Docker Compose.

Enable alerting high availability using Kubernetes

You can expose the Pod IP through an environment variable via the container definition.
YAML
```
env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
```

Add the port 9094 to the Grafana deployment:

ports:
  - name: grafana
    containerPort: 3000
    protocol: TCP
  - name: gossip-tcp
    containerPort: 9094
    protocol: TCP
  - name: gossip-udp
    containerPort: 9094
    protocol: UDP

Add the environment variables to the Grafana deployment:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Create a headless service that returns the Pod IP instead of the service IP, which is what the ha_peers need:

apiVersion: v1
kind: Service
metadata:
  name: grafana-alerting
  namespace: grafana
  labels:
    app.kubernetes.io/name: grafana-alerting
    app.kubernetes.io/part-of: grafana
spec:
  type: ClusterIP
  clusterIP: 'None'
  ports:
    - port: 9094
  selector:
    app: grafana

Make sure your grafana deployment has the label matching the selector, e.g. app:grafana:

Add in the grafana.ini:

[unified_alerting]
enabled = true
ha_listen_address = "${POD_IP}:9094"
ha_peers = "grafana-alerting.grafana:9094"
ha_advertise_address = "${POD_IP}:9094"
ha_peer_timeout = 15s
ha_reconnect_timeout = 2m

Verify your high availability setup

When running multiple Grafana instances, all alert rules are evaluated on every instance. This multiple evaluation of alert rules is visible in the state history and provides a straightforward way to verify that your high availability configuration is working correctly.

Note
If using a mix of execute_alerts=false and execute_alerts=true on the HA nodes, since the alert state is not shared amongst the Grafana instances, the instances with execute_alerts=false do not show any alert status.

The HA settings (ha_peers, etc.) apply only to communication between alertmanagers, synchronizing silences and attempting to avoid duplicate notifications, as described in the introduction.

You can also confirm your high availability setup by monitoring Alertmanager metrics exposed by Grafana.

Note
Starting in Grafana v12.4, these metrics are prefixed with grafana_ (for example, grafana_alertmanager_cluster_members). If you are upgrading from an earlier version, update your dashboards and alert rules accordingly.

Metric	Description
`grafana_alertmanager_cluster_members`	Number indicating current number of members in cluster.
`grafana_alertmanager_cluster_messages_received_total`	Total number of cluster messages received.
`grafana_alertmanager_cluster_messages_received_size_total`	Total size of cluster messages received.
`grafana_alertmanager_cluster_messages_sent_total`	Total number of cluster messages sent.
`grafana_alertmanager_cluster_messages_sent_size_total`	Total number of cluster messages received.
`grafana_alertmanager_cluster_messages_publish_failures_total`	Total number of messages that failed to be published.
`grafana_alertmanager_cluster_pings_seconds`	Histogram of latencies for ping messages.
`grafana_alertmanager_cluster_pings_failures_total`	Total number of failed pings.
`grafana_alertmanager_peer_position`	The position an Alertmanager instance believes it holds, which defines its role in the cluster. Peers should be numbered sequentially, starting from zero.

You can confirm the number of Grafana instances in your alerting high availability setup by querying the grafana_alertmanager_cluster_members and grafana_alertmanager_peer_position metrics.

Note that these alerting high availability metrics are exposed via the /metrics endpoint in Grafana, and are not automatically collected or displayed. If you have a Prometheus instance connected to Grafana, add a scrape_config to scrape Grafana metrics and then query these metrics in Explore.

- job_name: grafana
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
    - targets:
        - grafana:3000

For more information on monitoring alerting metrics, refer to Alerting meta-monitoring. For a demo, see alerting high availability examples using Docker Compose.

Prevent duplicate notifications

In high-availability mode, each Grafana instance runs its own pre-configured alertmanager to handle alert notifications.

When multiple Grafana instances are running, all alert rules are evaluated on each instance. By default, each instance sends firing alerts to its respective alertmanager. This results in notification handling being duplicated across all running Grafana instances.

Alertmanagers in HA mode communicate with each other to coordinate notification delivery. However, this setup can sometimes lead to duplicated or out-of-order notifications. By design, HA prioritizes sending duplicate notifications over the risk of missing notifications.

To avoid duplicate notifications, you can configure a shared alertmanager to manage notifications for all Grafana instances. For more information, refer to add an external alertmanager.

Meta monitoring

Fri, 03 Apr 2026 12:35:46 -0500

Meta monitoring

Monitor your alerting metrics to ensure you identify potential issues before they become critical.

Meta monitoring is the process of monitoring your monitoring system and alerting when your monitoring is not working as it should.

In order to enable you to meta monitor, Grafana provides predefined metrics.

Identify which metrics are critical to your monitoring system (i.e. Grafana) and then set up how you want to monitor them.

You can use meta-monitoring metrics to understand the health of your alerting system in the following ways:

Optional: Create a dashboard in Grafana that uses this metric in a panel (just like you would for any other kind of metric).
Optional: Create an alert rule in Grafana that checks this metric regularly (just like you would do for any other kind of alert rule).
Optional: Use the Explore module in Grafana.

Metrics for Grafana-managed alerts

To meta monitor Grafana-managed alerts, you can collect two types of metrics in a Prometheus instance:

State history metric (GRAFANA_ALERTS) — Exported by Grafana Alerting as part of alert state history.
Scraped metrics — Exported by Grafana’s /metrics endpoint to monitor alerting activity and performance.

You need a Prometheus-compatible server to collect and store these metrics.

`GRAFANA_ALERTS` metric

If you have configured Prometheus for alert state history, Grafana writes alert state changes to the ALERTS metric:

GRAFANA_ALERTS{alertname="", alertstate="", grafana_alertstate="", grafana_rule_uid="", <additional alert labels>}

This GRAFANA_ALERTS metric is compatible with the ALERTS metric used by Prometheus Alerting and includes two additional labels:

A new grafana_rule_uid label for the UID of the Grafana rule.
A new grafana_alertstate label for the Grafana alert state, which differs slightly from the equivalent Prometheus state included in the alertstate label.

Alert labels are automatically converted before being written to Prometheus to ensure compatibility. Prometheus requires label names to start with a letter or underscore (_), followed only by letters, numbers, or additional underscores. Invalid characters are replaced during conversion. For example, 1my-label becomes _my_label.

Grafana state	`alertstate`	`grafana_alertstate`
Alerting	`firing`	`alerting`
Recovering	`firing`	`recovering`
Pending	`pending`	`pending`
Error	`firing`	`error`
NoData	`firing`	`nodata`
Normal	(no metric emitted)	(no metric emitted)

You can then query this metric like any other Prometheus metric:

GRAFANA_ALERTS{grafana_alertstate="alerting"}

GRAFANA_ALERTS{grafana_alertstate="recovering"}

GRAFANA_ALERTS{grafana_alertstate="pending", severity="critical"}

Scraped metrics

To collect scraped Alerting metrics, configure Prometheus to scrape metrics from Grafana.

- job_name: grafana
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
    - targets:
        - grafana:3000

The Grafana ruler, which is responsible for evaluating alert rules, and the Grafana Alertmanager, which is responsible for sending notifications of firing and resolved alerts, provide a number of metrics that let you observe them.

grafana_alerting_alerts

This metric is a counter that shows you the number of normal, pending, alerting, nodata and error alerts. For example, you might want to create an alert that fires when grafana_alerting_alerts{state="error"} is greater than 0.

grafana_alerting_schedule_alert_rules

This metric is a gauge that shows you the number of alert rules scheduled. An alert rule is scheduled unless it is paused, and the value of this metric should match the total number of non-paused alert rules in Grafana.

grafana_alerting_schedule_periodic_duration_seconds_bucket

This metric is a histogram that shows you the time it takes to process an individual tick in the scheduler that evaluates alert rules. If the scheduler takes longer than 10 seconds to process a tick then pending evaluations start to accumulate such that alert rules might later than expected.

grafana_alerting_schedule_query_alert_rules_duration_seconds_bucket

This metric is a histogram that shows you how long it takes the scheduler to fetch the latest rules from the database. If this metric is elevated, schedule_periodic_duration_seconds is also evaluated.

grafana_alerting_scheduler_behind_seconds

This metric is a gauge that shows you the number of seconds that the scheduler is behind where it should be. This number increases if schedule_periodic_duration_seconds is longer than 10 seconds, and decrease when it is less than 10 seconds. The smallest possible value of this metric is 0.

grafana_alerting_notification_latency_seconds_bucket

This metric is a histogram that shows you the number of seconds taken to send notifications for firing and resolved alerts. This metric lets you observe slow or over-utilized integrations, such as an SMTP server that is being given emails faster than it can send them.

grafana_alerting_state_history_writes_failed_total

This metric is a counter that shows you the number of failed writes to the configured alert state history backend. It includes a backend label to distinguish between different backends (such as loki or prometheus).

For example, you might want to create an alert that fires when grafana_alerting_state_history_writes_failed_total{backend="prometheus"} is greater than 0 to detect when Prometheus remote write is failing.

Logs for Grafana-managed alerts

If you have configured Loki for alert state history, logs related to state changes in Grafana-managed alerts are stored in the Loki data source.

You can use Grafana Explore and the Loki query editor to search for alert state changes.

{from="state-history"} | json

{from="state-history"} | json | previous=~"Normal.*" | current=~"Alerting.*"

{from="state-history"} | json | current=~"Error.*"

In the Logs view, you can review details for individual alerts by selecting fields such as:

previous: previous alert instance state.
current: current alert instance state.
ruleTitle: alert rule title.
ruleID and ruleUID.
labels_alertname, labels_new_label, and labels_grafana_folder.
Additional available fields.

Alternatively, you can access the History page in Grafana to visualize and filter state changes for individual alerts or all alerts.

Metrics for Mimir-managed alerts

To meta monitor Grafana Mimir-managed alerts, open source and on-premise users need a Prometheus/Mimir server, or another metrics database to collect and store metrics exported by the Mimir ruler.

rule_evaluation_failures_total

This metric is a counter that shows you the total number of rule evaluation failures.

Metrics for Alertmanager

To meta monitor the Alertmanager, you need a Prometheus/Mimir server, or another metrics database to collect and store metrics exported by Alertmanager.

For example, if you are using Prometheus you should add a scrape_config to Prometheus to scrape metrics from your Alertmanager.

- job_name: alertmanager
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
    - targets:
        - alertmanager:9093

The following is a list of available metrics for Alertmanager.

alertmanager_alerts

This metric is a counter that shows you the number of active, suppressed, and unprocessed alerts in Alertmanager. Suppressed alerts are silenced alerts, and unprocessed alerts are alerts that have been sent to the Alertmanager but have not been processed.

alertmanager_alerts_invalid_total

This metric is a counter that shows you the number of invalid alerts that were sent to Alertmanager. This counter should not exceed 0, and so in most cases, create an alert that fires if whenever this metric increases.

alertmanager_notifications_total

This metric is a counter that shows you how many notifications have been sent by Alertmanager. The metric uses a label “integration” to show the number of notifications sent by integration, such as email.

alertmanager_notifications_failed_total

This metric is a counter that shows you how many notifications have failed in total. This metric also uses a label “integration” to show the number of failed notifications by integration, such as failed emails. In most cases, use the rate function to understand how often notifications are failing to be sent.

alertmanager_notification_latency_seconds_bucket

This metric is a histogram that shows you the amount of time it takes Alertmanager to send notifications and for those notifications to be accepted by the receiving service. This metric uses a label “integration” to show the amount of time by integration. For example, you can use this metric to show the 95th percentile latency of sending emails.

Metrics for Alertmanager in high availability mode

If you are using Alertmanager in high availability mode there are a number of additional metrics that you might want to create alerts for.

alertmanager_cluster_members

This metric is a gauge that shows you the current number of members in the cluster. The value of this gauge should be the same across all Alertmanagers. If different Alertmanagers are showing different numbers of members then this is indicative of an issue with your Alertmanager cluster. You should look at the metrics and logs from your Alertmanagers to better understand what might be going wrong.

alertmanager_cluster_failed_peers

This metric is a gauge that shows you the current number of failed peers.

alertmanager_cluster_health_score

This metric is a gauge showing the health score of the Alertmanager. Lower values are better, and zero means the Alertmanager is healthy.

alertmanager_cluster_peer_info

This metric is a gauge. It has a constant value 1, and contains a label called “peer” containing the Peer ID of each known peer.

alertmanager_cluster_reconnections_failed_total

This metric is a counter that shows you the number of failed peer connection attempts. In most cases you should use the rate function to understand how often reconnections fail as this may be indicative of an issue or instability in your network.

Performance considerations and limitations

Fri, 03 Apr 2026 12:35:46 -0500

Performance considerations and limitations

Grafana Alerting supports multi-dimensional alerting, where one alert rule can generate many alerts. For example, you can configure an alert rule to fire an alert every time the CPU of individual virtual machines max out. This topic discusses performance considerations resulting from multi-dimensional alerting.

Evaluating alerting rules consumes RAM and CPU to compute the output of an alerting query, and network resources to send alert notifications and write the results to the Grafana SQL database. The configuration of individual alert rules affects the resource consumption and, therefore, the maximum number of rules a given configuration can support.

The following section provides a list of alerting performance considerations.

Frequency of rule evaluation consideration. The “Evaluate Every” property of an alert rule controls the frequency of rule evaluation. It is recommended to use the lowest acceptable evaluation frequency to support more concurrent rules.
Cardinality of the rule’s result set. For example, suppose you are monitoring API response errors for every API path, on every virtual machine in your fleet. This set has a cardinality of n number of paths multiplied by v number of VMs. You can reduce the cardinality of a result set - perhaps by monitoring errors-per-VM instead of for each path per VM.
Complexity of the alerting query consideration. Queries that data sources can process and respond to quickly consume fewer resources. Although this consideration is less important than the other considerations listed above, if you have reduced those as much as possible, looking at individual query performance could make a difference.

Each evaluation of an alert rule generates a set of alert instances; one for each member of the result set. The state of all the instances is written to the alert_instance table in the Grafana SQL database. This number of write-heavy operations can cause issues when using SQLite.

Grafana Alerting exposes a metric, grafana_alerting_rule_evaluations_total that counts the number of alert rule evaluations. To get a feel for the influence of rule evaluations on your Grafana instance, you can observe the rate of evaluations and compare it with resource consumption. In a Prometheus-compatible database, you can use the query rate(grafana_alerting_rule_evaluations_total[5m]) to compute the rate over 5 minute windows of time. It’s important to remember that this isn’t the full picture of rule evaluation. For example, the load is unevenly distributed if you have some rules that evaluate every 10 seconds, and others every 30 minutes.

These factors all affect the load on the Grafana instance, but you should also be aware of the performance impact that evaluating these rules has on your data sources. Alerting queries are often the vast majority of queries handled by monitoring databases, so the same load factors that affect the Grafana instance affect them as well.

Limited rule sources support

Grafana Alerting can retrieve alerting and recording rules stored in most available Prometheus, Loki, Mimir, and Alertmanager compatible data sources.

It does not support reading or writing alerting rules from any other data sources but the ones previously mentioned at this time.

Prometheus version support

The latest two minor versions of both Prometheus and Alertmanager are supported. We cannot guarantee that older versions work.

As an example, if the current Prometheus version is 2.31.1, >= 2.29.0 is supported.

The Grafana Alertmanager can only receive Grafana managed alerts

Grafana cannot be used to receive external alerts. You can only send alerts to the Grafana Alertmanager using Grafana managed alerts.

You have the option to send Grafana-managed alerts to an external Alertmanager, you can find this option in the Admin tab on the Alerting page.

For more information, refer to this GitHub issue.

High load on database caused by a high number of alert instances

If you have a high number of alert rules or alert instances, the load on the database can get very high.

By default, Grafana performs one SQL update per alert rule after each evaluation, which updates all alert instances belonging to the rule.

You can change this behavior by disabling the alertingSaveStateCompressed feature flag. In this case, Grafana performs a separate SQL update for each state change of an alert instance. This configuration is rarely recommended, as it can add significant database overhead for alert rules with many instances.

Save state periodically

You can also reduce database load by writing states periodically instead of after every evaluation.

There are two approaches for periodic state saving:

Compressed periodic saves

You can combine compressed alert state storage with periodic saves by enabling both alertingSaveStateCompressed and alertingSaveStatePeriodic feature toggles together.

This approach groups all alert instances by rule UID and compresses them together for efficient storage.

When both feature toggles are enabled, Grafana will save compressed alert states at the interval specified by state_periodic_save_interval. Note that in compressed mode, the state_periodic_save_batch_size setting is ignored as the system groups instances by rule UID rather than by batch size.

Batch-based periodic saves

Alternatively, you can use batch-based periodic saves without compression:

This approach processes individual alert instances in batches of a specified size.

Enable the alertingSaveStatePeriodic feature toggle.
Disable the alertingSaveStateCompressed feature toggle.

By default, it saves the states every 5 minutes to the database and on each shutdown. The periodic interval can also be configured using the state_periodic_save_interval configuration flag. During this process, Grafana deletes all existing alert instances from the database and then writes the entire current set of instances back in batches in a single transaction. Configure the size of each batch using the state_periodic_save_batch_size configuration option.

Jitter for batch-based periodic saves

To further distribute database load, you can enable jitter for periodic state saves by setting state_periodic_save_jitter_enabled = true. When jitter is enabled, instead of saving all batches simultaneously, Grafana spreads the batch writes across a calculated time window of 85% of the save interval.

How jitter works:

Calculates delays for each batch: delay = (batchIndex * timeWindow) / (totalBatches - 1)
Time window uses 85% of save interval for safety margin
Batches are evenly distributed across the time window
All operations occur within a single database transaction

Configuration example:

[unified_alerting]
state_periodic_save_jitter_enabled = true
state_periodic_save_interval = 1m
state_periodic_save_batch_size = 100

Performance impact: For 2000 alert instances with 1-minute interval and 100 batch size:

Creates 20 batches (2000 ÷ 100)
Spreads writes across 51 seconds (85% of 60s)
Batch writes occur every ~2.68 seconds

This helps reduce database load spikes in environments with high alert cardinality by distributing writes over time rather than concentrating them at the beginning of each save cycle.

The time it takes to write to the database periodically can be monitored using the state_full_sync_duration_seconds metric that is exposed by Grafana.

If Grafana crashes or is force killed, then the database can be up to state_periodic_save_interval seconds out of date. When Grafana restarts, the UI might show incorrect state for some alerts until the alerts are re-evaluated. In some cases, alerts that were firing before the crash might fire again. If this happens, Grafana might send duplicate notifications for firing alerts.

Alert rule migrations for Grafana 11.6.0

When you upgrade to Grafana 11.6.0, a migration is performed on the alert_rule_versions table. If you experience a 11.6.0 upgrade that causes a migration failure, then your alert_rule_versions table has too many rows. To fix this, you need to truncated the alert_rule_versions table for the migration to complete.