Configure alert rules on Grafana Labs

Configure Grafana-managed alert rules

Fri, 03 Apr 2026 12:35:46 -0500

Configure Grafana-managed alert rules

Grafana-managed alert rules are the default way to create alert rules in Grafana.

Grafana-managed rules inherit their model from Prometheus Alerting and extend it with greater flexibility—such as multi-data source queries, expression-based transformations, advanced alert conditions, images in notifications, custom states, and more.

To create or edit Grafana-managed alert rules, follow the instructions below.

Tip
For quick-start tutorials on key alerting features, see Getting started with Grafana Alerting tutorials.

Before you begin

Before you create Grafana-managed alert rules, review the following requirements and options.

Supported data sources

Grafana-managed alert rules can query backend data sources when the data source’s plugin.json file sets {"backend": true, "alerting": true}.

Before you create an alert rule, verify that the data sources you plan to query are compatible and properly configured.

You can find the public data sources that support alert rules in the Grafana Plugins directory.

Alert rule limits in Grafana Cloud

In Grafana Cloud, the number of Grafana-managed alert rules you can create depends on your Grafana Cloud plan.

Free Forever plan: You can create up to 100 free alert rules, with each alert rule having a maximum of 1000 alert instances.
All paid plans: They have a soft limit of 2000 alert rules and support unlimited alert instances. To increase the limit, open a support ticket from the Cloud portal.

Permissions

Only users with Edit permissions for the folder storing the rules can edit or delete Grafana-managed alert rules. Only admins can restore deleted Grafana-managed alert rules.

Provisioning

Note that if you delete an alert resource created in the UI, you can no longer retrieve it.

To backup and manage alert rules, you can provision alerting resources using options such as configuration files, Terraform, or the Alerting API.

Default versus Advanced options

Grafana Alerting provides multiple configuration options to support different alerting needs.

To simplify alert rule creation, some settings display a Default and an Advanced configuration view:

Use Default to create basic query conditions and route alerts directly to a contact point. Use Advanced for more complex query definitions, or to route alerts using notification policies.

You can switch between options at any time. After you create an alert rule, Grafana remembers your last selection and uses it as the default for the next rule.

If you prefer to always use advanced configuration, you can disable the Default modes by disabling the alertingQueryAndExpressionsStepMode and alertingNotificationsStepMode feature toggles.

Set alert rule name

Click Alerts & IRM -> Alert rules -> + New alert rule.
Enter a name to identify your alert rule.

This name is displayed in the alert rule list. It is also the alertname label for every alert instance that is created from this rule.

Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

The Default option allows to configure one query and one alert condition. The Advanced option allows multiple queries and expressions for more complex rule definitions.

You can switch between Default and Advanced options at any time.

Switching from Advanced to Default may result in queries and expressions that can’t be converted. In this case, a warning message asks if you want to continue to reset to default settings.

If the Default versus Advanced options feature is not enabled in your Grafana instance, follow the Advanced options instructions.

Set folder and labels

Organize your alert rule with a folder and set of labels.

In the Labels section, you can optionally choose whether to add labels to organize your alert rules and their notifications. For more details, refer to alert rule labels.

Select a folder or click + New folder.
Add labels, if required.

Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.

Configure alert evaluation behavior

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.

Select an evaluation group or click + New evaluation group.

If you are creating a new evaluation group, specify the interval for the group.

All rules within the same group are evaluated concurrently over the same time interval.
Enter a pending period.

The pending period is the period in which an alert rule can be in breach of the condition until it fires.

After a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.
Optionally, set the Keep firing for period.

You can set the minimum amount of time that an alert remains firing after the breached threshold expression no longer returns any results. This sets an alert to a “Recovering” state for the duration of time set here. The Recovering state can be used to reduce noise from flapping alerts. Select “none” stop an alert from firing immediately after the breach threshold is cleared.
Turn on pause alert notifications, if required.

You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts. Pausing stops alert rule evaluation and doesn’t create any alert instances. This is different to mute timings, which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.

In Configure no data and error handling, you can define the alerting behavior and alerting state for two scenarios:

When the evaluation returns No data or all values are null.
When the evaluation returns Error or timeout.

Configure no data and error handling

Configure	Set alert state	Description
No Data	No Data	The default option for No Data events. Sets alert instance state to `No Data`. The alert rule immediately creates a new `DatasourceNoData` alert instance after evaluation, with the alert rule’s name, UID, and the data source UID as labels.
Error	Error	The default option for Error events. Sets alert instance state to `Error`. The alert rule immediately creates a new `DatasourceError` alert instance after evaluation, with the alert rule’s name, UID, and the data source UID as labels.
No Data or Error	Alerting	Sets the alert instance state to `Pending` and then transitions to `Alerting` once the pending period ends. If you sent the pending period to 0, the alert instance state is immediately set to `Alerting`.
No Data or Error	Normal	Sets alert instance state to `Normal`.
No Data or Error	Keep Last State	Maintains the alert instance in its last state. Useful for mitigating temporary issues.

For more details, refer to alert instance states and modify the no data or error state.

In Configure no data and error handling, you can also configure Missing series evaluations to resolve: how many consecutive evaluation intervals must pass without data before an alert instance is considered stale.

Configure notifications

Configure who receives notifications when the alert rule fires.

The Default option allows to select a contact point to handle notifications for this alert rule. The Advanced option routes notifications through notification policies.

You can switch between Default and Advanced options at any time.

If the Default versus Advanced options feature is not enabled in your Grafana instance, follow the Advanced options instructions.

Configure notification message

Use annotations to add information to alert messages that can help respond to the alert.

Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.

Grafana provides several optional annotations.

Optional: Add a summary.

Short summary of what happened and why.
Optional: Add a description.

Description of what the alert rule does.
Optional: Add a Runbook URL.

Webpage where you keep your runbook for the alert
Optional: Add a custom annotation.

Add any additional information that could help address the alert.
Optional: Link dashboard and panel.

Link the alert rule to a panel to facilitate alert investigation.
Click Save rule.

Create and link alert rules to panels

Fri, 03 Apr 2026 12:35:46 -0500

Create and link alert rules to panels

Grafana allows you to link an alert rule to a dashboard panel. This can help you:

Inform alert responders about where to investigate and which data to examine.
Visualize the alert state directly from dashboards.
Include a screenshot of the panel in notification messages.

An alert rule is linked to a panel by setting the __dashboardUid__ and __panelId__ annotations. Both annotations must be set together.

Link alert rules to panels

When configuring the alert rule, you can set the dashboard and panel annotations as shown in this video.

Configure the alert rule.
In the Configure notification message section, click Link dashboard and panel.
Select an existing dashboard, then choose a panel from the selected dashboard.
Complete the alert rule configuration and click Save rule to initiate the alert rule.

You can then view the alert state on the panel.

By default, notification messages include a link to the dashboard panel. Additionally, you can enable displaying panel screenshots in notifications.

Create alert rules from panels

To streamline alert creation, you can create an alert rule directly from a panel.

Navigate to a dashboard in the Dashboards section.
Hover over the top-right corner of a panel and click the panel menu icon.
From the dropdown menu, select More… > New alert rule.
This opens the Edit rule form and pre-fills some values:
- Sets the annotations to the corresponding dashboard and panel.
- Sets the alert rule query using the panel query.
Complete the alert rule configuration and click Save rule to initiate the alert rule.

You can then view the alert state on the panel.

By default, notification messages include a link to the dashboard panel. Additionally, you can enable displaying panel screenshots in notifications.

Note
Changes to panel and alert rule queries aren’t synchronized. If you change a query, you have to update it in both the panel and the alert rule.

Access linked alert rules from panels

This option is available only in time series panels. To access alert rules associated to a time series panel, complete the following steps.

Hover over the top-right corner of the panel and click the panel menu icon.
Click Edit.
Click the Alert tab to view existing alert rules or create a new one.

Tip
For a practical example, refer to our Getting started: Link alerts to visualizations tutorial.

Import data source-managed rules to Grafana-managed rules

Fri, 03 Apr 2026 12:35:46 -0500

Import data source-managed rules to Grafana-managed rules

You can convert existing alert rules from data sources such as Mimir, Loki, and Prometheus into Grafana-managed alert rules. This enables you to operate and manage these rules using Grafana Alerting.

This guide explains two methods for importing data source-managed rules:

Using the Grafana Alerting user interface to import rules from connected data sources or Prometheus rule YAML files.
Using command-line tools like mimirtool and cortextool to import rules from YAML files.

Importing rules is a safe operation: the original data source–managed rules remain intact in their original location. During import, the rules are converted into Grafana-managed rules while preserving their configuration and behavior.

Choose the method that best fits your workflow. As a best practice, test and validate your import process before migrating all alert rules.

How it works

When you use any of the import methods, data source–managed rules are copied to another folder as Grafana-managed rules.

The original data source–managed rules remain intact in their original location.

The copied rules are converted to Grafana-managed rules, preserving their behavior by using equivalent Grafana-managed features. The following settings are applied during the conversion:

Unique UIDs

The newly created rules are assigned unique UIDs. If you don’t want a UID to be auto-generated, you can specify one using the __grafana_alert_rule_uid__ label.
Query offset

A query offset is applied to each rule. For example, an offset of 1m adjusts the query’s time range to To: now-1m.

The rule query offset is taken from the query_offset value in the rule group configuration. If empty, it defaults to the rule_query_offset configuration setting, which is 1m by default.
Rule query conversion

For alert rules, adds prometheus_math and threshold expressions to preserve Prometheus no data behavior, ensuring the alert stays in Normal state when query returns no data.
Missing series evaluations to resolve

The Missing series evaluations to resolve setting is set to 1 to replicate Prometheus’s alert eviction behavior.
Rule group labels

Labels defined at the rule group level are added as labels to each imported rule within the group.
Sequential evaluation

Imported rules are evaluated sequentially within each rule group, mirroring Prometheus behavior. This differs from native Grafana-managed alert rules, where the evaluation order is not enforced. For more details, refer to evaluation strategies.
Feature compatibility

The rule group limit option and the query function within alert rule templates are not currently supported in Grafana-managed rules. If the limit option is present, the import fails. However, rules with query in templates are imported.

Note
Rules with the label __grafana_origin are not included in rule imports. These rules are typically created by apps such as Kubernetes Monitoring, Synthetic Monitoring, and other Grafana plugins.

Import rules with Grafana Alerting

You can use the Grafana Alerting user interface to import rules from the following sources:

Connected Mimir and Loki data sources with the ruler API enabled
Prometheus YAML rule files

Imported rules using this method are editable in the user interface. Like regular Grafana-managed rules, you can later export them for provisioning.

Before you begin

To use Grafana Alerting to migrate rules, you need the following RBAC permissions:

Alerting: Rules Writer, Set provisioning status.
Datasources: Reader.
Folders: Creator.

The Folders permission is optional and only necessary if you want to create new folders for your target namespace. If your account doesn’t have permissions to view a namespace, the tool creates a new one.

To convert data source-managed alert rules to Grafana managed alerts:

Go to Alerting > Alert rules.
Navigate to the Data source-managed alert rules section and click Import to Grafana-managed rules.
Choose the Import source from which you want to import rules:
- Select Existing data source-managed rules to import rules from connected Mimir or Loki data sources with the ruler API enabled.
- Select Prometheus YAML file to import rules by uploading a Prometheus YAML rule file.
In the Data source dropdown, select the data source that the imported alert rules will query.
(Optional) In Additional settings, select a target folder or designate a new folder to import the rules into.

If you import the rules into an existing folder, don’t choose a folder with existing alert rules, as they could get overwritten.
(Optional) Select a Namespace and/or Group to determine which rules are imported.
(Optional) Turn on Pause imported alerting rules.

Pausing stops alert rule evaluation and doesn’t create any alert instances for the newly created Grafana-managed alert rules.
(Optional) Turn on Pause imported recording rules.

Pausing stops alert rule evaluation behavior for the newly created Grafana-managed alert rules.
(Optional) In the Target data source of the Recording rules section, you can select the data source to which the imported recording rules will write metrics. By default, it is the data source selected in the Data source dropdown.
Click Import.

A preview shows the rules that will be imported. If your target folder contains folders with the same name of the imported folders, a warning displays to inform you. You can explore the warning to see a list of folders that might be overwritten.

Click Yes, import to import the rules.

Import rules with command-line tools

You can also use command-line tools to import data source-managed rules as Grafana-managed rules: use mimirtool for Mimir and Prometheus rules, and cortextool for Loki rules.

Both tools provide rules commands to import rules groups. For example:

rules load can import rule groups from files.
rules sync can read rule files and applies only the differences compared to existing Grafana-managed rules. This is useful for automation workflows and pipelines.

By default, rules imported using the API or command-line tools are Provisioned and not editable in the user interface. To make them editable, enable the X-Disable-Provenance header.

Before you begin

You need to have installed mimirtool or cortextool (version 0.11.3 or later).

You need a service account with the following RBAC permissions:

Alerting: Rules Reader, Rules Writer, Set provisioning status.
Datasources: Reader.
Folders: Creator, Reader, Writer.

You need to a service account token with your service account. For more details, refer to service accounts and service account tokens.

mimirtool

To convert and import them into a Grafana instance, you can use the mimirtool rules load command:

MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool rules load rule_file.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<DATASOURCE_UID_QUERY_TARGET>"

This command imports Prometheus alert rules defined in rule_file.yaml as Grafana-managed alert rules. It’s important to know that:

When using the <GRAFANA_BASE_URL>/api/convert/ endpoint, mimirtool interacts with Grafana—not with a Mimir instance. In this case, MIMIR_TENANT_ID must always be set to 1.
The X-Grafana-Alerting-Datasource-UID header configures the data source that the imported alert rules will query. Use multiple --extra-headers flags to include other optional headers.

Similarly, the rules sync command can import and update Grafana-managed alert rules.

MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool rules sync rule_file.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<DATASOURCE_UID_QUERY_TARGET>" \
  --concurrency 1

The --concurrency flag must be set to 1, as the default value of 8 may cause API errors.

This sync command reads rules from the file, compares them with the existing Grafana-managed rules in the instance, and applies only the differences—creating, updating, or deleting rules as needed.

## Sync Summary: 0 Groups Created, 1 Groups Updated, 0 Groups Deleted

For more information other Mimirtool commands and options, see the Mimirtool documentation and the Mimir HTTP Rule API documentation.

cortextool

For Loki alert rules, use cortextool (version 0.11.3 or later) with the --backend=loki flag. For example:

CORTEX_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
CORTEX_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
CORTEX_TENANT_ID=1 \
cortextool rules load loki_rules.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<LOKI_DATASOURCE_UID_QUERY_TARGET>" \
  --backend=loki

Optional Headers

Additional configuration headers for more granular import control include the following:

`X-Disable-Provenance`

When this header is set to true:

The imported rules are not marked as provisioned.
They can then be edited in the Grafana UI.
They are excluded from the GET and DELETE operations on the /api/convert endpoints.

Do not enable this header when using the rules sync command, as it relies on the GET and DELETE operations to detect and update existing rules.

`X-Grafana-Alerting-Alert-Rules-Paused`

Set to true to import alert rules in paused state.

`X-Grafana-Alerting-Recording-Rules-Paused`

Set to true to import recording rules in paused state.

`X-Grafana-Alerting-Datasource-UID`

The UID of the data source to use for alert rule queries.

If not specified in the header, Grafana uses the configured default from unified_alerting.prometheus_conversion.default_datasource_uid. If neither the header nor the configuration option is provided, the request fails.

`X-Grafana-Alerting-Target-Datasource-UID`

The UID of the target data source for recording rules. If not specified, the value from X-Grafana-Alerting-Datasource-UID is used.

`X-Grafana-Alerting-Folder-UID`

Enter the UID of the target destination folder for imported rules.

`X-Grafana-Alerting-Notification-Settings`

JSON-encoded AlertRuleNotificationSettings object that allows setting the contact point for the alert rules.

AlertRuleNotificationSettings object

When you set X-Grafana-Alerting-Notification-Settings, the header value must be a JSON-encoded object with the following keys:

Field	Type	Required	Example	Description
`receiver`	`string`	Yes	`"grafana-default"`	Name of the contact point (receiver) to which alerts are routed. Must exist in Grafana before import.
`group_by`	`[]string`	No	`["alertname","grafana_folder","cluster"]`	Label set used by Alertmanager to aggregate alerts into a single notification.
`group_wait`	`duration`	No	`"30s"`	How long Alertmanager waits before sending the first notification for a new group.
`group_interval`	`duration`	No	`"5m"`	Time to wait before adding new alerts to an existing group’s next notification.
`repeat_interval`	`duration`	No	`"4h"`	Minimum time before a previously-sent notification is repeated. Must not be less than `group_interval`.
`mute_time_intervals`	`[]string`	No	`["maintenance"]`	One or more mute time interval names that silence alerts during those windows.
`active_time_intervals`	`[]string`	No	`["maintenance"]`	List of active time interval names. Alerts are suppressed unless the current time matches one of them.

Compatible endpoints

The API endpoints listed in this section are supported in Grafana and are used by mimirtool and cortextool, as shown earlier. These endpoints are compatible with Mimir HTTP API.

In these endpoints, a “namespace” corresponds to a folder title in Grafana.

The POST endpoints can be used to import data source–managed alert rules. They accept requests in both YAML and JSON. If no media type is specified, YAML is assumed.

Endpoint	Method	Summary	Mimir equivalent
POST	`/convert/prometheus/config/v1/rules`	Create or update multiple rule groups across multiple namespaces. Requires `X-Grafana-Alerting-Datasource-UID`.	None
POST	`/convert/prometheus/config/v1/rules/:namespaceTitle`	Create or update a single rule group in a namespace. Requires `X-Grafana-Alerting-Datasource-UID`.	Set rule group

The GET and DELETE endpoints work only with provisioned and imported alert rules. All GET endpoints support both JSON and YAML response formats based on the Accept header: use application/json for JSON responses, or application/yaml for YAML responses. YAML is the default format when no Accept header is specified.

Endpoint	Method	Summary	Mimir equivalent
GET	`/convert/prometheus/config/v1/rules`	Get all imported rule groups across all namespaces.	List rule groups
GET	`/convert/prometheus/config/v1/rules/:namespaceTitle`	Get imported rule groups in a specific namespace.	Get rule groups by namespace
GET	`/convert/prometheus/config/v1/rules/:namespaceTitle/:group`	Get imported rule group in a specific namespace.	Get rule group
DELETE	`/convert/prometheus/config/v1/rules/:namespaceTitle`	Delete all imported alert rules in a namespace.	Delete namespace
DELETE	`/convert/prometheus/config/v1/rules/:namespaceTitle/:group`	Delete a specific imported rule group.	Delete rule group

Create or update multiple rule groups

POST /convert/prometheus/config/v1/rules

Creates or updates multiple rule groups across multiple namespaces. This endpoint expects a request with a map of namespace titles to arrays of rule groups, and returns 202 on success.

This endpoint has no Mimir equivalent and is Grafana-specific for bulk operations.

Example request body

namespace1:
  - name: MyGroupName1
    rules:
      - alert: MyAlertName1
        expr: up == 0
        labels:
          severity: warning
namespace2:
  - name: MyGroupName2
    rules:
      - alert: MyAlertName2
        expr: rate(http_requests_total[5m]) > 0.1
        labels:
          severity: critical

Configure data source-managed alert rules

Fri, 03 Apr 2026 12:35:46 -0500

Configure data source-managed alert rules

Data source-managed alert rules are alert rules that are stored in the data source, such as in Prometheus or Loki, rather than in Grafana.

In Grafana Alerting, you can:

Create and edit data source-managed rules for Grafana Mimir and Grafana Loki data sources.
View rules from Prometheus data sources when Manage alerts via Alerting UI is enabled. However, you cannot create or edit these rules in Grafana.
Import data source-managed rules from Loki, Mimir, and Prometheus into Grafana-managed rules.

Note
Data source-managed rules are supported for horizontal scalability, but they can introduce more operational complexity than Grafana-managed alert rules.

We recommend using Grafana-managed alert rules whenever possible, as they provide a richer feature set and better integration with the full Grafana Alerting workflow.

Comparison with Grafana-managed rules

The table below compares Grafana-managed and data source-managed alert rules.

Feature	Grafana-managed alert rule	Data source-managed alert rule
Supported data sources	All backend data sources enabling the `alerting` option	Only supports creating rules for Mimir and Loki data sources
Mix and match data sources	Yes	No
Add expressions to transform your data and set alert conditions	Yes	No
No data and error states	Yes	No
Stale alert instances	Yes	No
Images in alert notifications	Yes	No
Role-based access control	Yes	No
Alert state history	Yes	No
Alert version history	Yes	No
Terraform provisioning	Yes	No
Recording rules	Yes	Yes
Organization	Organize and manage access with folders	Use namespaces
Alert rule evaluation	Alert evaluation is done in Grafana	Alert rule evaluation is done in the data source and allow for horizontal scaling
Scaling	Alert rules are stored in the Grafana database.	Alert rules are stored within the data source and allow for horizontal scaling

The following diagram shows the architecture of a Mimir setup that uses data source-managed alert rules.

Create data source-managed alert rules

To create or edit data source-managed alert rules, follow these instructions.

Before you begin

Verify that you have write permission to the Mimir or Loki data source. Otherwise, you cannot create or update data source-managed alert rules.

Enable the Ruler API

For more information, refer to the Mimir Ruler API or Loki Ruler API.

Mimir - use the /prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.
Loki - The local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.

Permissions

Alert rules for Mimir or Loki instances can be edited or deleted by users with Editor or Admin roles.

If you do not want to manage alert rules for a particular data source, go to its settings and clear the Manage alerts via Alerting UI checkbox.

Provisioning

Note that if you delete an alert resource created in the UI, you can no longer retrieve it.

To backup and manage alert rules, you can provision alerting resources using options such as configuration files, Terraform, or the Alerting API.

Set alert rule name

Click Alerts & IRM -> Alert rules -> + New alert rule.
Enter a name to identify your alert rule.

This name is displayed in the alert rule list. It is also the alertname label for every alert instance that is created from this rule.

Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

Note
By default, new alert rules are Grafana-managed. To switch to Data source-managed, follow these instructions.

Select a Prometheus-based data source from the drop-down list.

You can also click Open advanced data source picker to find more options.
Enter a PromQL or LogQL query, including the alert condition.
In the Rule type option, select Data source-managed.
Click Preview alerts.

Set alert evaluation behavior

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

Select a namespace or click + New namespace.
Select an evaluation group or click + New evaluation group.

If you are creating a new evaluation group, specify the interval for the group.

All rules within the same group are evaluated sequentially over the same time interval. You can reorder them from the Alert rules page.
Enter a pending period.

The pending period is the period in which an alert rule can be in breach of the condition until it fires.

Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.

Configure labels and notifications

Add labels to your alert rules to set which notification policy should handle your firing alert instances.

All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.

Add labels if you want to change the way your notifications are routed.

Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.

Configure notification message

Use annotations to add information to alert messages that can help respond to the alert.

Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.

Grafana provides several optional annotations.

Optional: Add a summary.

Short summary of what happened and why.
Optional: Add a description.

Description of what the alert rule does.
Optional: Add a Runbook URL.

Webpage where you keep your runbook for the alert
Optional: Add a custom annotation.

Add any additional information that could help address the alert.
Optional: Link dashboard and panel.

Link the alert rule to a panel to facilitate alert investigation.
Click Save rule.

Create recording rules

Fri, 03 Apr 2026 12:35:46 -0500

Configure recording rules

Recording rules allows you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.

For instance, you can create a recording rule generating a new metric, error_9001_count, which counts occurrences of a specific log error within one minute. Then, query the error_9001_count metric in dashboards and alert rules.

Recording rules can be helpful in various scenarios, such as:

Faster queries are needed: Performing heavy aggregations or querying large data sets is quicker with precomputed results than real-time queries.
Reducing system load: Precomputing specific queries in advance can reduce system overload caused by multiple simultaneous queries.
Simplifying complex aggregations: Create a new metric from complex aggregations to facilitate alert and dashboard setup.
Reusing queries across alerts: Improve efficiency by reusing the same query across similar alert rules and dashboards.

The evaluation group of the recording rule determines how often the metric is pre-computed.

Recommendations

Use frequent evaluation intervals. Set frequent evaluation intervals for recording rules. Long intervals, such as an hour, can cause the recorded metric to be stale and lead to misaligned alert rule evaluations, especially when combined with a long pending period.
Align alert evaluation with recording frequency. The evaluation interval of an alert rule that depends on a recorded metric should be aligned with the recording rule’s interval. If a recording rule runs every 3 minutes, the alert rule should also be evaluated at a similar frequency to ensure it acts on fresh data.
Use _over_time functions for instant queries. Since all alert rules are ultimately executed as an instant query, you can use functions like max_over_time(my_metric[5m]) as an instant query. This allows you to get an aggregated value over a period without using a range query and a reduce expression.

Types of recording rules

Similar to alert rules, Grafana supports two types of recording rules:

Grafana-managed recording rules, which can query any Grafana data source supported by alerting. It’s the recommended option.
Data source-managed recording rules, which can query Prometheus-based data sources like Mimir or Loki.

Template annotations and labels

Fri, 03 Apr 2026 12:35:46 -0500

Template annotations and labels

You can use templates to customize alert and notification messages, including dynamic data from alert rule queries.

In Grafana Alerting, you can template alert messages in two ways.

Template annotations and labels: In the alert rule definition, you can template annotations and labels to include extra information from query data to the alert, adding meaningful details based on the query results.
Template notifications: You can template notifications to control the content and appearance of your notifications.

How templating works

In this diagram, you can see the differences between both types of templates.

Refer to Templates Introduction for a more detailed explanation of this diagram.

Both types of templates are written in the Go templating system. However, it’s important to understand that variables and functions used in notification templates are different from those used in annotation and label templates.

Template annotations and labels: These templates add extra information to individual alert instances. Template variables like $labels and $values represent alert query data of the individual alert instance.
Template notifications: Notification templates format the notification content for a group of alerts. Variables like .Alerts include all firing and resolved alerts in the notification.

Template annotations

Annotations add additional information to alert instances and are often used to help identify the alert and guide responders on how to address the issue.

Annotations are key-value pairs defined in the alert rule. They can contain plain text or template code that is evaluated when the alert fires.

Grafana includes several optional annotations, such as description, summary, and runbook_url, which can be edited in the alert rule. You can also create your custom annotations. For example, you might create a new annotation named location to report the location of the system that triggered the alert.

Here’s an example of a summary annotation explaining why the alert was triggered, using plain text.

CPU usage has exceeded 80% for the last 5 minutes.

However, if you want to display dynamic query values in annotations, you need to use template code. Common use cases include:

Displaying the query value that triggered the alert.
Highlighting label information that identifies the alert, such as the environment, instance, or region.
Providing specific instructions based on query values.
Customizing runbook links depending on query labels.
Including contact information based on query labels.

For instance, you can template the previous example to display the specific instance and CPU value that triggered the alert.

CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes.

Alternatively, you can use the index function to print query values.

CPU usage for {{ index $labels "instance" }} has exceeded 80% ({{ index $values "A" }}) for the last 5 minutes.

The result of the annotation would be as follows.

CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes.

How to template an annotation

Complete the following steps to template an annotation.

Navigate to Alerts & IRM -> Alert rules -> create or edit an alert rule.
Scroll down to the Configure notification message section.
Copy in your template in the corresponding annotation field (summary, description, runbook_url, custom).

Preview annotation templates

You can template annotations when creating or editing an alert rule.

Two common methods are used to test and preview annotation templates:

Trigger the alert and view the alert instance state in the Grafana UI, where all annotations of the alert instance are displayed.
Use a notification template that displays all annotations, then preview the notification template using the alert instance.

Template labels

The set of labels for an alert instance is used to uniquely identify that alert among all other alert instances.

Labels determine how alerts are routed and managed for notifications, making their design key to the effectiveness of your alerting system.

Labels can be returned from an alert rule query, such as the pod label in a Kubernetes Prometheus query. You can also define additional labels in the alert rule to provide extra information for processing alerts.

Like annotations, labels are key-value pairs that can contain plain text or template code evaluated when the alert fires.

Template labels when the labels returned by your queries are insufficient. For instance:

A new label based on a query value can group a subset of alerts differently, changing how notifications are sent.
A new label based on a query value can be used in a notification policy to alter the notification contact point.

Here’s an example of templating a severity label based on the query value.

{{ if (gt $values.A.Value 90.0) -}}
critical
{{ else if (gt $values.A.Value 80.0) -}}
high
{{ else if (gt $values.A.Value 60.0) -}}
medium
{{ else -}}
low
{{- end }}

In this example, the value of the severity label is determined by the query value, and the possible options are critical, high, medium, or low. You can then use the severity label to change their notifications—for instance, sending critical alerts immediately or routing low alerts to a specific team for further review.

Note: An alert instance is uniquely identified by its set of labels.

Avoid displaying query values in labels, as this can create numerous alert instances—one for each distinct label set. Instead, use annotations for query values.

If a templated label’s value changes, it maps to a different alert instance, and the previous instance is considered stale. Learn all the details in this example using dynamic labels.

How to template a label

Complete the following steps to template a label.

Navigate to Alerts & IRM -> Alert rules -> create or edit an alert rule.
Scroll down to the Configure labels and notifications section.
Click + Add labels.
Enter a key that identifies the label.
Copy in your template in the value field.

Preview label templates

You can template label values when creating or editing an alert rule.

To preview label values, select Use notification policy, and then click on Preview routing.

More information

For further details on how to template alert rules, refer to:

Tip
For a practical example of templating, refer to our Getting Started with Templating tutorial.