Introduction to Grafana Alerting on Grafana Labs

Alert rules

Fri, 03 Apr 2026 19:43:06 +0000

Alert rules

An alert rule is a set of evaluation criteria for when an alert rule should fire. An alert rule consists of:

Queries that select the dataset to evaluate.
An alert condition (the threshold) that the query must meet or exceed to trigger the alert instance.
An interval that specifies the frequency of alert rule evaluation and a duration indicating how long the condition must be met to trigger the alert instance.
Other customizable options, including expressions, labels, annotations, error and no data handling, notification routing, and more.

Alert rule types

Grafana Alerting inherits the Prometheus Alerting model for defining alert rules and supports two alert rule types:

Data source-managed alert rules

These alert rules can only query Prometheus-based data sources such as Mimir, Loki, and Prometheus. The rules are stored in the data source.

Grafana Alerting supports this alert rule type for horizontal scalability with these data sources.
Grafana-managed alert rules

The recommended alert rule type in Grafana Alerting.

These alert rules can query a wider range of backend data sources, including multiple data sources in a single alert rule. They support expression-based transformations, advanced alert conditions, images in notifications, handling of error and no data states, and more.

You can find the supported public data sources in the Grafana Plugins directory. For step-by-step instructions, see Configure Grafana-managed alert rules.

Recording rules

Similar to alert rules, recording rules are evaluated periodically. A recording rule pre-computes frequently used or computationally expensive queries, and saves the results as a new time series metric.

The new recording metric can then be used in alert rules and dashboards to optimize their queries. For further details, refer to Create recording rules.

Alert rule evaluation

Fri, 03 Apr 2026 19:43:06 +0000

Alert rule evaluation

The criteria determining when an alert rule fires are based on three settings:

Evaluation group: how frequently the alert rule is evaluated.
Pending period: how long the condition must be met to start firing.
Keep firing for: how long the alert continues to fire after the condition is no longer met.

Set alert rule evaluation

These settings affect how alert instances progress through their lifecycle.

Alerting lifecycle

Each alert rule can generate one or more alert instances.

An alert instance transitions between these common states based on how long the alert condition remains met or not met.

State	Description
Normal	The state of an alert when the condition (threshold) is not met.
Pending	The state of an alert that has breached the threshold but for less than the pending period.
Alerting	The state of an alert that has breached the threshold for longer than the pending period.
Recovering	The state of a firing alert when the threshold is no longer breached, but for less than the keep firing for period.

If an alert rule changes (except for updates to annotations, the evaluation interval, or other internal fields), its alert instances reset to the Normal state, and update accordingly during the next evaluation.

Note
To learn about additional alert instance states, see No Data and Error states.

Notification routing

Alert instances are routed for notifications in two scenarios:

When they transition to the Alerting state.
When they transition to Normal state and marked as Resolved, either from the Alerting or Recovering state.

Evaluation group

Every alert rule and recording rule is assigned to an evaluation group. Each evaluation group contains an evaluation interval that determines how frequently the rule is checked. For instance, the evaluation may occur every 10s, 30s, 1m, 10m, etc.

Rules can be evaluated concurrently or sequentially. For details, see How rules are evaluated within a group.

Pending period

You can set a Pending period to prevent unnecessary notifications caused by temporary issues.

When the alert condition is met, the alert instance enters the Pending state. It remains in this state until the condition has been continuously true for the entire Pending period.

This ensures the condition breach is stable before the alert transitions to the Alerting state and routed for notification.

Normal -> Pending -> Alerting^*

You can also set the Pending period to zero to skip the Pending state entirely and transition to Alerting immediately.

Keep firing for

You can set a Keep firing for period to avoid repeated firing-resolving-firing notifications caused by flapping conditions.

When the alert condition is no longer met during the Alerting state, the alert instance enters the Recovering state.

Alerting → Recovering → Normal (Resolved)^*
After the Keep firing for period elapses, the alert transitions to the Normal state and is marked as Resolved.
If the alert condition is met again, the alert transitions back to the Alerting state, and no new notifications are sent.

You can also set the Keep firing for period to zero to skip the Recovering state entirely.

Evaluation example

Keep in mind:

One alert rule can generate multiple alert instances—one for each series or dimension produced by the rule’s query. Alert instances from the same alert rule may be in different states.
Only alert instances in the Alerting and Normal (Resolved) state are routed for notifications.

Consider an alert rule with an evaluation interval set at every 30 seconds and a pending period of 90 seconds. The evaluation occurs as follows:

Time	Condition	Alert instance state	Pending counter
00:30 (first evaluation)	Not met	Normal	-
01:00 (second evaluation)	Breached	Pending	0s
01:30 (third evaluation)	Breached	Pending	30s
02:00 (fourth evaluation)	Breached	Pending	60s
02:30 (fifth evaluation)	Breached	Alerting 📩	90s

With a keep firing for period of 0 seconds, the alert instance transitions immediately from Alerting to Normal, and marked as Resolved:

Time	Condition	Alert instance state	Pending counter
03:00 (sixth evaluation)	Not met	Normal ^Resolved 📩	120s
03:30 (seventh evaluation)	Not met	Normal	150s

Notifications

Fri, 03 Apr 2026 19:43:06 +0000

Notifications

Choosing how, when, and where to send your alert notifications is an important part of setting up your alerting system. These decisions have a direct impact on your team’s ability to receive the necessary information to resolve issues quickly and minimize alert noise.

Start defining your contact points to specify how to receive your alert notifications. Then, configure your alert rules to send their alerts to either a contact point or use the Notification Policy Tree to flexibly route alerts to contact points.

How it works at a glance

Grafana alerting periodically evaluates your alert rules.
It triggers notifications for alert instances that are firing or resolved.
You can configure an alert rule to send notifications to a contact point or route them through notification policies for greater flexibility.
To reduce the number of notifications, you can group related alerts into a single notification by using label grouping and notification timings.

Fundamentals

Contact points

Contact points contain the configuration for sending alert notifications, specifying destinations like email, Slack, IRM, webhooks, and their notification messages.

A contact point is a list of integrations, each sending a message to a specific destination.

By default, notification messages include common alert details, such as the number of alerts, alert names, labels, annotations, and other alert information. You can also customize notification messages and use notification templates.

First, create the contact point and test the notifications. Then, configure the alert rule to send its notifications to either a contact point or through Notification Policies.

Notification policies

Notification policies are the backbone of a comprehensive alerting system. They provide a flexible and effective method to route alerts to distinct contact points, helping reduce alert noise while ensuring no alerts are missed.

The notification policy tree is responsible for:

Defining nested policies that can inherit or overwrite parent notification settings.
Routing alerts by matching alert labels to the appropriate notification policy.

Each notification policy handles specific tasks:

Deciding which contact point receives the alert notification.
Controlling when to send notifications based on its notification timing options.
Grouping multiple alerts into a single notification to reduce alert noise.

Group alert notifications

When something fails in our system, our alerting setup can easily trigger hundreds or even thousands of alert instances (notifications). Several alert rules often fail simultaneously. Additionally, each alert rule may generate multiple alert instances.

Grouping alert notifications is commonly necessary to avoid bombarding our alert inbox. Grouping combines similar alert instances in a given period into one single notification.

Notification grouping uses:

Labels: Group alert instances of the same type by using labels.
Timing options: Wait for a specified period before sending the notification, allowing for the grouping of incoming alert instances.

Templates, silences and mute timings

Grafana Alerting provides advanced notification capabilities that you’ll find useful as you and your team refine your initial alerting system.

For instance, you can customize notifications with shared templates that provide actionable alert information and can be reused for multiple notifications.

Additionally, you can use silences and mute timings to pause or suppress notifications without interrupting alert evaluation.

Architecture

Grafana Alerting is built on the Prometheus model, which separates two main components for scalability and performance:

An alert generator that evaluates alert rules and sends firing and resolved alerts to the alert receiver.
An alert receiver (also known as Alertmanager) that receives the alerts and is responsible for sending their notifications.

Grafana includes a custom Alertmanager that extends the Prometheus Alertmanager to manage and deliver alert notifications. You can also configure Grafana Alerting to work with other Alertmanagers.

Templates

Fri, 03 Apr 2026 19:43:06 +0000

Templates

Use templating to customize, format, and reuse alert notification messages. Create more flexible and informative alert notification messages by incorporating dynamic content, such as metric values, labels, and other contextual information.

In Grafana, you have various options to template your alert notification messages:

Alert rule annotations
- Annotations add extra information, like summary and description, to alert instances for notification messages.
- Template annotations to display query values that are meaningful to the alert, for example, the server name or the threshold query value.
Alert rule labels
- Labels are used to differentiate an alert instance from all other alert instances.
- Template labels to add an additional label based on a query value, or when the labels from the query are incomplete or not descriptive enough.
- Avoid displaying query values in labels as this can create numerous alert instances—use annotations instead.
Notification templates
- Notification templates are used by contact points for consistent messaging in notification titles and descriptions.
- Template notifications when you want to customize the appearance and information of your notifications.
- Avoid using notification templates to add extra information to alert instances—use annotations instead.

Tip
For a practical example of templating, refer to our Getting Started with Templating tutorial.

This diagram illustrates the entire templating process, from querying labels and templating the alert summary and notification to the final alert notification message.

In this diagram:

The alert rule query returns 12345, along with the values of the instance and job labels.
This query result breaches the alert rule condition, firing the alert instance.
The alert instance generates an annotation summary, defined by the template used in the alert rule summary. In this case, it displays the value of the instance label: server1.
The Alertmanager receives the firing alert instance, including the final annotation summary, and determines the contact point that will process the alert.
The Alertmanager uses the contact point’s notification template to format the message, then sends the notification to the configured destination(s)—an email address.

Template annotations

Annotations can be defined in the alert rule to add extra information to alert instances.

When creating an alert rule, Grafana suggests several optional annotations, such as description, summary, and runbook_url, which help identify and respond to alerts. You can also create custom annotations.

Annotations are key-value pairs, and their values can contain a combination of text and template code that is evaluated when the alert fires.

Annotations can contain plain text, but you should template annotations if you need to display query values that are relevant to the alert, for example:

Show the query value that triggers the alert.
Include labels returned by the query that identify the alert.
Format the annotation message depending on a query value.

Here’s an example of templating an annotation, which explains where and why the alert was triggered. In this case, the alert triggers when CPU usage exceeds a threshold, and the summary annotation provides the relevant details.

CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes.

The outcome of this template would be:

CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes.

Implement annotations that provide meaningful information to respond to your alerts. Annotations are displayed in the Grafana alert detail view and are included by default in notifications.

For more details on how to template annotations, refer to Template annotations and labels.

Template labels

Labels are used to differentiate one alert instance from all other alert instances, as the set of labels uniquely identifies an alert instance. Notification policies and silences use labels to handle alert instances.

You can also template labels based on query results. This is helpful if the labels you get from your query aren’t detailed enough. For instance:

Add a new label to change how alerts are identified and grouped into different alert groups.
Add a new label used by notification policies or silences to manage how the alert is handled.

Here’s an example of templating a new env label based on the value of a query label:

{{- if eq $labels.instance "prod-server-1" -}}
production
{{- else if eq $labels.instance "staging-server-1" -}}
staging
{{- else -}}
development
{{- end -}}

For more details on how to template labels, refer to Template annotations and labels.

Template notifications

Notification templates allow you to customize the content of your notifications, such as the subject of an email or the body of a Slack message.

Notification templates differ from templating annotations and labels in the following ways:

Notification templates are assigned to the Contact point, rather than the alert rule.
If not specified, the contact point uses a default template that includes relevant alert information.
The same template can be shared across multiple contact points, making it easier to maintain and ensuring consistency.
Notification templates should not be used to add additional information to individual alerts—use annotations for that purpose.
While both annotation/label templates and notification templates use the same templating language, the available variables and functions differ. For more details, refer to the notification template reference and annotation/label template reference.

Here is an example of a notification template that summarizes all firing and resolved alerts in a notification group:

{{ define "alerts.message" -}}
  {{ if .Alerts.Firing -}}
    {{ len .Alerts.Firing }} firing alert(s)
    {{ template "alerts.summarize" .Alerts.Firing }}
  {{- end }}
  {{- if .Alerts.Resolved -}}
    {{ len .Alerts.Resolved }} resolved alert(s)
    {{ template "alerts.summarize" .Alerts.Resolved }}
  {{- end }}
{{- end }}

{{ define "alerts.summarize" -}}
  {{ range . -}}
  - {{ index .Annotations "summary" }}
  {{ end }}
{{ end }}

The notification message to the contact point would look like this:

1 firing alert(s)
- The database server db1 has exceeded 75% of available disk space. Disk space used is 76%, please resize the disk size within the next 24 hours.

1 resolved alert(s)
- The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.

For more details, refer to Template notifications.