Grafana alerts on Grafana Labs

What's new in Grafana alerting

Sat, 04 Apr 2026 12:26:57 +0000

What’s new in Grafana alerting

Grafana alerting has several enhancements over legacy dashboard alerting.

Multi-dimensional alerting

You can now create alerts that give you system-wide visibility with a single alerting rule. Generate multiple alert instances from a single alert rule. For example, you can create a rule to monitor the disk usage of multiple mount points on a single host. The evaluation engine returns multiple time series from a single query, with each time series identified by its label set.

Create alerts outside of Dashboards

Unlike legacy dashboard alerts, Grafana alerts allow you to create queries and expressions that combine data from multiple sources in unique ways. You can still link dashboards and panels to alerting rules using their ID and quickly troubleshoot the system under observation.

Since unified alerts are no longer directly tied to panel queries, they do not include images or query values in the notification email. You can use customized notification templates to view query values.

Create Loki and Cortex alerting rules

In Grafana alerting, you can manage Loki and Cortex alerting rules using the same UI and API as your Grafana managed alerts.

View and search for alerts from Prometheus compatible data sources

Alerts for Prometheus compatible data sources are now listed under the Grafana alerts section. You can search for labels across multiple data sources to quickly find relevant alerts.

Special alerts for alert state NoData and Error

Grafana alerting introduced a new concept of the alert states. When evaluation of an alerting rule produces state NoData or Error, Grafana alerting will generate special alerts that will have the following labels:

alertname with value DatasourceNoData or DatasourceError depending on the state.
rulename name of the alert rule the special alert belongs to.
datasource_uid will have the UID of the data source that caused the state.
all labels and annotations of the original alert rule

You can handle these alerts the same way as regular alerts by adding a silence, route to a contact point, and so on.

Note: If the rule uses many data sources and one or many returns no data, the special alert will be created for each data source that caused the alert state.

Opt-in to Grafana alerting

Sat, 04 Apr 2026 12:26:57 +0000

Opt-in to Grafana alerting

Grafana alerting is enabled by default for new Cloud and OSS installations.

Note: If you are an existing Grafana Cloud user and want to explore unified alerting, contact Grafana Support. They will enable unified alerting for your Cloud stack.

For older OSS installations that use legacy dashboard alerts, unified alerting is still an opt-in feature. This topic describes how to opt-in to Grafana alerting if you have an existing Grafana installation and the rules and restrictions that govern the migration of existing dashboard alerts to the new alerting system. You can disable Grafana alerts and use the legacy dashboard alerting if needed.

Before you begin, we recommend that you backup Grafana’s database. If you are using PostgreSQL as the backend database, then the minimum required version is 9.5.

Enable Grafana alerting

To enable Grafana alerts:

In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the unified alerts section.
Set the enabled property to true.
Next, for legacy dashboard alerting, set the enabled flag to false.
Restart Grafana for the configuration changes to take effect.

Note: The ngalert toggle previously used to enable or disable Grafana alerting is no longer available.

Before v8.2, notification logs and silences were stored on a disk. If you did not use persistent disks, you would have lost any configured silences and logs on a restart, resulting in unwanted or duplicate notifications. We no longer require the use of a persistent disk. Instead, the notification logs and silences are stored regularly (every 15 minutes). If you used the file-based approach, Grafana reads the existing file and persists it eventually.

Migrating legacy alerts to Grafana alerting system

When Grafana alerting is enabled or Grafana is upgraded to version 8.3, existing legacy dashboard alerts migrate in a format compatible with the Grafana alerting. In the Alerting page of your Grafana instance, you can view the migrated alerts alongside new alerts.

Read and write access to legacy dashboard alerts and Grafana alerts are governed by the permissions of the folders storing them. During migration, legacy dashboard alert permissions are matched to the new rules permissions as follows:

If alert’s dashboard has permissions, it will create a folder named like Migrated {"dashboardUid": "UID", "panelId": 1, "alertId": 1} to match permissions of the dashboard (including the inherited permissions from the folder).
If there are no dashboard permissions and the dashboard is under a folder, then the rule is linked to this folder and inherits its permissions.
If there are no dashboard permissions and the dashboard is under the General folder, then the rule is linked to the General Alerting folder, and the rule inherits the default permissions.

Note: Since there is no Keep Last State option for No Data in Grafana alerting, this option becomes NoData during the legacy rules migration. Option “Keep Last State” for Error handling is migrated to a new option Error. To match the behavior of the Keep Last State, in both cases, during the migration Grafana automatically creates a silence for each alert rule with a duration of 1 year.

Notification channels are migrated to an Alertmanager configuration with the appropriate routes and receivers. Default notification channels are added as contact points to the default route. Notification channels not associated with any Dashboard alert go to the autogen-unlinked-channel-recv route.

Since Hipchat and Sensu notification channels are no longer supported, legacy alerts associated with these channels are not automatically migrated to Grafana alerting. Assign the legacy alerts to a supported notification channel so that you continue to receive notifications for those alerts. Silences (expiring after one year) are created for all paused dashboard alerts.

Limitation

Grafana alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch alerting rules from all other supported data sources at this time.

Disable Grafana alerts

To disable Grafana alerts and enable legacy dashboard alerts:

In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the Grafana alerting section.
Set the enabled property to false.
For legacy dashboard alerting, set the enabled flag to true.
Restart Grafana for the configuration changes to take effect.

Note: Switching from one flavor of alerting to another can result in data loss. This is applicable to the fresh installation as well as upgraded setups.

Alerting fundamentals

Sat, 04 Apr 2026 12:26:57 +0000

Alerting fundamentals

This section covers the fundamental concepts of Grafana 8 alerting.

Create and manage rules

Sat, 04 Apr 2026 12:26:57 +0000

Create and manage Grafana alerting rules

An alerting rule is a set of evaluation criteria that determines whether an alert will fire. The rule consists of one or more queries and expressions, a condition, the frequency of evaluation, and optionally, the duration over which the condition is met.

While queries and expressions select the data set to evaluate, a condition sets the threshold that an alert must meet or exceed to create an alert. An interval specifies how frequently an alerting rule is evaluated. Duration, when configured, indicates how long a condition must be met. The rules can also define alerting behavior in the absence of data.

You can:

Alert groups

Sat, 04 Apr 2026 12:26:57 +0000

Alert groups

Alert groups show grouped alerts from an Alertmanager instance. By default, the alerts are grouped by the label keys for the root policy in notification policies. Grouping common alerts into a single alert group prevents duplicate alerts from being fired.

View alert groupings

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
Click Alert grouping to open the page listing existing groups.
From the Alertmanager drop-down, select an external Alertmanager as your data source. By default, the Grafana Alertmanager is selected.
From custom group by drop-down, select a combination of labels to view a grouping other than the default. This is useful for debugging and verifying your grouping of notification policies.

If an alert does not contain labels specified either in the grouping of the root policy or the custom grouping, then the alert is added to a catch all group with a header of No grouping.

Filter alerts

You can use the following filters to view alerts that match specific criteria:

Search by label: In Search, enter an existing label to view alerts matching the label. For example, environment=production,region=~US|EU,severity!=warning
Filter alerts by state: In States, select from Active, Suppressed, or Unprocessed states to view alerts matching your selected state. All other alerts are hidden.

Silences

Sat, 04 Apr 2026 12:26:57 +0000

Silences

Use silences to stop notifications from one or more alerting rules. Silences do not prevent alert rules from being evaluated. Nor do they not stop alerting instances from being shown in the user interface. Silences only stop notifications from getting created. A silence lasts for only a specified window of time.

Silences do not prevent alert rules from being evaluated. They also do not stop alert instances being shown in the user interface. Silences only prevent notifications from being created.

You can configure Grafana managed silences as well as silences for an external Alertmanager data source. For more information, see Alertmanager.

Note: Before Grafana v8.2, the configuration of the embedded Alertmanager was shared across organisations. Users of Grafana 8.0 and 8.1 are advised to use the new Grafana 8 Alerts only if they have one organisation. Otherwise, silences for the Grafana managed alerts will be visible by all organizations.

Add a silence

To add a silence:

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
In the Alerting page, click Silences to open the page listing existing contact points.
From Alertmanager drop-down, select an external Alertmanager to create and manage silences for the external data source. Otherwise, keep the default option of Grafana.
Click New Silence to open the Create silence page.
In Silence start and end, select the start and end date to indicate when the silence should go into effect and expire.
Optionally, in Duration, specify how long the silence is enforced. This automatically updates the end time in the Silence start and end field.
In the Name and Value fields, enter one or more Matching Labels. Matchers determine which rules the silence will apply to. For more information, see Label matching for alert suppression.
In Comment, add details about the silence.
In Creator, enter the name of the silence owner or keep the default owner.
Click Create.

Label matching for alert suppression

Grafana suppresses notifications only for alerts with labels that match all the “Matching Labels” specified in the silence.

The Label field is the name of the label to match. It must exactly match the label name.
The Operator field is the operator to match against the label value. The available operators are:
- =: Select labels that are exactly equal to the provided string.
- !=: Select labels that are not equal to the provided string.
- =~: Select labels that regex-match the provided string.
- !~: Select labels that do not regex-match the provided string.
The Value field matches against the corresponding value for the specified Label name. How it matches depends on the Operator value.

In the Alerting page, click Silences to view the list of existing silences.
Find the silence you want to edit, then click Edit (pen icon).
Make changes, then click Submit to save your changes.

Remove silences

In the Alerting page, click Silences to view the list of existing silences.
Find the silence you want to end, then click Unsilence.

Note: Silences that have ended are retained and listed for five days. You cannot remove a silence manually.

Create a URL to silence form with defaults filled in

When linking to a silence form, provide the default matching labels and comment via matchers and comment query parameters. The matchers parameter requires one more matching labels of the type [label][operator][value] joined by a comma while the operator parameter can be one of the following: = (equals, not regex), != (not equals, not regex), =~ (equals, regex), !~ (not equals, regex).

For example, to link to silence form with matching labels severity=critical & cluster!~europe-.* and comment Silence critical EU alerts, create a URL https://mygrafana/alerting/silence/new?matchers=severity%3Dcritical%2Ccluster!~europe-*&comment=Silence%20critical%20EU%20alert.

To link to a new silence page for an external Alertmanager, add a alertmanager query parameter with the Alertmanager data source name.

Contact points

Sat, 04 Apr 2026 12:26:57 +0000

Contact points

Use contact points to define how your contacts are notified when an alert fires. A contact point can have one or more contact point types, for example, email, slack, webhook, and so on. When an alert fires, a notification is sent to all contact point types listed for a contact point. Optionally, use message templates to customize notification messages for the contact point types.

You can configure Grafana managed contact points as well as contact points for an external Alertmanager data source. For more information, see Alertmanager.

Add a contact point

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
Click Contact points to open the page listing existing contact points.
Click New contact point.
From the Alertmanager dropdown, select an Alertmanager. By default, Grafana Alertmanager is selected.
In Name, enter a descriptive name for the contact point.
From Contact point type, select a type and fill out mandatory fields. For example, if you choose email, enter the email addresses. Or if you choose Slack, enter the Slack channel(s) and users who should be contacted.
Some contact point types, like email or webhook, have optional settings. In Optional settings, specify additional settings for the selected contact point type.
In Notification settings, optionally select Disable resolved message if you do not want to be notified when an alert resolves.
To add another contact point type, click New contact point type and repeat steps 6 through 8.
Click Save contact point to save your changes.

Edit a contact point

In the Alerting page, click Contact points to open the page listing existing contact points.
Find the contact point to edit, then click Edit (pen icon).
Make any changes and click Save contact point.

Test a contact point

For Grafana managed contact points, you can send a test notification which helps verify a contact point is configured correctly.

To send a test notification:

In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click Contact points.
Find the contact point to test, then click Edit (pen icon). You can also create a new contact point if needed.
Click Test (paper airplane icon) to open the contact point testing modal.
Choose whether to send a predefined test notification or choose custom to add your own custom annotations and labels to include in the notification.
Click Send test notification to fire the alert.

Delete a contact point

In the Alerting page, click Contact points to open the page listing existing contact points.
Find the contact point to delete, then click Delete (trash icon).
In the confirmation dialog, click Yes, delete.

Note: You cannot delete contact points that are in use by a notification policy. You will have to either delete the notification policy or update it to use another contact point.

Edit Alertmanager global config

To edit global configuration options for an external Alertmanager, like SMTP server, that is used by default for all email contact types:

In the Alerting page, click Contact points to open the page listing existing contact points.
From the Alertmanager drop-down, select an external Alertmanager data source.
Click the Edit global config option.
Add global configuration settings.
Click Save global config to save your changes.

Note This option is available only for external Alertmanagers. You can configure some global options for Grafana contact types, like email settings, via Grafana configuration.

List of notifiers supported by Grafana

Name	Type	Grafana Alertmanager	Other Alertmanagers
DingDing	`dingding`	Supported	N/A
Discord	`discord`	Supported	N/A
Email	`email`	Supported	Supported
Google Hangouts Chat	`googlechat`	Supported	N/A
Kafka	`kafka`	Supported	N/A
Line	`line`	Supported	N/A
Microsoft Teams	`teams`	Supported	N/A
Opsgenie	`opsgenie`	Supported	Supported
Pagerduty	`pagerduty`	Supported	Supported
Prometheus Alertmanager	`prometheus-alertmanager`	Supported	N/A
Pushover	`pushover`	Supported	Supported
Sensu	`sensu`	Supported	N/A
Sensu Go	`sensugo`	Supported	N/A
Slack	`slack`	Supported	Supported
Telegram	`telegram`	Supported	N/A
Threema	`threema`	Supported	N/A
VictorOps	`victorops`	Supported	Supported
Webhook	`webhook`	Supported	Supported (different format)
WeCom	`wecom`	Supported	N/A
Zenduty	`webhook`	Supported	N/A

Webhook

Example JSON body:

{
  "receiver": "My Super Webhook",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High memory usage",
        "team": "blue",
        "zone": "us-1"
      },
      "annotations": {
        "description": "The system has high memory usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone us-1"
      },
      "startsAt": "2021-10-12T09:51:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
      "fingerprint": "c6eadffa33fcdf37",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1",
      "dashboardURL": "",
      "panelURL": "",
      "valueString": "[ metric='' labels={} value=14151.331895396988 ]"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage",
        "team": "blue",
        "zone": "eu-1"
      },
      "annotations": {
        "description": "The system has high CPU usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone eu-1"
      },
      "startsAt": "2021-10-12T09:56:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/d1rdpdv7k/edit",
      "fingerprint": "bc97ff14869b13e3",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1",
      "dashboardURL": "",
      "panelURL": "",
      "valueString": "[ metric='' labels={} value=47043.702386305304 ]"
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "team": "blue"
  },
  "commonAnnotations": {},
  "externalURL": "https://play.grafana.org/",
  "version": "1",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0,
  "title": "[FIRING:2]  (blue)",
  "state": "alerting",
  "message": "**Firing**\n\nLabels:\n - alertname = T2\n - team = blue\n - zone = us-1\nAnnotations:\n - description = This is the alert rule checking the second system\n - runbook_url = https://myrunbook.com\n - summary = This is my summary\nSource: https://play.grafana.org/alerting/1afz29v7z/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1\n\nLabels:\n - alertname = T1\n - team = blue\n - zone = eu-1\nAnnotations:\nSource: https://play.grafana.org/alerting/d1rdpdv7k/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1\n"
}

Webhook fields

Body

Key	Type	Description
receiver	string	Name of the webhook
status	string	Current status of the alert, `firing` or `resolved`
orgId	number	ID of the organization related to the payload
alerts	array of alerts	Alerts that are triggering
groupLabels	object	Labels that are used for grouping, map of string keys to string values
commonLabels	object	Labels that all alarms have in common, map of string keys to string values
commonAnnotations	object	Annotations that all alarms have in common, map of string keys to string values
externalURL	string	External URL to the Grafana instance sending this webhook
version	string	Version of the payload
groupKey	string	Key that is used for grouping
truncatedAlerts	number	Number of alerts that were truncated
title	string	Will be deprecated soon
state	string	Will be deprecated soon
message	string	Will be deprecated soon

Alert

Key	Type	Description
status	string	Current status of the alert, `firing` or `resolved`
labels	object	Labels that are part of this alert, map of string keys to string values
annotations	object	Annotations that are part of this alert, map of string keys to string values
startsAt	string	Start time of the alert
endsAt	string	End time of the alert, default value when not resolved is `0001-01-01T00:00:00Z`
valueString	string	Values that triggered the current status
generatorURL	string	URL of the alert rule in the Grafana UI
fingerprint	string	The labels fingerprint, alarms with the same labels will have the same fingerprint
silenceURL	string	URL to silence the alert rule in the Grafana UI
dashboardURL	string	Will be deprecated soon
panelURL	string	Will be deprecated soon

Alerts are not coupled to dashboards anymore therefore the fields related to dashboards dashboardId and panelId have been removed.

WeCom

WeCom contact points need a Webhook URL. These are obtained by setting up a WeCom robot on the corresponding group chat. To obtain a Webhook URL using the WeCom desktop Client please follow these steps:

Click the “…” in the top right corner of a group chat that you want your alerts to be delivered to
Click “Add Group Robot”, select “New Robot” and give your robot a name. Click “Add Robot”
There should be a Webhook URL in the panel.

Setting	Description
Url	WeCom webhook URL.

Message templating

Sat, 04 Apr 2026 12:26:57 +0000

Message templating

Notifications sent via contact points are built using messaging templates. Grafana’s default templates are based on the Go templating system where some fields are evaluated as text, while others are evaluated as HTML (which can affect escaping). The default template, defined in default_template.go, is a useful reference for custom templates.

Since most of the contact point fields can be templated, you can create reusable custom templates and use them in multiple contact points. The template data topic lists variables that are available for templating. The default template is defined in default_template.go which can serve as a useful reference or starting point for custom templates.

Using templates

The following example shows how to use default templates to render an alert message in Slack. The message title contains a count of alerts that are firing or were resolved. The message body lists the alerts and their status.

The following example shows the use of a custom template within one of the contact point fields.

Create a message template

Note: Before Grafana v8.2, the configuration of the embedded Alertmanager was shared across organisations. Users of Grafana 8.0 and 8.1 are advised to use the new Grafana 8 alerts only if they have one organisation. Otherwise, silences for the Grafana managed alerts will be visible by all organizations.

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
In the Alerting page, click Contact points to open the page listing existing contact points.
From Alertmanager drop-down, select an external Alertmanager to create and manage templates for the external data source. Otherwise, keep the default option of Grafana.

Select Alertmanager
Click Add template.
In Name, add a descriptive name.
In Content, add the content of the template.
Click Save template button at the bottom of the page.

The define tag in the Content section assigns the template name. This tag is optional, and when omitted, the template name is derived from the Name field. When both are specified, it is a best practice to ensure that they are the same.

Edit a message template

In the Alerting page, click Contact points to open the page listing existing contact points.
In the Template table, find the template you want to edit, then click the Edit (pen icon).
Make your changes, then click Save template.

Delete a message template

In the Alerting page, click Contact points to open the page listing existing contact points.
In the Template table, find the template you want to delete, then click the Delete (trash icon).
In the confirmation dialog, click Yes, delete to delete the template.

Use caution when deleting a template since Grafana does not prevent you from deleting templates that are in use.

Nested templates

You can embed templates within other templates.

For example, you can define a template fragment using the define keyword:

{{ define "mytemplate" }}
  {{ len .Alerts.Firing }} firing. {{ len .Alerts.Resolved }} resolved.
{{ end }}

You can then embed custom templates within this fragment using the template keyword. For example:

Alert summary:
{{ template "mytemplate" . }}

You can use any of the following built-in template options to embed custom templates.

Name	Notes
`default.title`	Displays high-level status information.
`default.message`	Provides a formatted summary of firing and resolved alerts.
`teams.default.message`	Similar to `default.messsage`, formatted for Microsoft Teams.

Custom template examples

Here are a few examples of how to use custom templates.

Template to render a single alert:

{{ define "myalert" }}
  [{{.Status}}] {{ .Labels.alertname }}

  Labels:
  {{ range .Labels.SortedPairs }}
    {{ .Name }}: {{ .Value }}
  {{ end }}

  {{ if gt (len .Annotations) 0 }}
  Annotations:
  {{ range .Annotations.SortedPairs }}
    {{ .Name }}: {{ .Value }}
  {{ end }}
  {{ end }}

  {{ if gt (len .SilenceURL ) 0 }}
    Silence alert: {{ .SilenceURL }}
  {{ end }}
  {{ if gt (len .DashboardURL ) 0 }}
    Go to dashboard: {{ .DashboardURL }}
  {{ end }}
{{ end }}

Template to render entire notification message:

{{ define "mymessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ len .Alerts.Firing }} firing:
    {{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ len .Alerts.Resolved }} resolved:
    {{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}
  {{ end }}
{{ end }}

HTML in Message Templates

HTML in alerting message templates is escaped. We do not support rendering of HTML in the resulting notification.

Some notifiers support alternative methods of changing the look and feel of the resulting notification. For example, Grafana installs the base template for alerting emails to <grafana-install-dir>/public/emails/ng_alert_notification.html. You can edit this file to change the appearance of all alerting emails.

High availability

Sat, 04 Apr 2026 12:26:57 +0000

High availability

The Grafana alerting system has two main components: a Scheduler and an internal Alertmanager. The Scheduler is responsible for the evaluation of your alert rules while the internal Alertmanager takes care of the routing and grouping.

When it comes to running Grafana alerting in high availability the operational mode of the scheduler is unaffected such that all alerts continue be evaluated in each Grafana instance. Rather the operational change happens in the Alertmanager which deduplicates alert notifications across Grafana instances.

The coordination between Grafana instances happens via a Gossip protocol. Alerts are not gossiped between instances. It is expected that each scheduler delivers the same alerts to each Alertmanager.

The two types of messages that are gossiped between instances are:

Notification logs: Who (which instance) notified what (which alert)
Silences: If an alert should fire or not

These two states are persisted in the database periodically and when Grafana is gracefully shutdown.

Enable high availability

To enable high availability support you need to add at least 1 Grafana instance to the [ha_peer] configuration option within the [unified_alerting] section:

In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set [ha_peers] to the number of hosts for each grafana instance in the cluster (using a format of host:port) e.g. ha_peers=10.0.0.5:9094,10.0.0.6:9094,10.0.0.7:9094
Gossiping of notifications and silences uses both TCP and UDP port 9094. Each Grafana instance will need to be able to accept incoming connections on these ports.
Set [ha_listen_address] to the instance IP address using a format of host:port (or the Pod’s IP in the case of using Kubernetes) by default it is set to listen to all interfaces (0.0.0.0).

Kubernetes

If you are using Kubernetes, you can expose the pod IP through an environment variable via the container definition such as:

env:
- name: POD_IP
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

Notification policies

Sat, 04 Apr 2026 12:26:57 +0000

Notification policies

Notification policies determine how alerts are routed to contact points. Policies have a tree structure, where each policy can have one or more child policies. Each policy, except for the root policy, can also match specific alert labels. Each alert is evaluated by the root policy and subsequently by each child policy. If you enable the Continue matching subsequent sibling nodes option is enabled for a specific policy, then evaluation continues even after one or more matches. A parent policy’s configuration settings and contact point information govern the behavior of an alert that does not match any of the child policies. A root policy governs any alert that does not match a specific policy.

You can configure Grafana managed notification policies as well as notification policies for an external Alertmanager data source. For more information, see Alertmanager.

Grouping

Grouping is a new and key concept of Grafana alerting that categorizes alert notifications of similar nature into a single funnel. This allows you to properly route alert notifications during larger outages when many parts of a system fail at once causing a high number of alerts to fire simultaneously.

For example, suppose you have 100 services connected to a database in different environments. These services are differentiated by the label env=environmentname. An alert rule is in place to monitor whether your services can reach the database named alertname=DatabaseUnreachable.

When a network partition occurs, half of your services can no longer reach the database. As a result, 50 different alerts (assuming half of your services) are fired. For this situation, you want to receive a single-page notification (as opposed to 50) with a list of the environments that are affected.

You can configure grouping to be group_by: [alertname] (take note that the env label is omitted). With this configuration in place, Grafana sends a single compact notification that has all the affected environments for this alert rule.

Note: Grafana also has a special label named ... that you can use to group all alerts by all labels (effectively disabling grouping), therefore each alert will go into its own group. It is different from the default of group_by: null where all alerts go into a single group.

Edit root notification policy

Note: Before Grafana v8.2, the configuration of the embedded Alertmanager was shared across organisations. Users of Grafana 8.0 and 8.1 are advised to use the new Grafana 8 Alerts only if they have one organisation. Otherwise, silences for the Grafana managed alerts will be visible by all organizations.

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
Click Notification policies.
From the Alertmanager dropdown, select an external Alertmanager. By default, the Grafana Alertmanager is selected.
In the Root policy section, click Edit (pen icon).
In Default contact point, update the contact point to whom notifications should be sent for rules when alert rules do not match any specific policy.
In Group by, choose labels to group alerts by. If multiple alerts are matched for this policy, then they are grouped by these labels. A notification is sent per group. If the field is empty (default), then all notifications are sent in a single group. Use a special label ... to group alerts by all labels (which effectively disables grouping).
In Timing options, select from the following options:
- Group wait Time to wait to buffer alerts of the same group before sending an initial notification. Default is 30 seconds.
- Group interval Minimum time interval between two notifications for a group. Default is 5 minutes.
- Repeat interval Minimum time interval for re-sending a notification if no new alerts were added to the group. Default is 4 hours.
Click Save to save your changes.

Add new specific policy

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
Click Notification policies.
From the Alertmanager dropdown, select an Alertmanager. By default, the Grafana Alertmanager is selected.
To add a top level specific policy, go to the Specific routing section and click New specific policy.
In Matching labels section, add one or more rules for matching alert labels. For more information, see “How label matching works”.
In Contact point, add the contact point to send notification to if alert matches only this specific policy and not any of the nested policies.
Optionally, enable Continue matching subsequent sibling nodes to continue matching nested policies even after the alert matched the parent policy. When this option is enabled, you can get more than one notification. Use it to send notification to a catch-all contact point as well as to one of more specific contact points handled by nested policies.
Optionally, enable Override grouping to specify the same grouping as the root policy. If this option is not enabled, the root policy grouping is used.
Optionally, enable Override general timings to override the timing options configured in the group notification policy.
Click Save policy to save your changes.

Add nested policy

Expand the specific policy you want to update.
Click Add nested policy, then add the details using information in Add new specific policy.
Click Save policy to save your changes.

Edit specific policy

In the Alerting page, click Notification policies to open the page listing existing policies.
Find the policy you want to edit, then click Edit (pen icon).
Make any changes using instructions in Add new specific policy.
Click Save policy.

How label matching works

A policy will match an alert if the alert’s labels match all the “Matching Labels” specified on the policy.

The Label field is the name of the label to match. It must exactly match the label name.
The Operator field is the operator to match against the label value. The available operators are:
- =: Select labels that are exactly equal to the provided string.
- !=: Select labels that are not equal to the provided string.
- =~: Select labels that regex-match the provided string.
- !~: Select labels that do not regex-match the provided string.
The Value field matches against the corresponding value for the specified Label name. How it matches depends on the Operator value.

Example

An example of an alert configuration.

Create a “default” contact point for slack notifications, and set it on root policy.
Edit the root policy grouping to group alerts by cluster, namespace and severity so that you get a notification per alert rule and specific kubernetes cluster and namespace.
Create specific route for alerts coming from the development cluster with an appropriate contact point.
Create a specific route for alerts with “critical” severity with a more invasive contact point type, like pager duty notification.
Create specific routes for particular teams that handle their own onduty rotations.

Grafana alerts on Grafana Labs

What's new in Grafana alerting

What’s new in Grafana alerting

Multi-dimensional alerting

Create alerts outside of Dashboards

Create Loki and Cortex alerting rules

View and search for alerts from Prometheus compatible data sources

Special alerts for alert state NoData and Error

Opt-in to Grafana alerting

Opt-in to Grafana alerting

Enable Grafana alerting

Migrating legacy alerts to Grafana alerting system

Limitation

Disable Grafana alerts

Alerting fundamentals

Alerting fundamentals

Create and manage rules

Create and manage Grafana alerting rules

Alert groups

Alert groups

View alert groupings

Filter alerts

Silences

Silences

Add a silence

Label matching for alert suppression

Remove silences

Create a URL to silence form with defaults filled in

Contact points

Contact points

Add a contact point

Edit a contact point

Test a contact point

Delete a contact point

Edit Alertmanager global config

List of notifiers supported by Grafana

Webhook

Webhook fields

Body

Alert

Removed fields related to dashboards

WeCom

Message templating

Message templating

Using templates

Create a message template

Edit a message template

Delete a message template

Nested templates

Custom template examples

HTML in Message Templates

High availability

High availability

Enable high availability

Kubernetes

Notification policies

Notification policies

Grouping

Edit root notification policy

Add new specific policy

Add nested policy

Edit specific policy

How label matching works

Example