Legacy Grafana Alerts on Grafana Labs

Alert notifications

Sat, 04 Apr 2026 12:26:57 +0000

Alert notifications

When an alert changes state, it sends out notifications. Each alert rule can have multiple notifications. In order to add a notification to an alert rule you first need to add and configure a notification channel (can be email, PagerDuty, or other integration).

This is done from the Notification channels page.

Note: Alerting is only available in Grafana v4.0 and above.

Add a notification channel

In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click Notification channels.
Click Add channel.
Fill out the fields or select options described below.

New notification channel fields

Default (send on all alerts)

Name - Enter a name for this channel. It will be displayed when users add notifications to alert rules.
Type - Select the channel type. Refer to the List of supported notifiers for details.
Default (send on all alerts) - When selected, this option sends a notification on this channel for all alert rules.
Include Image - See Enable images in notifications for details.
Disable Resolve Message - When selected, this option disables the resolve message [OK] that is sent when the alerting state returns to false.
Send reminders - When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders should be sent using number of seconds (s), minutes (m) or hours (h), for example 30s, 3m, 5m or 1h.

Important: Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured alert rule evaluation interval.

These examples show how often and when reminders are sent for a triggered alert.

Alert rule evaluation interval	Send reminders every	Reminder sent every (after last alert notification)
`30s`	`15s`	~30 seconds
`1m`	`5m`	~5 minutes
`5m`	`15m`	~15 minutes
`6m`	`20m`	~24 minutes
`1h`	`15m`	~1 hour
`1h`	`2h`	~2 hours

List of supported notifiers

Name	Type	Supports images	Supports alert rule tags
DingDing	`dingding`	yes, external only	no
Discord	`discord`	yes	no
Email	`email`	yes	no
Google Hangouts Chat	`googlechat`	yes, external only	no
Hipchat	`hipchat`	yes, external only	no
Kafka	`kafka`	yes, external only	no
Line	`line`	yes, external only	no
Microsoft Teams	`teams`	yes, external only	no
Opsgenie	`opsgenie`	yes, external only	yes
Pagerduty	`pagerduty`	yes, external only	yes
Prometheus Alertmanager	`prometheus-alertmanager`	yes, external only	yes
Pushover	`pushover`	yes	no
Sensu	`sensu`	yes, external only	no
Sensu Go	`sensugo`	yes, external only	no
Slack	`slack`	yes	no
Telegram	`telegram`	yes	no
Threema	`threema`	yes, external only	no
VictorOps	`victorops`	yes, external only	yes
Webhook	`webhook`	yes, external only	yes

Email

To enable email notifications you have to set up SMTP settings in the Grafana config. Email notifications will upload an image of the alert graph to an external image destination if available or fallback to attaching the image to the email. Be aware that if you use the local image storage email servers and clients might not be able to access the image.

Note: Template variables are not supported in email alerts.

Setting	Description
Single email	Send a single email to all recipients. Disabled per default.
Addresses	Email addresses to recipients. You can enter multiple email addresses using a “;” separator.

Slack

To set up Slack, you need to configure an incoming Slack webhook URL. You can follow Sending messages using Incoming Webhooks on how to do that. If you want to include screenshots of the firing alerts in the Slack messages you have to configure either the external image destination in Grafana or a bot integration via Slack Apps. Follow Slack’s guide to set up a bot integration and use the token provided, which starts with “xoxb”.

Setting	Description
Url	Slack incoming webhook URL, or eventually the chat.postMessage Slack API endpoint.
Username	Set the username for the bot’s message.
Recipient	Allows you to override the Slack recipient. You must either provide a channel Slack ID, a user Slack ID, a username reference (@<user>, all lowercase, no whitespace), or a channel reference (#<channel>, all lowercase, no whitespace). If you use the `chat.postMessage` Slack API endpoint, this is required.
Icon emoji	Provide an emoji to use as the icon for the bot’s message. Ex :smile:
Icon URL	Provide a URL to an image to use as the icon for the bot’s message.
Mention Users	Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the overflow button on each user’s Slack profile).
Mention Groups	Optionally mention one or more groups in the Slack notification sent by Grafana. You have to refer to groups, comma-separated, via their corresponding Slack IDs (which you can get from each group’s Slack profile URL).
Mention Channel	Optionally mention either all channel members or just active ones.
Token	If provided, Grafana will upload the generated image via Slack’s file.upload API method, not the external image destination. If you use the `chat.postMessage` Slack API endpoint, this is required.

If you are using the token for a slack bot, then you have to invite the bot to the channel you want to send notifications and add the channel to the recipient field.

Opsgenie

To setup Opsgenie you will need an API Key and the Alert API Url. These can be obtained by configuring a new Grafana Integration.

Setting	Description
Alert API URL	The API URL for your Opsgenie instance. This will normally be either `https://api.opsgenie.com` or, for EU customers, `https://api.eu.opsgenie.com`.
API Key	The API Key as provided by Opsgenie for your configured Grafana integration.
Override priority	Configures the alert priority using the `og_priority` tag. The `og_priority` tag must have one of the following values: `P1`, `P2`, `P3`, `P4`, or `P5`. Default is `False`.
Send notification tags as	Specify how you would like Notification Tags delivered to Opsgenie. They can be delivered as `Tags`, `Extra Properties` or both. Default is Tags. See note below for more information.

Note: When notification tags are sent as Tags they are concatenated into a string with a key:value format. If you prefer to receive the notifications tags as key/values under Extra Properties in Opsgenie then change the Send notification tags as to either Extra Properties or Tags & Extra Properties.

PagerDuty

To set up PagerDuty, all you have to do is to provide an integration key.

Setting	Description
Integration Key	Integration key for PagerDuty.
Severity	Level for dynamic notifications, default is `critical` (1)
Auto resolve incidents	Resolve incidents in PagerDuty once the alert goes back to ok
Message in details	Removes the Alert message from the PD summary field and puts it into custom details instead (2)

Note: The tags Severity, Class, Group, dedup_key, and Component have special meaning in the Pagerduty Common Event Format - PD-CEF. If an alert panel defines these tag keys, then they are transposed to the root of the event sent to Pagerduty. This means they will be available within the Pagerduty UI and Filtering tools. A Severity tag set on an alert overrides the global Severity set on the notification channel if it’s a valid level.

Using Message In Details will change the structure of the custom_details field in the PagerDuty Event. This might break custom event rules in your PagerDuty rules if you rely on the fields in payload.custom_details. Move any existing rules using custom_details.myMetric to custom_details.queries.myMetric. This behavior will become the default in a future version of Grafana.

Note: The dedup_key tag overrides the Grafana-generated dedup_key with a custom key.

Note: The state tag overrides the current alert state inside the custom_details payload.

Note: Grafana uses the Events API V2 integration. This can be configured for each service.

VictorOps

To configure VictorOps, provide the URL from the Grafana Integration and substitute $routing_key with a valid key.

Note: The tag Severity has special meaning in the VictorOps Incident Fields. If an alert panel defines this key, then it replaces the message_type in the root of the event sent to VictorOps.

Pushover

To set up Pushover, you must provide a user key and an API token. Refer to What is Pushover and how do I use it for instructions on how to generate them.

Setting	Description
API Token	Application token
User key(s)	A comma-separated list of user keys
Device(s)	A comma-separated list of devices
Priority	The priority alerting nottifications are sent
OK priority	The priority OK notifications are sent; if not set, then OK notifications are sent with the priority set for alerting notifications
Retry	How often (in seconds) the Pushover servers send the same notification to the user. (minimum 30 seconds)
Expire	How many seconds your notification will continue to be retried for (maximum 86400 seconds)
Alerting sound	The sound for alerting notifications
OK sound	The sound for OK notifications

Webhook

The webhook notification is a simple way to send information about a state change over HTTP to a custom endpoint. Using this notification you could integrate Grafana into a system of your choosing.

Example json body:

{
  "dashboardId": 1,
  "evalMatches": [
    {
      "value": 1,
      "metric": "Count",
      "tags": {}
    }
  ],
  "imageUrl": "https://grafana.com/assets/img/blog/mixed_styles.png",
  "message": "Notification Message",
  "orgId": 1,
  "panelId": 2,
  "ruleId": 1,
  "ruleName": "Panel Title alert",
  "ruleUrl": "http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
  "state": "alerting",
  "tags": {
    "tag name": "tag value"
  },
  "title": "[Alerting] Panel Title alert"
}

state - The possible values for alert state are: ok, paused, alerting, pending, no_data.

DingDing/DingTalk

DingTalk supports the following “message type”: text, link and markdown. Only the link message type is supported. Refer to the configuration instructions in Chinese language.

In DingTalk PC Client:

Click “more” icon on upper right of the panel.
Click “Robot Manage” item in the pop menu, there will be a new panel call “Robot Manage”.
In the “Robot Manage” panel, select “customized: customized robot with Webhook”.
In the next new panel named “robot detail”, click “Add” button.
In “Add Robot” panel, input a nickname for the robot and select a “message group” which the robot will join in. click “next”.
There will be a Webhook URL in the panel, looks like this: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx. Copy this URL to the Grafana DingTalk setting page and then click “finish”.

Discord

To set up Discord, you must create a Discord channel webhook. For instructions on how to create the channel, refer to Intro to Webhooks.

Setting	Description
Webhook URL	Discord webhook URL.
Message Content	Mention a group using @ or a user using <@ID> when notifying in a channel.
Avatar URL	Optionally, provide a URL to an image to use as the avatar for the bot’s message.
Use Discord’s Webhook Username	Use the username configured in Discord’s webhook settings. Otherwise, the username will be ‘Grafana.’

Alternately, use the Slack notifier by appending /slack to a Discord webhook URL.

Kafka

Notifications can be sent to a Kafka topic from Grafana using the Kafka REST Proxy. There are a couple of configuration options which need to be set up in Grafana UI under Kafka Settings:

Kafka REST Proxy endpoint.
Kafka Topic.

Once these two properties are set, you can send the alerts to Kafka for further processing or throttling.

Google Hangouts Chat

Notifications can be sent by setting up an incoming webhook in Google Hangouts chat. For more information about configuring a webhook, refer to webhooks.

Prometheus Alertmanager

Alertmanager handles alerts sent by client applications such as Prometheus server or Grafana. It takes care of deduplicating, grouping, and routing them to the correct receiver. Grafana notifications can be sent to Alertmanager via a simple incoming webhook. Refer to the official Prometheus Alertmanager documentation for configuration information.

Caution: In case of a high-availability setup, do not load balance traffic between Grafana and Alertmanagers to keep coherence between all your Alertmanager instances. Instead, point Grafana to a list of all Alertmanagers, by listing their URLs comma-separated in the notification channel configuration.

Sensu Go

Grafana alert notifications can be sent to Sensu Go as events via the API. This operation requires an API key. For information on creating this key, refer to Sensu Go documentation.

Enable images in notifications

Grafana can render the panel associated with the alert rule as a PNG image and include that in the notification. Read more about the requirements and how to configure image rendering.

You must configure an external image storage provider in order to receive images in alert notifications. If your notification channel requires that the image be publicly accessible (e.g. Slack, PagerDuty), configure a provider which uploads the image to a remote image store like Amazon S3, Webdav, Google Cloud Storage, or Azure Blob Storage. Otherwise, the local provider can be used to serve the image directly from Grafana.

Notification services which need public image access are marked as ’external only'.

Configure the link back to Grafana from alert notifications

All alert notifications contain a link back to the triggered alert in the Grafana instance. This URL is based on the domain setting in Grafana.

Notification templating

Note: Alert notification templating is only available in Grafana v7.4 and above.

The alert notification template feature allows you to take the label value from an alert query and inject that into alert notifications.

Alert notification templating

Sat, 04 Apr 2026 12:26:57 +0000

Alert notification templating

You can provide detailed information to alert notification recipients by injecting alert query data into an alert notification. This topic explains how you can use alert query labels in alert notifications.

Labels that exist from the evaluation of the alert query can be used in the alert rule name and in the alert notification message fields. The alert label data is injected into the notification fields when the alert is in the alerting state. When there are multiple unique values for the same label, the values are comma-separated.

This topic explains how you can use alert query labels in alert notifications.

Adding alert label data into your alert notification

Navigate to the panel you want to add or edit an alert rule for.
Click on the panel title, and then click Edit.
On the Alert tab, click Create Alert. If an alert already exists for this panel, then you can edit the alert directly.
Refer to the alert query labels in the alert rule name and/or alert notification message field by using the ${Label} syntax.
Click Save in the upper right corner to save the alert rule and the dashboard.

Create alerts

Sat, 04 Apr 2026 12:26:57 +0000

Create alerts

Grafana alerting allows you to attach rules to your dashboard panels. When you save the dashboard, Grafana extracts the alert rules into a separate alert rule storage and schedules them for evaluation.

In the Alert tab of the graph panel you can configure how often the alert rule should be evaluated and the conditions that need to be met for the alert to change state and trigger its notifications.

Currently only the graph panel supports alert rules.

Add or edit an alert rule

Navigate to the panel you want to add or edit an alert rule for, click the title, and then click Edit.
On the Alert tab, click Create Alert. If an alert already exists for this panel, then you can just edit the fields on the Alert tab.
Fill out the fields. Descriptions are listed below in Alert rule fields.
When you have finished writing your rule, click Save in the upper right corner to save alert rule and the dashboard.
(Optional but recommended) Click Test rule to make sure the rule returns the results you expect.

Delete an alert

To delete an alert, scroll to the bottom of the alert and then click Delete.

Alert rule fields

This section describes the fields you fill out to create an alert.

Rule

Name - Enter a descriptive name. The name will be displayed in the Alert Rules list. This field supports templating.
Evaluate every - Specify how often the scheduler should evaluate the alert rule. This is referred to as the evaluation interval.
For - Specify how long the query needs to violate the configured thresholds before the alert notification triggers.

You can set a minimum evaluation interval in the alerting.min_interval_seconds configuration field, to set a minimum time between evaluations. Refer to Configuration for more information.

Caution: Do not use For with the If no data or all values are null setting set to No Data. The triggering of No Data will trigger instantly and not take For into consideration. This may also result in that an OK notification not being sent if alert transitions from No Data -> Pending -> OK.

If an alert rule has a configured For and the query violates the configured threshold, then it will first go from OK to Pending. Going from OK to Pending Grafana will not send any notifications. Once the alert rule has been firing for more than For duration, it will change to Alerting and send alert notifications.

Typically, it’s always a good idea to use this setting since it’s often worse to get false positive than wait a few minutes before the alert notification triggers. Looking at the Alert list or Alert list panels you will be able to see alerts in pending state.

Below you can see an example timeline of an alert using the For setting. At ~16:04 the alert state changes to Pending and after 4 minutes it changes to Alerting which is when alert notifications are sent. Once the series falls back to normal the alert rule goes back to OK.

Conditions

Currently the only condition type that exists is a Query condition that allows you to specify a query letter, time range and an aggregation function.

Query condition example

avg() OF query(A, 15m, now) IS BELOW 14

avg() Controls how the values for each series should be reduced to a value that can be compared against the threshold. Click on the function to change it to another aggregation function.
query(A, 15m, now) The letter defines what query to execute from the Metrics tab. The second two parameters define the time range, 15m, now means 15 minutes ago to now. You can also do 10m, now-2m to define a time range that will be 10 minutes ago to 2 minutes ago. This is useful if you want to ignore the last 2 minutes of data.
IS BELOW 14 Defines the type of threshold and the threshold value. You can click on IS BELOW to change the type of threshold.

The query used in an alert rule cannot contain any template variables. Currently we only support AND and OR operators between conditions and they are executed serially. For example, we have 3 conditions in the following order: condition:A(evaluates to: TRUE) OR condition:B(evaluates to: FALSE) AND condition:C(evaluates to: TRUE) so the result will be calculated as ((TRUE OR FALSE) AND TRUE) = TRUE.

We plan to add other condition types in the future, like Other Alert, where you can include the state of another alert in your conditions, and Time Of Day.

Multiple Series

If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series. What Grafana does not do currently is track alert rule state per series. This has implications that are detailed in the scenario below.

Alert condition with query that returns 2 series: server1 and server2
server1 series causes the alert rule to fire and switch to state Alerting
Notifications are sent out with message: load peaking (server1)
In a subsequent evaluation of the same alert rule, the server2 series also causes the alert rule to fire
No new notifications are sent as the alert rule is already in state Alerting.

So, as you can see from the above scenario Grafana will not send out notifications when other series cause the alert to fire if the rule already is in state Alerting. To improve support for queries that return multiple series we plan to track state per series in a future release.

Starting with Grafana v5.3 you can configure reminders to be sent for triggered alerts. This will send additional notifications when an alert continues to fire. If other series (like server2 in the example above) also cause the alert rule to fire they will be included in the reminder notification. Depending on what notification channel you’re using you may be able to take advantage of this feature for identifying new/existing series causing alert to fire.

No Data & Error Handling

Below are conditions you can configure how the rule evaluation engine should handle queries that return no data or only null values.

No Data Option	Description
No Data	Set alert rule state to `NoData`
Alerting	Set alert rule state to `Alerting`
Keep Last State	Keep the current alert rule state, whatever it is.
Ok	Not sure why you would want to send yourself an alert when things are okay, but you could.

Execution errors or timeouts

Tell Grafana how to handle execution or timeout errors.

Error or timeout option	Description
Alerting	Set alert rule state to `Alerting`
Keep Last State	Keep the current alert rule state, whatever it is.

If you have an unreliable time series store from which queries sometime timeout or fail randomly you can set this option to Keep Last State in order to basically ignore them.

Notifications

In alert tab you can also specify alert rule notifications along with a detailed message about the alert rule. The message can contain anything, information about how you might solve the issue, link to runbook, and so on.

The actual notifications are configured and shared between multiple alerts. Read Alert notifications for information on how to configure and set up notifications.

Send to - Select an alert notification channel if you have one set up.
Message - Enter a text message to be sent on the notification channel. Some alert notifiers support transforming the text to HTML or other rich formats. This field supports templating.
Tags - Specify a list of tags (key/value) to be included in the notification. It is only supported by some notifiers.

Alert state history and annotations

Alert state changes are recorded in the internal annotation table in Grafana’s database. The state changes are visualized as annotations in the alert rule’s graph panel. You can also go into the State history submenu in the alert tab to view and clear state history.

Pause alert rule

Sat, 04 Apr 2026 12:26:57 +0000

Pause an alert rule

Pausing the evaluation of an alert rule can sometimes be useful. For example, during a maintenance window, pausing alert rules can avoid triggering a flood of alerts.

In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click Alert Rules. All configured alert rules are listed, along with their current state.
Find your alert in the list, and click the Pause icon on the right. The Pause icon turns into a Play icon.
Click the Play icon to resume evaluation of your alert.

View alerts

Sat, 04 Apr 2026 12:26:57 +0000

View existing alert rules

Grafana stores individual alert rules in the panels where they are defined, but you can also view a list of all existing alert rules and their current state.

In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click Alert Rules. All configured alert rules are listed, along with their current state.

You can do several things while viewing alerts.

Filter alerts by name - Type an alert name in the Search alerts field.
Filter alerts by state - In States, select which alert states you want to see. All others will be hidden.
Pause or resume an alert - Click the Pause or Play icon next to the alert to pause or resume evaluation. See Pause an alert rule for more information.
Access alert rule settings - Click the alert name or the Edit alert rule (gear) icon. Grafana opens the Alert tab of the panel where the alert rule is defined. This is helpful when an alert is firing but you don’t know which panel it is defined in.

Troubleshoot alerts

Sat, 04 Apr 2026 12:26:57 +0000

Troubleshoot alerts

If alerts are not behaving as you expect, here are some steps you can take to troubleshoot and figure out what is going wrong.

The first level of troubleshooting you can do is click Test Rule. You will get result back that you can expand to the point where you can see the raw data that was returned from your query.

Further troubleshooting can also be done by inspecting the grafana-server log. If it’s not an error or for some reason the log does not say anything you can enable debug logging for some relevant components. This is done in Grafana’s ini config file.

Example showing loggers that could be relevant when troubleshooting alerting.

[log]
filters = alerting.scheduler:debug \
          alerting.engine:debug \
          alerting.resultHandler:debug \
          alerting.evalHandler:debug \
          alerting.evalContext:debug \
          alerting.extractor:debug \
          alerting.notifier:debug \
          alerting.notifier.slack:debug \
          alerting.notifier.pagerduty:debug \
          alerting.notifier.email:debug \
          alerting.notifier.webhook:debug \
          tsdb.graphite:debug \
          tsdb.prometheus:debug \
          tsdb.opentsdb:debug \
          tsdb.influxdb:debug \
          tsdb.elasticsearch:debug \
          tsdb.elasticsearch.client:debug \

If you want to log raw query sent to your TSDB and raw response in log you also have to set grafana.ini option app_mode to development.