Alerting fundamentals on Grafana Labs

Alerting on numeric data

Sat, 04 Apr 2026 12:26:57 +0000

Alerting on numeric data

This topic describes how Grafana managed alerts are evaluated by the backend engine as well as how Grafana handles alerting on numeric rather than time series data.

Alert evaluation

Grafana managed alerts query the following backend data sources that have alerting enabled:

built-in data sources or those developed and maintained by Grafana: Graphite, Prometheus, Loki, InfluxDB, Elasticsearch, Google Cloud Monitoring, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, Oracle, and Azure Monitor
community developed backend data sources with alerting enabled (backend and alerting properties are set in the plugin.json)

Metrics from the alerting engine

The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes internal metrics. See also, View alert rules and their current state.

Metric Name	Type	Description
`grafana_alerting_alerts`	gauge	How many alerts by state
`grafana_alerting_request_duration`	histogram	Histogram of requests to the Alerting API
`grafana_alerting_active_configurations`	gauge	The number of active, non default Alertmanager configurations for grafana managed alerts
`grafana_alerting_rule_evaluations_total`	counter	The total number of rule evaluations
`grafana_alerting_rule_evaluation_failures_total`	counter	The total number of rule evaluation failures
`grafana_alerting_rule_evaluation_duration`	summary	The duration for a rule to execute
`grafana_alerting_rule_group_rules`	gauge	The number of rules

Alerting on numeric data

Among certain data sources numeric data that is not time series can be directly alerted on, or passed into Server Side Expressions (SSE). This allows for more processing and resulting efficiency within the data source, and it can also simplify alert rules. When alerting on numeric data instead of time series data, there is no need to reduce each labeled time series into a single number. Instead labeled numbers are returned to Grafana instead.

Tabular Data

This feature is supported with backend data sources that query tabular data:

SQL data sources such as MySQL, Postgres, MSSQL, and Oracle.
The Azure Kusto based services: Azure Monitor (Logs), Azure Monitor (Azure Resource Graph), and Azure Data Explorer.

A query with Grafana managed alerts or SSE is considered numeric with these data sources, if:

The “Format AS” option is set to “Table” in the data source query.
The table response returned to Grafana from the query includes only one numeric (e.g. int, double, float) column, and optionally additional string columns.

If there are string columns then those columns become labels. The name of column becomes the label name, and the value for each row becomes the value of the corresponding label. If multiple rows are returned, then each row should be uniquely identified their labels.

Example

For a MySQL table called “DiskSpace”:

Time	Host	Disk	PercentFree
2021-June-7	web1	/etc	3
2021-June-7	web2	/var	4
2021-June-7	web3	/var	8
…	…	…	…

You can query the data filtering on time, but without returning the time series to Grafana. For example, an alert that would trigger per Host, Disk when there is less than 5% free space:

SELECT Host, Disk, CASE WHEN PercentFree < 5.0 THEN PercentFree ELSE 0 END FROM (
  SELECT
      Host,
      Disk,
      Avg(PercentFree)
  FROM DiskSpace
  Group By
    Host,
    Disk
  Where __timeFilter(Time)

This query returns the following Table response to Grafana:

Host	Disk	PercentFree
web1	/etc	3
web2	/var	4
web3	/var	0

When this query is used as the condition in an alert rule, then the non-zero will be alerting. As a result, three alert instances are produced:

Labels	Status
{Host=web1,disk=/etc}	Alerting
{Host=web2,disk=/var}	Alerting
{Host=web3,disk=/var}	Normal

Alertmanager

Sat, 04 Apr 2026 12:26:57 +0000

Alertmanager

The Alertmanager helps both group and manage alert rules, adding a layer of orchestration on top of the alerting engines. To learn more, see Prometheus Alertmanager documentation.

Grafana includes built-in support for Prometheus Alertmanager. By default, notifications for Grafana managed alerts are handled by the embedded Alertmanager that is part of core Grafana. You can configure the Alertmanager’s contact points, notification policies, silences, and templates from the alerting UI by selecting the Grafana option from the Alertmanager drop-down.

Note: Before v8.2, the configuration of the embedded Alertmanager was shared across organizations. If you are on an older Grafana version, we recommend that you use Grafana alerts only if you have one organization. Otherwise, your contact points are visible to all organizations.

Grafana alerting added support for external Alertmanager configuration. When you add an Alertmanager data source, the Alertmanager drop-down shows a list of available external Alertmanager data sources. Select a data source to create and manage alerting for standalone Cortex or Loki data sources.

You can configure one or several external Alertmanagers to receive alerts from Grafana. Once configured, both the embedded Alertmanager and any configured external Alertmanagers will receive all alerts.

You can do the setup in the “Admin” tab within the Grafana v8 Alerts UI.

Add a new external Alertmanager

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
Click Admin and then scroll down to the External Alertmanager section.
Click Add Alertmanager and a modal opens.
Add the URL and the port for the external Alertmanager. You do not need to specify the path suffix, for example, /api/v(1|2)/alerts. Grafana automatically adds this.

The external URL is listed in the table with a pending status. Once Grafana verifies that the Alertmanager is discovered, the status changes to active. No requests are made to the external Alertmanager at this point; the verification signals that alerts are ready to be sent.

Edit an external Alertmanager

Click the pen symbol to the right of the Alertmanager row in the table.
When the edit modal opens, you can view all the URLs that were added.

The edited URL will be pending until Grafana verifies it again.

State and health of alerting rules

Sat, 04 Apr 2026 12:26:57 +0000

State and health of alerting rules

The state and health of alerting rules help you understand several key status indicators about your alerts. There are three key components: alert state, alerting rule state, and alerting rule health. Although related, each component conveys subtly different information.

Alerting rule state

Normal: None of the time series returned by the evaluation engine is in a Pending or Firing state.
Pending: At least one time series returned by the evaluation engine is Pending.
Firing: At least one time series returned by the evaluation engine is Firing.

Alert state

Normal: Condition for the alerting rule is false for every time series returned by the evaluation engine.
Alerting: Condition of the alerting rule is true for at least one time series returned by the evaluation engine. The duration for which the condition must be true before an alert fires, if set, is met or has exceeded.
Pending: Condition of the alerting rule is true for at least one time series returned by the evaluation engine. The duration for which the condition must be true before an alert fires, if set, has not been met.
NoData: the alerting rule has not returned a time series, all values for the time series are null, or all values for the time series are zero.
Error: Error when attempting to evaluate an alerting rule.

Alerting rule health

Ok: No error when evaluating an alerting rule.
Error: Error when evaluating an alerting rule.
NoData: The absence of data in at least one time series returned during a rule evaluation.