Integration references on Grafana Labs

Alertmanager integration for Grafana OnCall

Fri, 03 Apr 2026 16:45:36 +0000

Alertmanager integration for Grafana OnCall

Note
⚠️ Legacy integration ⚠️ Integrations that were created before version 1.3.21 (1 August 2023) were marked as (Legacy) and migrated. These integrations are receiving and escalating alerts, but some manual adjustments may be required.

The Alertmanager integration handles alerts from Prometheus Alertmanager. This integration is the recommended way to send alerts from Prometheus deployed in your infrastructure, to Grafana OnCall.

Pro tip: Create one integration per team, and configure alertmanager labels selector to send alerts only related to that team

Configuring Grafana OnCall to Receive Alerts from Prometheus Alertmanager

In the Integrations tab, click + New integration.
Select Alertmanager Prometheus from the list of available integrations.
Enter a name and description for the integration, click Create
A new page will open with the integration details. Copy the OnCall Integration URL from HTTP Endpoint section. You will need it when configuring Alertmanager.

Configuring Alertmanager to Send Alerts to Grafana OnCall

Add a new Webhook receiver to receivers section of your Alertmanager configuration
Set url to the OnCall Integration URL from previous section
- Note: The url has a trailing slash that is required for it to work properly.
Set send_resolved to true, so Grafana OnCall can autoresolve alert groups when they are resolved in Alertmanager
It is recommended to set max_alerts to less than 100 to avoid requests that are too large.
Use this receiver in your route configuration

Here is the example of final configuration:

route:
  receiver: "oncall"
  group_by: [alertname, datacenter, app]

receivers:
  - name: "oncall"
    webhook_configs:
      - url: <integration-url>
        send_resolved: true
        max_alerts: 100

Complete the Integration Configuration

Complete configuration by setting routes, templates, maintenances, etc. Read more in this section

Note about grouping and autoresolution

Grafana OnCall relies on the Alertmanager grouping and autoresolution mechanism to ensure consistency between alert state in OnCall and AlertManager. It’s recommended to configure grouping on the Alertmanager side and use default grouping and autoresolution templates on the OnCall side. Changing this templates might lead to incorrect grouping and autoresolution behavior. This is unlikely to be what you want, unless you have disabled grouping on the AlertManager side.

Configuring OnCall Heartbeats (optional)

An OnCall heartbeat acts as a monitoring for monitoring systems. If your monitoring is down and stop sending alerts, Grafana OnCall will notify you about that.

Configuring Grafana OnCall Heartbeat

Go to Integration Page, click on three dots on top right, click Heartbeat settings
Copy OnCall Heartbeat URL, you will need it when configuring Alertmanager
Set up Heartbeat Interval, time period after which Grafana OnCall will start a new alert group if it doesn’t receive a heartbeat request

Configuring Alertmanager to send heartbeats to Grafana OnCall Heartbeat

You can configure Alertmanager to regularly send alerts to the heartbeat endpoint. Add vector(1) as a heartbeat generator to prometheus.yaml. It will always return true and act like always firing alert, which will be sent to Grafana OnCall once in a given period of time:

groups:
  - name: meta
    rules:
      - alert: heartbeat
        expr: vector(1)
        labels:
          severity: none
        annotations:
          description: This is a heartbeat alert for Grafana OnCall
          summary: Heartbeat for Grafana OnCall

Add receiver configuration to prometheus.yaml with the OnCall Heartbeat URL:

  ...
  route:
  ...
      routes:
        - match:
          alertname: heartbeat
        receiver: 'grafana-oncall-heartbeat'
        group_wait: 0s
        group_interval: 1m
        repeat_interval: 50s
  receivers:
    - name: 'grafana-oncall-heartbeat'
    webhook_configs:
      - url: https://oncall-dev-us-central-0.grafana.net/oncall/integrations/v1/alertmanager/1234567890/heartbeat/
        send_resolved: false

Note about legacy integration

Legacy integration was using each alert from AlertManager group as a separate payload:

{
  "labels": {
    "severity": "critical",
    "alertname": "InstanceDown"
  },
  "annotations": {
    "title": "Instance localhost:8081 down",
    "description": "Node has been down for more than 1 minute"
  },
  ...
}

This behaviour was leading to mismatch in alert state between OnCall and AlertManager and draining of rate-limits, since each AlertManager alert was counted separately.

We decided to change this behaviour to respect AlertManager grouping by using AlertManager group as one payload.

{
  "alerts": [...],
  "groupLabels": {
    "alertname": "InstanceDown"
  },
  "commonLabels": {
    "job": "node", 
    "alertname": "InstanceDown"
  },
  "commonAnnotations": {
    "description": "Node has been down for more than 1 minute"
  },
  "groupKey": "{}:{alertname=\"InstanceDown\"}",
  ...
}

You can read more about AlertManager Data model here.

After-migration checklist

Integration URL will stay the same, so no need to change AlertManager or Grafana Alerting configuration. Integration templates will be reset to suit new payload. It is needed to adjust routes and outgoing webhooks manually to new payload.

Send a new demo alert to the migrated integration.
Adjust routes to the new shape of payload. You can use payload of the demo alert from previous step as an example.
If outgoing webhooks utilized the alerts payload from the migrated integration in the trigger or data template it’s needed to adjust them as well.

Elastalert integration for Grafana OnCall

Fri, 03 Apr 2026 16:45:36 +0000

ElastAlert integration for Grafana OnCall

The ElastAlert integration for Grafana OnCall handles ticket events sent from ElastAlert webhooks. The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.

You must have the role of Admin to be able to create integrations in Grafana OnCall.

Configuring Grafana OnCall to Receive Alerts from ElastAlert

In the Integrations tab, click + New integration.
Select ElastAlert from the list of available integrations.
Enter a name and description for the integration, click Create
A new page will open with the integration details. Copy the OnCall Integration URL from HTTP Endpoint section.

Configuring ElastAlert to Send Alerts to Grafana OnCall

To send an alert from ElastAlert to a webhook, follow these steps:

Refer to ElastAlert http-post docs for more details

Open your ElastAlert configuration file (e.g., config.yaml).
Locate the alert section.
Add the following configuration for the webhook alert:

alert: post
http_post_url: "http://example.com/api"
http_post_static_payload:
  title: abc123

Replace "abc123" with a suitable name for your alert, and "http://example.com/api" with OnCall Integration URL. 4. Save the configuration file.

After configuring the webhook, ElastAlert will send alerts to the specified endpoint when triggered. Make sure your webhook endpoint is configured to receive and process the incoming alerts.

Grouping, auto-acknowledge and auto-resolve

Grafana OnCall provides grouping, auto-acknowledge and auto-resolve logic for the ElastAlert integration:

Alerts created from ticket events are grouped by ticket ID
Alert groups are auto-acknowledged when the ticket status is set to “Pending”
Alert groups are auto-resolved when the ticket status is set to “Solved”

To customize this behaviour, consider modifying alert templates in integration settings.

Configuring Elastalert to send heartbeats to Grafana OnCall Heartbeat

Add the following rule to ElastAlert

    index: elastalert_status
    type: any
    alert: post
    http_post_url: {{ heartbeat_url }}
    realert:
        minutes: 1
    alert_text: elastalert is still running
    alert_text_type: alert_text_only

Grafana Alerting integration for Grafana OnCall

Fri, 03 Apr 2026 16:45:36 +0000

Grafana Alerting integration for Grafana OnCall

Note
⚠️ Legacy integration ⚠️ Integrations that were created before version 1.3.21 were marked as (Legacy) and recently migrated. These integrations are receiving and escalating alerts, but some manual adjustments might be required.

Grafana Alerting for Grafana OnCall can be set up using two methods:

Grafana OnCall is connected to the same Grafana instance being used to manage Grafana OnCall.
Grafana OnCall is connected to one or more Grafana instances, separate from the one being used to manage Grafana OnCall.

Configure Grafana Alerting in same Grafana instance

Use the following method if you are connecting Grafana OnCall with alerts coming from the same Grafana instance from which Grafana OnCall is being managed.

In Grafana OnCall, navigate to the Integrations tab and select New Integration to receive alerts.
Click Quick connect in the Grafana Alerting tile. This will open a New Grafana Alerting integration configuration window.
In the opened window, enter a name and description for the integration and choose existing or create a new contact point. This contact point will send alerts to the created integration.

Note: You must connect the contact point with a notification policy in Grafana Alerting to receive alerts in Grafana OnCall. For more information, see Contact points in Grafana Alerting
Determine the escalation chain for the new integration by either selecting an existing one or by creating a new escalation chain.
In Grafana Alerting, navigate to Alerting > Contact Points and find a contact point with a name matching the integration you created in Grafana OnCall.
Click the Edit (pencil) icon, then click Test. This will send a test alert to Grafana OnCall.

Configure external Grafana Alerting from other Grafana Instance

Connect Grafana OnCall with alerts coming from a Grafana instance that is different from the instance that Grafana OnCall is being managed:

In Grafana OnCall, navigate to the Integrations tab and select New Integration to receive alerts.
Select the Alertmanager tile.
Enter a name and description for the integration, click Create
A new page will open with the integration details. Copy the OnCall Integration URL from HTTP Endpoint section.
Go to the other Grafana instance to connect to Grafana OnCall and navigate to Alerting > Contact Points.
Select New Contact Point.
Choose the contact point type webhook, then paste the URL generated in step 3 into the URL field.

Note: You must connect the contact point with a notification policy in Grafana Alerting to receive alerts in Grafana OnCall. For more information, see Contact points in Grafana Alerting
Click the Edit (pencil) icon, then click Test. This will send a test alert to Grafana OnCall.

Note about grouping and autoresolution

Grafana OnCall relies on the Grafana Alerting grouping and autoresolution mechanism to ensure consistency between alert state in OnCall and AlertManager. It’s recommended to configure grouping on the Grafana Alerting side and use default grouping and autoresolution templates on the OnCall side. Changing this templates might lead to incorrect grouping and autoresolution behavior.

Note about legacy integration

Before we were using each alert from Grafana Alerting group as a separate payload:

{
  "labels": {
    "severity": "critical",
    "alertname": "InstanceDown"
  },
  "annotations": {
    "title": "Instance localhost:8081 down",
    "description": "Node has been down for more than 1 minute"
  },
  ...
}

This behaviour was leading to mismatch in alert state between OnCall and Grafana Alerting and draining of rate-limits, since each Grafana Alerting alert was counted separately.

We decided to change this behaviour to respect Grafana Alerting grouping by using AlertManager group as one payload.

{
  "alerts": [...],
  "groupLabels": {
    "alertname": "InstanceDown"
  },
  "commonLabels": {
    "job": "node", 
    "alertname": "InstanceDown"
  },
  "commonAnnotations": {
    "description": "Node has been down for more than 1 minute"
  },
  "groupKey": "{}:{alertname=\"InstanceDown\"}",
  ...
}

You can read more about AlertManager Data model here.

After-migration checklist

Integration URL will stay the same, so no need to change AlertManager or Grafana Alerting configuration. Integration templates will be reset to suit new payload. It is needed to adjust routes and outgoing webhooks manually to new payload.

Send a new demo alert to the migrated integration.
Adjust routes to the new shape of payload. You can use payload of the demo alert from previous step as an example.
If outgoing webhooks utilized the alerts payload from the migrated integration in the trigger or data template it’s needed to adjust them as well.

Inbound email integration for Grafana OnCall

Fri, 03 Apr 2026 16:45:36 +0000

Inbound Email integration for Grafana OnCall

Inbound Email integration will consume emails from dedicated email address and make alert groups from them.

Configure required environment variables

Refer to [Inbound Email Setup] for details.

Configure Inbound Email integration for Grafana OnCall

You must have an Admin role to create integrations in Grafana OnCall.

In the Integrations tab, click + New integration to receive alerts.
Select Inbound Email from the list of available integrations.
Get your dedicated email address in the Integration email section and use it to send your emails.

Grouping and auto-resolve

Alert groups will be grouped by email subject and auto-resolved if the email message text equals “OK”. This behaviour can be modified via custom templates.

Alerts from Inbound Email integration have the following payload:

{
   "subject": "<your_email_subject>",
   "message": "<your_email_message>",
   "sender": "<your_email_sender_address>"
}

Inbound Webhook for Grafana OnCall

Fri, 03 Apr 2026 16:45:36 +0000

Inbound Webhook integrations for Grafana OnCall

Grafana OnCall directly supports many integrations, those that aren’t currently listed in the Integrations menu can be connected using the webhook integration and configured alert templates.

With the webhook integration, you can connect to any alert source that isn’t listed in the Create Integration page.

There are two available formats, Webhook and Formatted Webhook.

Webhook will pull all of the raw JSON payload and display it in the manner that it’s received.
Formatted Webhook can be used if the alert payload sent by your monitoring service is formatted in a way that OnCall recognizes.

The following fields are recognized, but none are required:
- alert_uid: a unique alert ID for grouping.
- title: a title.
- image_url: a URL for an image attached to alert.
- state: either ok or alerting. Helpful for auto-resolving.
- link_to_upstream_details: link back to your monitoring system.
- message: alert details.

To configure a webhook integration:

In the Integrations tab, click + New integration to receive alerts.
Select either Webhook or Formatted Webhook integration.
Follow the configuration steps in the How to connect section of the integration settings.
Use the unique webhook URL to complete any configuration in your monitoring service to send POST requests. Use any http client, e.g. curl to send POST requests with any payload.

For example:

curl -X POST \
https://a-prod-us-central-0.grafana.net/integrations/v1/formatted_webhook/m12xmIjOcgwH74UF8CN4dk0Dh/ \
-H 'Content-Type: Application/json' \
-d '{
    "alert_uid": "08d6891a-835c-e661-39fa-96b6a9e26552",
    "title": "The whole system is down",
    "image_url": "https://upload.wikimedia.org/wikipedia/commons/e/ee/Grumpy_Cat_by_Gage_Skidmore.jpg",
    "state": "alerting",
    "link_to_upstream_details": "https://en.wikipedia.org/wiki/Downtime",
    "message": "Smth happened. Oh no!"
}'

To learn how to use custom alert templates for formatted webhooks, see Configure alerts templates.

Kapacitor integration for Grafana OnCall

Fri, 03 Apr 2026 16:45:36 +0000

Kapacitor integration for Grafana OnCall

The Kapacitor integration for Grafana OnCall handles ticket events sent from Kapacitor webhooks. The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.

Configuring Grafana OnCall to Receive Alerts from Kapacitor

In the Integrations tab, click + New integration.
Select Kapacitor from the list of available integrations.
Enter a name and description for the integration, click Create
A new page will open with the integration details. Copy the OnCall Integration URL from HTTP Endpoint section.

Configuring Kapacitor to Send Alerts to Grafana OnCall

To send an alert from Kapacitor, you can follow these steps:

Create a Kapacitor TICKscript or modify an existing one to define the conditions for triggering the alert. The TICKscript specifies the data source, data processing, and the alert rule. Here’s an example of a simple TICKscript:

stream
    |from()
        .measurement('measurement_name')
        .where(lambda: <condition>)
    |alert()
        .webhook('<webhook_url>')

Replace 'measurement_name' with the name of the measurement you want to monitor, <condition> with the condition that triggers the alert, and '<webhook_url>' with OnCall Integration URL 2. Save the TICKscript file with a .tick extension, for example, alert_script.tick. 3. Start the Kapacitor service using the TICKscript:

kapacitor define <alert_name> -tick /path/to/alert_script.tick
kapacitor enable <alert_name>
kapacitor reload

Replace <alert_name> with a suitable name for your alert. 4. Ensure that the Kapacitor service is running and actively monitoring the data. When the condition defined in the TICKscript is met, Kapacitor will trigger the alert and send a POST request to the specified webhook URL with the necessary information. Make sure your webhook endpoint is configured to receive and process the incoming alerts from Kapacitor.

Manual paging

Fri, 03 Apr 2026 16:45:36 +0000

Page people manually

Grafana OnCall relies on automated and pre-configured workflows, such as integrations, routes, and escalation chains to handle most of the incident response process. However, sometimes you might need to page a team or request assistance from specific people that are not part of these pre-defined rules.

For such ad-hoc scenarios, Grafana OnCall allows you to create an alert group, input necessary information, and decide who will be alerted – a team, or a set of users.

Page a team

Click on + Escalation on the Alert groups page to start creating a new alert group. From there, you can configure the alert group to notify a particular team and optionally include additional users. Here are the inputs you need to fill in:

Message: Write a message to provide more details or instructions to those whom you are paging.
Team: Select the team you want to page. The team’s direct paging integration will be used for notification. Note that you will only see teams that have a “contactable” direct paging integration (ie. it has an escalation chain assigned to it, or has at least one Chatops integration connected to send notifications to).
Users: Include more users to the alert group. For each additional user, you can select a notification policy: default or important.

The same feature is also available as /escalate Slack command.

Add users to an existing alert group

If you want to page more people for an existing alert group, you can do so using the + Add button, within the “Participants” section on the specific alert group’s page. The same functionality is available in Slack using the Responders button in the alert group’s message.

Notifying additional users doesn’t disrupt or interfere with the escalation chain configured for the alert group; it simply adds more responders and notifies them immediately. Note that adding users for an existing alert group will page them even if the alert group is silenced or acknowledged, but not if the alert group is resolved.

It’s not possible to page a team for an existing alert group. To page a specific team, you need to create a new alert group.

Learn the flow and handle warnings

When you pick a team to page, Grafana OnCall will automatically use the right direct paging integration for the team. “Direct paging” is a special kind of integration in Grafana OnCall that is unique per team and is used to send alerts to the team’s ChatOps channels and start an appropriate escalation chain.

Set up direct paging for a team

By default all teams will have a direct paging integration created for them. Each direct paging integration will be created with two routes:

a non-default route which has a Jinja2 filtering term of {{ payload.oncall.important }} (see Important Escalations below for more details)
a default route to capture all other alerts

However, these integrations are not configured by default to be “contactable” (ie. their routes will have no escalation chains assigned to them, nor any Chatops integrations connected to send notifications to). If a team does not have their direct paging integration configured, such that it is “contactable” , you will not be able to direct page this team. If this happens, consider following the following steps for the team (or reach out to the relevant team and suggest doing so).

Navigate to the Integrations page and find the “Direct paging” integration for the team in question. From the integration’s detail page, you can customize its settings, link it to an escalation chain, and configure associated ChatOps channels. To confirm that the integration is functioning as intended, create a new alert group and select the same team for a test run.

Important escalations

Sometimes you really need to get the attention of a particular team. When directly paging a team, it is possible to page them using an “important escalation”. Practically speaking, this will create an alert, using the specified team’s direct paging integration as such:

{
  "oncall": {
    "title": "IRM is paging Network team to join escalation",
    "message": "I really need someone from your team to come take a look! The k8s cluster is down!",
    "uid": "8a20b8d1-56fd-482e-824e-43fbd1bd7b10",
    "author_username": "irm",
    "permalink": null,
    "important": true
  }
}

When you are directly paging a team, either via the web UI, chatops apps, or the API, you can specify that this esclation be “important”, which will effectively set the value of oncall.important to true. As mentioned above in Set up direct paging for a team, direct paging integrations come pre-configured with two routes, with the non-default route having a Jinja2 filtering term of {{ payload.oncall.important }}.

This allows teams to be contacted via different escalation chains, depending on whether or not the user paging them believes that this is an “important escalation”.

Manage your Direct Paging Integrations via IaC

By default, Grafana OnCall will automatically create and manage direct paging integrations for each team. However, you can opt-out of this behavior by enabling the “Manually manage direct paging integrations” setting in the Settings page.

When this setting is enabled, you will be able to manually manage direct paging integrations for each team. This can be useful if you would like to manage your direct paging integrations via IaC (e.g. Terraform or Crossplane).

Note: once you’ve enabled this feature, we will not delete the existing direct paging integrations for your teams. Instead, you can either “import” these into your IaC tooling, or delete them manually and recreate via IaC.