Grafana Cloud

Incident response and management

Use the Assistant to stay on top of your on-call duties and incident response workflows. You can check and set up schedules, find coverage for your shifts, and quickly summarize ongoing or past incidents without leaving the chat interface.

What you’ll achieve

  • Identify who is currently on-call for specific teams or services.
  • View your upcoming shifts and request coverage.
  • Create on-call schedules for your team, and add rotations and overrides.
  • Find OnCall alert groups and review their response state.
  • Acknowledge and resolve alert groups.
  • List and filter incidents by severity, status, or date.
  • Generate summaries and timelines for specific incidents.
  • Declare incidents and drills.
  • Add notes to incidents and change their status, severity, title, and labels.
  • Assign incident roles, and add, edit, reassign, and delete incident tasks.

Before you begin

  • Grafana IRM must be enabled and configured in your Grafana stack.
  • To view schedules and incidents, your user account requires Viewer permissions or higher in Grafana IRM.
  • To request shift swaps, you must have Editor permissions.
  • To acknowledge or resolve alert groups, you need the alert groups: write permission (grafana-irm-app.alert-groups:write) in Grafana IRM. The Admin, Editor, OnCaller, and Alert Groups Editor roles include it.
  • To create schedules, add rotations or overrides, or cancel overrides, you need the schedules: write permission (grafana-irm-app.schedules:write) in Grafana IRM. The Admin, Editor, OnCaller, and Schedules Editor roles include it. If your organization turns on Require team membership for updates, you also need to be a member of the team that owns the schedule.

For more information, refer to Roles and permissions for Grafana IRM.

Manage on-call schedules

Use the Assistant to quickly find the right person to contact during an outage or to plan your own schedule.

Check who is on call

Identify the current on-call engineer for a specific service or team to ensure the right person is notified during an issue.

Who is on call for @backend-service?

Refine your request by specifying a team name or different service if the initial result isn’t what you need.

Who is on call for the platform team?

Check your schedule

Retrieve your upcoming shifts to plan your availability and ensure you are prepared for your on-call rotation.

When am I on call next?

Refine the query to look at specific time ranges.

Show my shifts for the next two weeks.

Set up on-call schedules

The Assistant can create an on-call schedule for your team, add the rotation that decides who’s on call, and add overrides when someone covers part of the schedule. It works with web schedules, the ones you edit in Grafana IRM, not iCal or API/Terraform schedules. It asks for your approval before each change and makes the change with your own Grafana IRM permissions. If you can’t edit schedules, the Assistant tells you, so you can ask your Grafana IRM administrator for access.

Create a schedule

Ask the Assistant to create a schedule for your team. It asks for anything you leave out, one step at a time, and suggests a time zone and handoff time that suit the people who take turns.

Create an on-call schedule for the platform team.

Refine the request with the people, the time zone, and the first shift.

Create a weekly on-call schedule for the checkout team with Alice, Bob, and Carol in the Europe/Madrid time zone, starting Monday at 10:00.

Before it adds the rotation, the Assistant previews the first shifts so you can check them. When it’s done, it links to the new schedule.

A new schedule doesn’t page anyone until an escalation chain notifies it. The Assistant can’t set up escalation chains, so it reminds you to add the schedule to one in Grafana IRM. For more information, refer to Escalation chains.

Add a rotation

Add a rotation to an existing schedule, for example, to cover weekends or to add a second layer. Describe who takes turns and how often. Before it adds the rotation, the Assistant describes the first shifts so you can check them, then asks for your approval.

Add a weekly rotation to the Platform on-call schedule with Alice and Bob, starting Monday at 09:00.

Refine the rotation with working days, a layer, or an end date.

Add a daily rotation on weekdays from 09:00 to 17:00 with Alice, Bob, and Carol until the end of the year.

The Assistant also tells you if Grafana IRM can’t page someone in the rotation, or if someone has no notification rules and only gets email until they add some.

Add or cancel an override

Use an override when someone covers part of the schedule, for example, during a vacation.

Put Carol on call for the Platform on-call schedule tomorrow from 10:00 to 18:00.

Cancel an override you no longer need. If the override has already started, it ends now.

Cancel Carol’s override on the Platform on-call schedule.

Manage shifts

The Assistant helps you manage your work-life balance by facilitating shift swaps directly from the conversation.

Request a shift swap

Initiate a swap request if you cannot make a scheduled shift. The Assistant will help you find the specific shift and create the request.

I need to swap my shift on Friday.

Refine the request by providing the reason for the swap.

Request a swap for my shift tomorrow due to a doctor’s appointment.

A swap request notifies the people who can cover your shift, so the Assistant asks for your approval before it sends one. You can’t request a swap from the CLI, which can only read IRM data.

Configure automated investigations

Configure webhooks to automatically trigger investigations when alert groups are created or incidents are declared in Grafana IRM. This ensures investigations start immediately, providing context before human responders arrive.

Set up webhooks

You can configure webhooks directly in the Assistant settings.

  1. Navigate to AI > Settings > Investigations.
  2. For incidents: Click Enable incident webhooks to trigger investigations when incidents are updated.
  3. For alert groups: Select your integrations, name the webhook, and click Enable alert group webhooks to trigger on alert creation and status changes.

The Assistant will automatically post investigation links back to the alert group or incident page and update context as the situation evolves.

Track incidents

Stay informed about the health of your system by querying the incident history and getting rapid summaries.

List incidents

Filter incidents to see what is currently active or review past outages to understand system stability.

Show me active critical incidents.

Refine the list by filtering for specific services, dates, or teams.

List resolved incidents from last week for @frontend.

Status filters can use your IRM environment’s statuses, including custom incident statuses.

Track alert groups

Alert groups show the operational side of alerts routed through OnCall, including notification, acknowledgment, escalation, silence, and resolution state. Use the Assistant to find alert groups, open the full alert group, and compare alert response state with related incidents.

List alert groups

Filter alert groups by status, title text, public ID, alert group number, or labels. Alert group search covers recent alert groups and returns links back to Grafana.

Show unresolved alert groups for the platform team.

Find alert group #42.

Search alert groups with label service:checkout.

Review alert group details

Ask for details when you need the alert labels, status, timing, and related navigation link.

Get details for alert group I3XYZABC.

Acknowledge or resolve an alert group

Ask the Assistant to acknowledge an alert group you’re working on, or resolve it once the issue is fixed. The Assistant asks for your approval before it changes the alert group.

Acknowledge alert group I3XYZABC.

If your organization requires a resolution note, include it in the request.

Resolve alert group #42 with the note “Fixed by rolling back the checkout deploy.”

Summarize an incident

Get a quick catch-up on an incident’s timeline and key events without navigating through full logs.

Summarize the last critical incident.

Refine the summary to focus on specific aspects or ask for a link to the full report.

Give me a timeline for incident #123.

Declare an incident

Ask the Assistant to declare an incident when an issue needs a coordinated response. Declaring an incident starts your incident response in Grafana IRM, for example, it creates the incident room and runs your incident hooks, so the Assistant first checks for a matching active incident and asks for your approval before it declares one. When it’s done, it links to the new incident.

Declare an incident for checkout errors.

Refine the request with a severity, labels, or the alert group that started it. If you leave out the severity, Grafana IRM uses your organization’s default.

Declare a critical incident for checkout 5xx errors from alert group I3XYZABC, with the label service:checkout.

To practice your response without affecting your incident dashboards, ask for a drill.

Declare a drill for a database failover.

Update incidents

The Assistant can record what you find on the incident timeline, keep the incident’s details current, and organize the people and work on it. It asks for your approval before each change and makes the change with your own Grafana IRM permissions.

To stop being asked for small changes that are easy to undo, select Always allow when you approve one. This saves a preference under your account that covers adding and removing labels and changing a task’s text or status. Every other incident change, such as declaring an incident, changing its status or severity, assigning roles, or deleting a task, still asks each time.

Add a note to an incident

Add a finding, a check you ran, or a handover to the incident timeline.

Add a note to incident #123 that the error rate dropped after the rollback.

Change status or severity

Resolve an incident once the impact is over, or change its severity as you learn more.

Resolve incident #123.

Set the severity of incident #123 to critical.

Status changes can use your IRM environment’s custom incident statuses.

Rename or label an incident

Give an incident a clearer title as you learn more, or add and remove labels so it’s easier to find and report on. Grafana IRM labels have a key and a value, such as service:checkout, and a label without a key goes under tags.

Rename incident #123 to “Checkout 5xx after the v42 deploy”.

Add the label service:checkout to incident #123.

Assign incident roles

Assign the commander, investigator, or observer role, or one of your organization’s custom roles. The Assistant can assign roles to you or to someone it can identify by Grafana login or email address. Grafana IRM may decline some changes, for example, a second role for someone who already has one, and the Assistant tells you when it does.

Make me the commander of incident #123.

Remove Alice as investigator on incident #123.

Track incident tasks

Add tasks to an incident, assign them to someone, and mark them in progress or done as the work moves along. You can also reword a task, hand it to someone else, unassign it, or delete a task you no longer need. Grafana IRM maintains some tasks itself, and the Assistant can’t change those.

Add a task to incident #123 to roll back the checkout deploy, assigned to Bob.

Mark the rollback task on incident #123 as done.

Reassign the rollback task on incident #123 to Carol and change it to “Roll back checkout to v41”.

Delete the “notify customers” task on incident #123.

Next steps