Google Cloud Monitoring data source on Grafana Labs

Configure the Google Cloud Monitoring data source

Fri, 03 Apr 2026 19:43:06 +0000

Configure the Google Cloud Monitoring data source

This document provides instructions for configuring the Google Cloud Monitoring data source in Grafana.

Before you begin

Before you begin, ensure you have the following:

Grafana permissions: You must have the Organization administrator role to configure data sources.
GCP project: A Google Cloud Platform project.
GCP permissions: Permissions to create a service account or configure GCE default service account settings in your GCP project.

Grafana includes built-in support for Google Cloud Monitoring, so you don’t need to install a plugin.

Set up GCP authentication

Before you can request data from Google Cloud Monitoring, you must configure authentication. All requests to Google APIs are performed on the server-side by the Grafana backend.

For authentication options and configuration details, refer to Google authentication.

When you configure Google authentication, note the following requirements specific to Google Cloud Monitoring.

Configure a GCP Service Account

When you create a Google Cloud Platform (GCP) Service Account and key file, the Service Account must have the Monitoring Viewer role (Role > Select a role > Monitoring > Monitoring Viewer):

Grant the GCE Default Service Account scope

If Grafana is running on a Google Compute Engine (GCE) virtual machine, when you configure a GCE Default Service Account, you must also grant that Service Account access to the “Cloud Monitoring API” scope.

Enable Google Cloud Platform APIs

Before you can request data from Google Cloud Monitoring, you must enable the necessary APIs in your GCP project.

Open the Monitoring and Cloud Resource Manager API pages:
- Monitoring API
- Cloud Resource Manager API
On each page, click Enable.

Enable GCP APIs

Add the data source

To add the Google Cloud Monitoring data source:

Click Connections in the left-side menu.
Click Add new connection.
Enter Google Cloud Monitoring in the search bar.
Select Google Cloud Monitoring.
Click Add new data source in the upper right.

You’re taken to the Settings tab where you configure the data source.

Configure the data source in the UI

The following are configuration options for the Google Cloud Monitoring data source.

Setting	Description
Name	Sets the name you use to refer to the data source in panels and queries.
Default	Sets whether the data source is pre-selected for new panels.
Universe Domain	The universe domain to connect to. For more information, refer to the Google Cloud universe domains documentation. Defaults to `googleapis.com`.

Authentication

Configure how Grafana authenticates with Google Cloud.

Setting	Description
Authentication type	Select the authentication method. Choose Google JWT File to use a service account key file, or GCE Default Service Account if Grafana is running on a GCE virtual machine.

JWT Key Details

These settings appear when you select Google JWT File as the authentication type.

Setting	Description
JWT token	Upload or paste your Google JWT token. You can drag and drop a `.json` key file, click Click to browse files to upload, or use Paste JWT Token or Fill In JWT Token manually.

Service account impersonation

Use service account impersonation to have Grafana authenticate as a different service account than the one provided in the JWT token.

Setting	Description
Enable	Toggle to enable service account impersonation.
Service account to impersonate	Enter the email address of the service account to impersonate when making requests to Google Cloud.

Private data source connect

Only available for Grafana Cloud.

Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud. For more information on how PDC works, refer to Private data source connect. For steps on setting up a PDC connection, refer to Configure Grafana private data source connect (PDC).

Setting	Description
Private data source connect	Select a PDC connection from the drop-down menu or create a new connection.

Save and test

Click Save & test to test the connection. A successful connection displays the following message:

Successfully queried the Google Cloud Monitoring API.

Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to Provisioning Grafana.

Provisioning examples

Using the JWT (Service Account key file) authentication type:

apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      tokenUri: https://oauth2.googleapis.com/token
      clientEmail: stackdriver@myproject.iam.gserviceaccount.com
      authenticationType: jwt
      defaultProject: my-project-name
      universeDomain: googleapis.com
    secureJsonData:
      privateKey: |
        -----BEGIN PRIVATE KEY-----
        POSEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCb1u1Srw8ICYHS
        ...
        yA+23427282348234=
        -----END PRIVATE KEY-----

Using the JWT (Service Account private key path) authentication type:

apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      tokenUri: https://oauth2.googleapis.com/token
      clientEmail: stackdriver@myproject.iam.gserviceaccount.com
      authenticationType: jwt
      defaultProject: my-project-name
      universeDomain: googleapis.com
      privateKeyPath: /etc/secrets/gce.pem

Using GCE Default Service Account authentication:

apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      authenticationType: gce
      universeDomain: googleapis.com

Provision the data source using Terraform

You can provision the Google Cloud Monitoring data source using Terraform with the Grafana Terraform provider.

For more information about provisioning resources with Terraform, refer to the Grafana as code using Terraform documentation.

Terraform prerequisites

Before you begin, ensure you have the following:

Terraform installed.
Grafana Terraform provider configured with appropriate credentials.
For Grafana Cloud: A Cloud Access Policy token with data source permissions.

Provider configuration

Configure the Grafana provider to connect to your Grafana instance:

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.0.0"
    }
  }
}

# For Grafana Cloud
provider "grafana" {
  url  = "<YOUR_GRAFANA_CLOUD_STACK_URL>"
  auth = "<YOUR_SERVICE_ACCOUNT_TOKEN>"
}

# For self-hosted Grafana
# provider "grafana" {
#   url  = "http://localhost:3000"
#   auth = "<API_KEY_OR_SERVICE_ACCOUNT_TOKEN>"
# }

Terraform examples

The following examples show how to configure the Google Cloud Monitoring data source for each authentication method.

Using the JWT (Service Account key file) authentication type:

resource "grafana_data_source" "google_cloud_monitoring" {
  type = "stackdriver"
  name = "Google Cloud Monitoring"

  json_data_encoded = jsonencode({
    tokenUri           = "https://oauth2.googleapis.com/token"
    clientEmail        = "<SERVICE_ACCOUNT_EMAIL>"
    authenticationType = "jwt"
    defaultProject     = "<GCP_PROJECT_ID>"
    universeDomain     = "googleapis.com"
  })

  secure_json_data_encoded = jsonencode({
    privateKey = "<PRIVATE_KEY_CONTENT>"
  })
}

Using the JWT (Service Account private key path) authentication type:

resource "grafana_data_source" "google_cloud_monitoring" {
  type = "stackdriver"
  name = "Google Cloud Monitoring"

  json_data_encoded = jsonencode({
    tokenUri           = "https://oauth2.googleapis.com/token"
    clientEmail        = "<SERVICE_ACCOUNT_EMAIL>"
    authenticationType = "jwt"
    defaultProject     = "<GCP_PROJECT_ID>"
    universeDomain     = "googleapis.com"
    privateKeyPath     = "/etc/secrets/gce.pem"
  })
}

Using GCE Default Service Account authentication:

resource "grafana_data_source" "google_cloud_monitoring" {
  type = "stackdriver"
  name = "Google Cloud Monitoring"

  json_data_encoded = jsonencode({
    authenticationType = "gce"
    universeDomain     = "googleapis.com"
  })
}

For all available configuration options, refer to the Grafana provider data source resource documentation.

Next steps

After you configure the Google Cloud Monitoring data source, you can:

Query GCP metrics using the visual Builder, MQL, SLO, or PromQL query types.
Create template variables for dynamic, reusable dashboards.
Add annotations to overlay GCP events on your graphs.
Set up alerting to receive notifications based on GCP metrics and SLOs.
Explore your data to investigate metrics without building a dashboard.
Import pre-configured dashboards for popular GCP services.

Google authentication

Fri, 03 Apr 2026 19:43:06 +0000

Google authentication

This document explains how to configure authentication between Grafana and Google Cloud Platform (GCP). You must configure authentication before you can use the Google Cloud Monitoring data source to query metrics and SLOs.

All requests to Google APIs are performed on the server-side by the Grafana backend.

Before you begin

Before you configure authentication, ensure you have the following:

A Google Cloud Platform project with the Monitoring API enabled.
Permissions to create service accounts or configure GCE instance settings in your GCP project.
Access to the Grafana data source configuration page.

Supported authentication methods

The Google Cloud Monitoring data source supports the following authentication methods:

Method	Use case
Google JWT File	Use when Grafana runs outside of GCP, or when you need explicit control over credentials.
GCE Default Service Account	Use when Grafana runs on a Google Compute Engine VM with a configured service account.
Service Account Impersonation	Use when you need Grafana to act as a different service account than the one it authenticates with.

Use a Google service account key file

Use this method when Grafana runs outside of Google Cloud Platform, or when you need explicit control over which credentials are used.

Each Grafana data source connects to one GCP project by default. To visualize data from multiple GCP projects, create one data source per project, or use service account impersonation.

Create a GCP service account and key file

To create a service account and download its key file:

Navigate to the APIs and Services Credentials page in the GCP Console.
Click the Create credentials dropdown and select Service account.
In Service account name, enter a name for the account.
Click Create and continue.
In the Grant this service account access to project section, select the Monitoring Viewer role from the Role dropdown.
Click Continue, then click Done.
In the service accounts list, click the service account you created.
Go to the Keys tab and click Add key > Create new key.
Select JSON and click Create.

A JSON key file downloads to your computer.
Store the key file securely. It grants access to your Google Cloud data.

Upload the key file to Grafana

In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
Under Authentication type, select Google JWT File.
Upload the JSON key file using one of the available methods (drag and drop, browse, or paste).
Click Save & test to verify the connection.

Grant access to multiple projects

You can configure a single service account to access multiple GCP projects:

Create a service account and key file following the steps above.
Note the service account email address (for example, grafana-monitoring@my-project.iam.gserviceaccount.com).
In each additional project you want to access:
1. Navigate to IAM & Admin > IAM.
2. Click Grant access.
3. Enter the service account email address.
4. Assign the Monitoring Viewer role.
5. Click Save.
Use the original key file in Grafana. The service account can now access all configured projects.

Use GCE Default Service Account

When Grafana runs on a Google Compute Engine (GCE) virtual machine, it can automatically retrieve credentials from the GCE metadata server. This method doesn’t require you to create or manage key files.

Prerequisites for GCE authentication

Before using this method, ensure the following:

Grafana is running on a GCE virtual machine.
The VM has a service account assigned with the Monitoring Viewer role.
The VM has the Cloud Monitoring API scope enabled.

Configure the GCE instance

In the GCP Console, navigate to Compute Engine > VM instances.
Stop the VM if it’s running (you can’t change scopes on a running VM).
Click the VM name, then click Edit.
Under Service account, select a service account with the Monitoring Viewer role.
Under Access scopes, select Set access for each API and enable the Cloud Monitoring API (read-only).
Click Save and restart the VM.

Configure the data source

In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
Under Authentication type, select GCE Default Service Account.
Click Save & test to verify the connection.

For more information about GCE service accounts, refer to the Google documentation on service accounts for instances.

Configure service account impersonation

Service account impersonation allows Grafana to authenticate as one service account but act as a different service account when making API requests. This is useful for:

Accessing resources across multiple projects with a single configuration.
Following the principle of least privilege by separating authentication from authorization.
Auditing and tracking which service account accessed specific resources.

Prerequisites for impersonation

The service account used by Grafana (the “caller”) must have the following:

The iam.serviceAccounts.getAccessToken permission on the target service account.
This permission is included in the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator).

The service account being impersonated (the “target”) must have:

The Monitoring Viewer role on the projects you want to access.

Configure impersonation

In the GCP Console, grant the caller service account the Service Account Token Creator role on the target service account.
Grant the target service account the Monitoring Viewer role on the relevant projects.
In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
Configure authentication using either Google JWT File or GCE Default Service Account.
Enable Service Account Impersonation.
Enter the email address of the target service account.
Click Save & test to verify the connection.

For more information, refer to the Google documentation on service account impersonation.

Google Cloud Monitoring query editor

Fri, 03 Apr 2026 19:43:06 +0000

Google Cloud Monitoring query editor

This topic explains querying specific to the Google Cloud Monitoring data source. For general documentation on querying data sources in Grafana, see Query and transform data.

Query types

The Google Cloud Monitoring query editor supports the following query types:

Query type	Description
Builder	Build metrics queries visually by selecting a service, metric, filters, and aggregation options.
MQL	Write queries using the Monitoring Query Language for advanced use cases.
Service Level Objectives (SLO)	Query SLO data to track service reliability and error budgets.
PromQL	Write Prometheus-style queries against Google Cloud Monitoring metrics.

Query metrics

The metrics query editor helps you select metrics, group and aggregate by labels and time, and use filters to specify which time series you want to query.

Create a metrics query

Select Builder in the Query type dropdown.
Select a project from the Project dropdown.
Select a Google Cloud Platform service from the Service dropdown.
Select a metric from the Metric dropdown.
(Optional) Use the plus and minus icons in the filter and group-by sections to add and remove filters or group-by clauses.

Google Cloud Monitoring supports several metrics types, such as GAUGE, DELTA, and CUMULATIVE. Each supports different aggregation options, such as reducers and aligners. Additionally, metrics have specific value types that can be either scalar or a distribution.

The metrics query editor lists available aggregation methods for a selected metric, and sets a default aggregation, reducer and aligner when you select a metric.

In the case that the metric value type is a distribution, the aggregation will be set by default to the mean. For scalar value types, there is no aggregation by default.

The various metrics are documented here and further details on the kinds and types of metrics can be found here.

Note
Distribution metrics are typically best visualized as either a heatmap or histogram. When visualizing in this way, aggregation is not necessary. However, for other visualization types, performance degradation may be observed when attempting to query distribution metrics that are not aggregated due to the number of potential buckets that can be returned. For more information on how to visualize distribution metrics refer to this page.

Apply a filter

To add and apply a filter:

Click the plus icon in the filter section.
Select a field to filter by.
Enter a filter value, such as instance_name = grafana-1.

To remove the filter, click the trash icon.

Use simple wildcards

When you set the operator to = or !=, you can add wildcards to the filter value field.

For example, entering us-* captures all values that start with “us-”, and entering *central-a captures all values that end with “central-a”. *-central-* captures all values with the substring of “-central-”.

Simple wildcards perform better than regular expressions.

Use regular expressions

When you set the operator to =~ or !=~, you can add regular expressions to the filter value field.

For example, entering us-central[1-3]-[af] matches all values that start with “us-central”, then are followed by a number in the range of 1 to 3, a dash, and then either an “a” or an “f”.

You don’t need leading and trailing slashes when you create these regular expressions.

Configure pre-processing options

The query editor displays pre-processing options when the selected metric has a metric type of DELTA or CUMULATIVE.

The Rate option aligns and converts data points to a rate per time series.
The Delta option aligns data points by their delta (difference) per time series.

Group time series

To combine multiple time series and reduce the amount of data returned for a metric, specify a grouping and a function.

Group by

Use the Group By segment to group resource or metric labels, which reduces the number of time series and aggregates the results by group.

For example, if you group by instance_name, Grafana displays an aggregated metric for the specified Compute instance.

Metadata labels

Resource metadata labels contain information that can uniquely identify a resource in Google Cloud. A time series response returns metadata labels only if they’re part of the Group By segment in the time series request.

There’s no API for retrieving metadata labels, so you can’t populate the Group By list with the metadata labels that are available for the selected service and metric. However, the Group By field list comes with a pre-defined set of common system labels.

You can’t pre-define user labels, but you can enter them manually in the Group By field. If you include a metadata label, user label, or system label in the Group By segment, you can create filters based on it and expand its value on the Alias field.

Group by function

To combine the time series in the group into a single time series, select a grouping function.

Align data

You can align all data points received in a fixed length of time by applying an alignment function to combine those data points, and then assigning a timestamp to the result.

Select an alignment function

During alignment, all data points are received in a fixed interval. Within each interval, as defined by the alignment period), and for each time series, the data is aggregated into a single point. The value of that point is determined by the type of alignment function you use.

For more information on alignment functions, refer to alignment metric selector.

Define the alignment period

You can set the Alignment Period field to group a metric by time if you’ve chosen an aggregation.

The default, called “cloud monitoring auto”, uses GCP Google Cloud Monitoring’s default groupings. These enable you to compare graphs in Grafana with graphs in the Google Cloud Monitoring UI.

The default values for “cloud monitoring auto” are:

1m for time ranges < 23 hours
5m for time ranges >= 23 hours and < 6 days
1h for time ranges >= 6 days

The other automatic option is “grafana auto”, which automatically sets the Group By time depending on the time range chosen and width of the time series panel.

For more information about “grafana auto”, refer to Interval variable.

You can also choose fixed time intervals to group by, like 1h or 1d.

Set alias patterns

The Alias By helps you control the format of legend keys. By default, Grafana shows the metric name and labels, which can be long and difficult to read.

You can use patterns in the alias field to customize the legend key’s format:

Alias pattern	Description	Alias pattern example	Example result
`{{metric.label.xxx}}`	Returns the metric label value.	`{{metric.label.instance_name}}`	`grafana-1-prod`
`{{metric.type}}`	Returns the full Metric Type.	`{{metric.type}}`	`compute.googleapis.com/instance/cpu/utilization`
`{{metric.name}}`	Returns the metric name part.	`{{metric.name}}`	`instance/cpu/utilization`
`{{metric.service}}`	Returns the service part.	`{{metric.service}}`	`compute`
`{{resource.label.xxx}}`	Returns the resource label value.	`{{resource.label.zone}}`	`us-east1-b`
`{{resource.type}}`	Returns the name of the monitored resource type.	`{{resource.type}}`	`gce_instance`
`{{metadata.system_labels.xxx}}`	Returns the metadata system label value.	`{{metadata.system_labels.name}}`	`grafana`
`{{metadata.user_labels.xxx}}`	Returns the metadata user label value.	`{{metadata.user_labels.tag}}`	`production`

Alias pattern examples

Using metric labels:

You can select from a list of metric and resource labels for a metric in the Group By dropdown. You can then include them in the legend key using alias patterns.

For example, given this value of Alias By: {{metric.type}} - {{metric.label.instance_name}}

An expected result would look like: compute.googleapis.com/instance/cpu/usage_time - server1-prod

Using the Monitored Resource Type:

You can also resolve the name of the Monitored Resource Type with the resource.type alias.

For example, given this value of Alias By: {{resource.type}} - {{metric.type}}

An expected result would look like: gce_instance - compute.googleapis.com/instance/cpu/usage_time

Deep-link from Grafana panels to the Google Cloud Console Metrics Explorer

You can click on a time series in the panel to access a context menu, which contains a link to View in Metrics Explorer in Google Cloud Console. The link points to the Google Cloud Console’s Metrics Explorer and runs the Grafana panel’s query there.

If you select the link, you first select an account in the Google Account Chooser. Google then redirects you to the Metrics Explorer.

The provided link is valid for any account, but displays the query only if your account has access to the query’s specified GCP project.

Understand automatic unit detection

Grafana issues one query to the Cloud Monitoring API per query editor row, and each API response includes a unit. Grafana attempts to convert the returned unit into a unit compatible with its time series panel.

If successful, Grafana displays the unit on the panel’s Y-axis. If the query editor rows return different units, Grafana uses the unit from the last query editor row in the time series panel.

Use the Monitoring Query Language

The Monitoring Query Language (MQL) query builder helps you query and display MQL results in time series format. To understand basic MQL concepts, refer to Introduction to Monitoring Query Language.

Create an MQL query

To create an MQL query:

Select MQL in the Query type dropdown.
Select a project from the Project dropdown.
Enter your MQL query in the text area.
(Optional) Configure the Graph period setting.

Press Shift+Enter to run the query.

Configure MQL options

The following options are available for MQL queries:

Setting	Description
Alias by	Control the format of legend keys. Refer to Set alias patterns for MQL queries for available patterns.
Graph period	Enable the toggle to override the default time period. Select a period from the dropdown to control the granularity of the returned time series data.

Set alias patterns for MQL queries

MQL queries use the same alias patterns as metric queries.

However, {{metric.service}} is not supported, and {{metric.type}} and {{metric.name}} show the time series key in the response.

Query Service Level Objectives

The SLO query builder helps you visualize SLO data in time series format. To understand basic concepts in service monitoring, refer to the Google Cloud Monitoring documentation.

Create an SLO query

To create an SLO query:

Select Service Level Objectives (SLO) in the Query type dropdown.
Select a project from the Project dropdown.
Select an SLO service from the Service dropdown.
Select an SLO from the SLO dropdown.
Select a time series selector from the Selector dropdown.

Grafana’s time series selectors use descriptive names that map to system names that Google uses in the Service Monitoring documentation:

Selector dropdown value	Corresponding time series selector
SLI Value	`select_slo_health`
SLO Compliance	`select_slo_compliance`
SLO Error Budget Remaining	`select_slo_budget_fraction`
SLO Burn Rate	`select_slo_burn_rate`

Alias patterns for SLO queries

The Alias By field helps you control the format of legend keys for SLO queries.

Alias pattern	Description	Example result
`{{project}}`	Returns the GCP project name.	`myProject`
`{{service}}`	Returns the service name.	`myService`
`{{slo}}`	Returns the SLO.	`latency-slo`
`{{selector}}`	Returns the selector.	`select_slo_health`

Alignment period and group-by time for SLO queries

SLO queries use the same alignment period functionality as metric queries.

Query with PromQL

The PromQL query type allows you to query Google Cloud Monitoring metrics using Prometheus Query Language (PromQL) syntax. This is useful if you’re familiar with PromQL from Prometheus or Grafana Mimir and want to use the same query syntax with Google Cloud Monitoring data.

For more information about PromQL support in Google Cloud Monitoring, refer to the Google Cloud documentation on PromQL.

Create a PromQL query

To create a PromQL query:

Select PromQL in the Query type dropdown.
Select a project from the Project dropdown.
Enter your PromQL query in the text area.

Configure PromQL options

The following options are available for PromQL queries:

Setting	Description
Min step	Defines the lower bounds on the interval between data points. For example, set this to `1h` to hint that measurements are taken hourly. Supports the `$__interval` and `$__rate_interval` macros.

PromQL query examples

The following examples show common PromQL query patterns for Google Cloud Monitoring:

Query CPU utilization for Compute Engine instances:

compute_googleapis_com:instance_cpu_utilization

Filter by label:

compute_googleapis_com:instance_cpu_utilization{instance_name="my-instance"}

Calculate the rate of a counter metric:

rate(logging_googleapis_com:log_entry_count[5m])

Google Cloud Monitoring template variables

Fri, 03 Apr 2026 19:43:06 +0000

Google Cloud Monitoring template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the Templating and Add and manage variables documentation.

Use query variables

Variables of the type Query help you query Google Cloud Monitoring for various types of data. The Google Cloud Monitoring data source provides the following Query Types:

Name	List returned
Metric Types	Metric type names available for the specified service.
Labels Keys	Keys for `metric label` and `resource label` in the specified metric.
Labels Values	Values for the label in the specified metric.
Resource Types	Resource types for the specified metric.
Aggregations	Aggregations (cross-series reducers) for the specified metric.
Aligners	Aligners (per-series aligners) for the specified metric.
Alignment periods	All alignment periods available in the query editor in Grafana.
Selectors	Selectors for SLO (Service Level Objectives) queries.
SLO Services	Service Monitoring services for SLO queries.
Service Level Objectives (SLO)	SLOs for the specified SLO service.

Use variables in queries

Use Grafana’s variable syntax to include variables in queries. For details, refer to the variable syntax documentation.

Google Cloud Monitoring annotations

Fri, 03 Apr 2026 19:43:06 +0000

Google Cloud Monitoring annotations

Annotations overlay rich event information on top of graphs. You can use annotations to mark important events, deployments, or incidents on your dashboards.

Before you begin

Before you configure annotations, ensure you have the following:

A configured Google Cloud Monitoring data source.
A dashboard where you want to add annotations.

Annotation limitations

Keep the following limitations in mind when using annotations:

Performance: Rendering annotations is expensive. Limit the number of rows returned to maintain dashboard performance.
Native events: There’s no support for displaying Google Cloud Monitoring’s native annotations and events. However, annotations work well with custom metrics in Google Cloud Monitoring.

Add an annotation query

To add an annotation query to a dashboard:

Open the dashboard where you want to add annotations.
Click Dashboard settings (gear icon).
Select Annotations in the left menu.
Click Add annotation query.
Select your Google Cloud Monitoring data source.
Configure the annotation query using the query editor.

Configure the annotation query

With the query editor for annotations, you can select a metric and filters to define which data points create annotations.

The Title and Text fields support templating and can use data returned from the query.

For example, the Title field could have the following text:

{{metric.type}} has value: {{metric.value}}

Example result: monitoring.googleapis.com/uptime_check/http_status has this value: 502

Annotation patterns

Use the following patterns in the Title and Text fields to display metric data in your annotations:

Pattern format	Description	Example	Result
`{{metric.value}}`	Value of the metric/point.	`{{metric.value}}`	`555`
`{{metric.type}}`	Returns the full Metric Type.	`{{metric.type}}`	`compute.googleapis.com/instance/cpu/utilization`
`{{metric.name}}`	Returns the metric name part.	`{{metric.name}}`	`instance/cpu/utilization`
`{{metric.service}}`	Returns the service part.	`{{metric.service}}`	`compute`
`{{metric.label.xxx}}`	Returns the metric label value.	`{{metric.label.instance_name}}`	`grafana-1-prod`
`{{resource.label.xxx}}`	Returns the resource label value.	`{{resource.label.zone}}`	`us-east1-b`

Example: Annotate uptime check failures

To create annotations for uptime check failures:

Add an annotation query using the Google Cloud Monitoring data source.
Select the monitoring.googleapis.com/uptime_check/check_passed metric.
Add a filter for check_passed = false.
Set the Title to: Uptime check failed: {{metric.label.check_id}}
Set the Text to: Region: {{resource.label.zone}}

This creates an annotation marker on your graph each time an uptime check fails.

Google Cloud Monitoring alerting

Fri, 03 Apr 2026 19:43:06 +0000

Google Cloud Monitoring alerting

The Google Cloud Monitoring data source supports Grafana Alerting, allowing you to create alert rules based on GCP metrics and Service Level Objectives (SLOs). You can monitor your Google Cloud environment and receive notifications when specific conditions are met.

Before you begin

Before you create alert rules, ensure the following:

You have appropriate permissions to create alert rules in Grafana.
Your Google Cloud Monitoring data source is configured and working correctly. Refer to Configure the data source.
You’re familiar with Grafana Alerting concepts.

Supported query types for alerting

The following query types support alerting:

Query type	Use case	Notes
Builder	Threshold-based alerts on GCP resource metrics	Best suited for alerting; returns time-series data
MQL	Complex metric queries using Monitoring Query Language	Use for advanced filtering and aggregations
Service Level Objectives (SLO)	Alert on SLO compliance, error budgets, or burn rate	Monitor service reliability
PromQL	Prometheus-style queries on GCP metrics	Familiar syntax for Prometheus users

Note
Alert queries must return numeric data that Grafana can evaluate against a threshold. Queries that return only text or non-numeric data can’t be used directly for alerting.

Authentication requirements

Alerting rules run as background processes without a user context. Both supported authentication methods work with alerting:

Authentication method	Supported
Google JWT File	✓
GCE Default Service Account	✓

Create an alert rule

To create an alert rule using Google Cloud Monitoring data:

Go to Alerting > Alert rules.
Click New alert rule.
Enter a name for your alert rule.
In the Define query and alert condition section:
- Select your Google Cloud Monitoring data source.
- Configure your query (for example, a Builder query for CPU usage or an SLO query for error budget).
- Add a Reduce expression if your query returns multiple series.
- Add a Threshold expression to define the alert condition.
Configure the Set evaluation behavior:
- Select or create a folder and evaluation group.
- Set the evaluation interval (how often the alert is checked).
- Set the pending period (how long the condition must be true before firing).
Add labels and annotations to provide context for notifications.
Click Save rule.

For detailed instructions, refer to Create a Grafana-managed alert rule.

Example: VM CPU usage alert

This example creates an alert that fires when Compute Engine VM CPU utilization exceeds 80%:

Create a new alert rule.
Configure the query:
- Query type: Builder
- Project: Select your GCP project
- Service: Compute Engine
- Metric: instance/cpu/utilization
- Group by function: mean
Add expressions:
- Reduce: Last (to get the most recent data point)
- Threshold: Is above 0.8 (CPU utilization is returned as a decimal)
Set evaluation to run every 1 minute with a 5-minute pending period.
Save the rule.

Example: SLO error budget alert

This example alerts when an SLO’s error budget remaining drops below 20%:

Create a new alert rule.
Configure the query:
- Query type: Service Level Objectives (SLO)
- Project: Select your GCP project
- Service: Select your SLO service
- SLO: Select your SLO
- Selector: SLO Error Budget Remaining
Add expressions:
- Reduce: Last
- Threshold: Is below 0.2 (20% remaining)
Set evaluation to run every 5 minutes.
Save the rule.

Example: Cloud SQL memory alert

This example alerts when Cloud SQL instance memory usage exceeds 90%:

Create a new alert rule.
Configure the query:
- Query type: Builder
- Project: Select your GCP project
- Service: Cloud SQL
- Metric: database/memory/utilization
- Filter: Add a filter for specific database instances if needed
Add expressions:
- Reduce: Last
- Threshold: Is above 0.9
Set evaluation to run every 1 minute.
Save the rule.

Best practices

Follow these recommendations to create reliable and efficient alerts with Google Cloud Monitoring data.

Use appropriate query intervals

Set the alert evaluation interval to be greater than or equal to the minimum data resolution from Google Cloud Monitoring.
Most GCP metrics have 1-minute granularity at minimum.
Avoid very short intervals (less than 1 minute) as they may cause evaluation timeouts or miss data points.

Reduce multiple series

When your query returns multiple time series (for example, CPU usage across multiple VMs), use the Reduce expression to aggregate them:

Last: Use the most recent value
Mean: Average across all series
Max/Min: Use the highest or lowest value
Sum: Total across all series

Use appropriate alignment periods

For alerting queries, ensure the alignment period provides enough data points:

Use “cloud monitoring auto” or “grafana auto” for most cases.
For more precise control, set a fixed alignment period that matches your evaluation interval.

Handle no data conditions

Configure what happens when no data is returned:

In the alert rule, find Configure no data and error handling.
Choose an appropriate action:
- No Data: Keep the alert in its current state
- Alerting: Treat no data as an alert condition
- OK: Treat no data as a healthy state

Test queries before alerting

Always verify your query returns expected data before creating an alert:

Go to Explore.
Select your Google Cloud Monitoring data source.
Run the query you plan to use for alerting.
Confirm the data format and values are correct.
Verify the query returns numeric data suitable for threshold evaluation.

Recording rules

The Google Cloud Monitoring data source supports Grafana-managed recording rules. Recording rules periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.

Use recording rules to:

Reduce query load on Google Cloud Monitoring by pre-computing complex aggregations.
Create derived metrics from GCP data for use in alerts and dashboards.
Import Google Cloud Monitoring data into a Prometheus-compatible database.

Note
Grafana-managed recording rules write results to a Prometheus-compatible database (such as Grafana Mimir or the Grafana Cloud managed Prometheus). You must configure a target data source for storing the recorded metrics.

For instructions on creating recording rules, refer to Create recording rules.

Troubleshooting

If your Google Cloud Monitoring alerts aren’t working as expected, use the following sections to diagnose and resolve common issues.

Alerts not firing

Check that the query returns numeric data in Explore.
Ensure the evaluation interval allows enough time for data to be available.
Verify the threshold is set correctly (remember that many GCP metrics return decimals, not percentages).
Review the alert rule’s health and any error messages in the Alerting UI.

Authentication errors in alert evaluation

If you see authentication errors when alerts evaluate:

Verify the service account has the Monitoring Viewer role.
If using a JWT key file, ensure it hasn’t been deleted or revoked.
Check that the required APIs (Monitoring API, Cloud Resource Manager API) are enabled.

Query timeout errors

Increase the alignment period to reduce the number of data points.
Reduce the time range in the query.
Simplify complex MQL queries.
Add filters to narrow the result set.

For additional troubleshooting help, refer to Troubleshoot Google Cloud Monitoring.

Additional resources

Troubleshoot Google Cloud Monitoring data source issues

Fri, 03 Apr 2026 19:43:06 +0000

Troubleshoot Google Cloud Monitoring data source issues

This document provides solutions to common issues you may encounter when configuring or using the Google Cloud Monitoring data source. For configuration instructions, refer to Configure Google Cloud Monitoring.

Authentication errors

These errors occur when GCP credentials are invalid, missing, or don’t have the required permissions.

“Permission denied” or “Access denied”

Symptoms:

Save & test fails with permission errors
Queries return authorization errors
Projects, metrics, or labels don’t load

Possible causes and solutions:

Cause	Solution
Service account missing required permissions	Assign the Monitoring Viewer role to the service account in the GCP Console under IAM & Admin > IAM. Refer to Configure Google Cloud Monitoring for details.
Incorrect service account key file	Verify the JSON key file was downloaded correctly and contains valid credentials. Generate a new key if necessary.
Service account key has been deleted	Check the service account in GCP Console under IAM & Admin > Service Accounts. If the key was deleted, create a new one.
Wrong project selected	Verify the default project in the data source configuration matches a project the service account has access to.
APIs not enabled	Enable the Monitoring API and Cloud Resource Manager API in the GCP Console. Refer to Configure Google Cloud Monitoring for links.

“Invalid JWT” or “JWT token error”

Symptoms:

Authentication fails when using Google JWT File
Error message references invalid or malformed JWT

Solutions:

Verify you uploaded the complete JSON key file, not just the private key portion.
Check that the JSON file is properly formatted and not corrupted.
Ensure the key file contains all required fields: type, project_id, private_key_id, private_key, client_email, client_id, auth_uri, token_uri.
Generate a new service account key and re-upload it.

GCE Default Service Account not working

Symptoms:

Data source test fails when using GCE Default Service Account
Works with JWT but fails with GCE authentication

Solutions:

Verify Grafana is running on a Google Compute Engine (GCE) virtual machine.
Check that the GCE instance has the Cloud Monitoring API scope enabled.
Verify the GCE default service account has the Monitoring Viewer role.
If the VM was created without the required scope, you may need to stop the instance, edit it to add the scope, and restart.

Service account impersonation errors

Symptoms:

Authentication fails when service account impersonation is enabled
Error: “Unable to impersonate service account”

Solutions:

Verify the primary service account has the roles/iam.serviceAccountTokenCreator role on the target service account.
Check that the target service account email is entered correctly.
Ensure the target service account has the Monitoring Viewer role.
Verify both service accounts are in projects that have the required APIs enabled.

Connection errors

These errors occur when Grafana cannot reach Google Cloud Monitoring endpoints.

“Request timed out” or connection failures

Symptoms:

Data source test times out
Queries fail with timeout errors
Intermittent connection issues

Solutions:

Verify network connectivity from the Grafana server to Google Cloud endpoints (monitoring.googleapis.com).
Check firewall rules allow outbound HTTPS (port 443) to Google Cloud services.
For Grafana Cloud connecting to private resources, configure Private data source connect.
Check if a corporate proxy is blocking connections to Google Cloud.

“SSL certificate problem”

Symptoms:

SSL/TLS handshake errors
Certificate verification failures

Solutions:

Ensure the system time is correct on the Grafana server.
Verify the Grafana server has up-to-date CA certificates installed.
Check if a corporate proxy is intercepting HTTPS traffic.

Metrics query errors

These errors occur when querying Google Cloud Monitoring metrics.

“No data” or empty results

Symptoms:

Query executes without error but returns no data
Charts show “No data” message

Possible causes and solutions:

Cause	Solution
Time range doesn’t contain data	Expand the dashboard time range. GCP metrics have different retention periods based on the metric type.
Wrong project selected	Verify you’ve selected the correct project in the query editor.
Incorrect metric type	Verify the service, metric type, and metric are correct. Check available metrics in the GCP Console Metrics Explorer.
Missing labels/filters	Some metrics require specific label filters to return data. Try removing filters to see if data appears.
Resource not emitting metrics	Verify the resource exists and is actively emitting metrics. Some metrics only populate under certain conditions.

Metrics don’t appear in drop-down

Symptoms:

Expected metrics don’t appear in the query editor
Metric drop-down is empty for a service

Solutions:

Verify the metric exists in the selected project and region.
Check that the service account has the Monitoring Viewer role.
Some metrics are only available for specific resource types. Check the Google Cloud metrics list.
Use the Query Inspector to verify the API request and response.

Label values not loading

Symptoms:

Label value drop-down doesn’t populate
Filters can’t be applied

Solutions:

Verify the service account has the Monitoring Viewer role.
Ensure a project, service, and metric are selected before label values can load.
Label values are populated from existing metric data. If no metrics match the current selection, no values appear.

MQL query errors

Symptoms:

Monitoring Query Language (MQL) queries fail with syntax errors
MQL query returns unexpected results

Solutions:

Validate your MQL syntax using the MQL reference documentation.
Check for typos in metric types, label names, or function names.
Ensure time range syntax is valid in your query.
Test the query in the GCP Console Metrics Explorer before using it in Grafana.

“Too many data points” or API throttling

Symptoms:

Queries fail with quota errors
Performance degrades with multiple panels

Solutions:

Increase the alignment period to reduce the number of data points.
Reduce the time range of your queries.
Use fewer metric queries per panel.
Request a quota increase in the GCP Console under APIs & Services > Quotas.
Enable query caching in Grafana to reduce API calls.

SLO query errors

These errors are specific to Service Level Objective (SLO) queries.

SLO services don’t appear

Symptoms:

SLO service selector is empty
Can’t find expected SLO services

Solutions:

Verify SLOs are defined in Google Cloud Monitoring for the selected project.
Check that the service account has access to view SLOs.
Ensure the project has services configured in the Service Monitoring section of the GCP Console.

SLO query returns no data

Symptoms:

SLO query executes but returns no data
SLO values show as empty

Solutions:

Verify the SLO exists and is active in the GCP Console.
Check that the time range includes periods when the SLO had data.
Ensure the selected SLO selector (SLI value, compliance, error budget, etc.) is appropriate for the SLO type.
Some SLOs may not have data if the underlying service hasn’t received traffic.

Template variable errors

These errors occur when using template variables with the Google Cloud Monitoring data source.

Variables return no values

Symptoms:

Variable drop-down is empty
Dashboard fails to load with variable errors

Solutions:

Verify the data source connection is working.
Check that the service account has permissions to list the requested resources.
For dependent variables, ensure parent variables have valid selections.
Verify the project is selected correctly in the variable query.

Variables are slow to load

Symptoms:

Dashboard takes a long time to load
Variable selectors are slow to populate

Solutions:

Set variable refresh to On dashboard load instead of On time range change.
Reduce the scope of variable queries (filter by specific project or service).
Limit the number of dependent variables in a chain.

For more information on template variables, refer to the template variables documentation.

Pre-configured dashboard issues

These issues occur with the bundled pre-configured dashboards.

Imported dashboards show no data

Symptoms:

Imported dashboards show empty panels
Template variables don’t load

Solutions:

Verify the data source name in the dashboard matches your Google Cloud Monitoring data source.
Check that the service account has access to the projects shown in the project variable.
Ensure the resources (Compute Engine instances, Cloud SQL, etc.) exist and are emitting metrics.
Verify the required GCP services are enabled in your project.

Enable debug logging

To capture detailed error information for troubleshooting:

Set the Grafana log level to debug in the configuration file:
ini
```
[log]
level = debug
```
Review logs in /var/log/grafana/grafana.log (or your configured log location).
Look for Google Cloud Monitoring-specific entries that include request and response details.
Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried the solutions above and still encounter issues:

Check the Grafana community forums for similar issues.
Review Google Cloud Monitoring issues on GitHub for known bugs.
Consult the Google Cloud Monitoring documentation for service-specific guidance.
Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Contracted user.
When reporting issues, include:
- Grafana version
- GCP project (redact if sensitive)
- Error messages (redact sensitive information)
- Steps to reproduce
- Query configuration (redact credentials)