Amazon CloudWatch data source on Grafana Labs

Configure CloudWatch

Fri, 03 Apr 2026 12:35:46 -0500

Configure the Amazon CloudWatch data source

This document provides instructions for configuring the Amazon CloudWatch data source and explains available configuration options. For general information on adding and managing data sources, refer to Data source management.

Before you begin

You must have the Organization administrator role to configure the CloudWatch data source. Organization administrators can also configure the data source via YAML with the Grafana provisioning system.
Grafana comes with a built-in CloudWatch data source plugin, so you do not need to install a plugin.
Familiarize yourself with your CloudWatch security configuration and gather any necessary security certificates, client certificates, and client keys.

Add the CloudWatch data source

Complete the following steps to set up a new CloudWatch data source:

Click Connections in the left-side menu.
Click Add new connection
Type CloudWatch in the search bar.
Select the CloudWatch data source.
Click Add new data source in the upper right.

Grafana takes you to the Settings tab, where you will set up your CloudWatch configuration.

Configure the data source in the UI

The following are configuration options for the CloudWatch data source.

Setting	Description
Name	The data source name. Sets the name you use to refer to the data source in panels and queries.
Default	Toggle to select as the default name in dashboard panels. When you go to a dashboard panel, this will be the default selected data source.

Grafana plugin requests to AWS are made on behalf of an AWS Identity and Access Management (IAM) role or IAM user. The IAM user or IAM role must have the associated policies to perform certain API actions.

For authentication options and configuration details, refer to AWS authentication.

Setting	Description
Authentication	Specify which AWS credentials chain to use. A Grafana plugin’s requests to AWS are made on behalf of an IAM role or IAM user. The IAM user or IAM role must have the necessary policies to perform the required API actions.

Access & secret key:

You must use both an access key ID and a secret access key to authenticate.

Setting	Description
Access Key ID	Enter your key ID.
Secret Access Key	Enter the secret access key.

Assume Role:

Setting	Description
Assume Role ARN	Optional. Specify the ARN of an IAM role to assume. This ensures the selected authentication method is used to assume the role, not used directly.
External ID	If you’re assuming a role in another AWS account that requires an external ID, specify it here.

Additional Settings:

Setting	Description
Endpoint	Optional. Specify a custom endpoint for the AWS service.
Default Region	Specify the AWS region. Example: If the region is US West (Oregon), use `us-west-2`.
Namespaces of Custom Metrics	Add one or more custom metric namespaces, separated by commas (for example,`Namespace1,Namespace2`). Grafana can’t automatically load custom namespaces using the CloudWatch GetMetricData API. To make custom metrics available in the query editor, manually specify the namespaces in the `Namespaces of Custom Metrics` field in the data source configuration.

CloudWatch Logs:

Setting	Description
Query timeout result	Grafana polls CloudWatch Logs every second until AWS returns a `Done` status or the timeout is reached. An error is returned if the timeout is exceeded. For alerting, the timeout defined in the Grafana configuration file takes precedence. Enter a valid duration string, such as `30m`, `30s` or `200ms`. The default is `30m`.
Default Log Groups	Optional. Specify the default log groups for CloudWatch Logs queries.

X-Ray trace link:

Setting	Description
Data source	Select the X-Ray data source from the drop-down menu.

Grafana automatically creates a link to a trace in X-Ray data source if logs contain the @xrayTraceId field. To use this feature, you must already have an X-Ray data source configured. For details, see the X-Ray data source docs. To view the X-Ray link, select the log row in either the Explore view or dashboard Logs panel to view the log details section.

To log the @xrayTraceId, refer to the AWS X-Ray documentation. To provide the field to Grafana, your log queries must also contain the @xrayTraceId field, for example by using the query fields @message, @xrayTraceId.

Private data source connect - Only for Grafana Cloud users.

Setting	Description
Private data source connect	Establishes a private, secured connection between a Grafana Cloud stack and data sources within a private network. Use the drop-down to locate the PDC URL. For setup instructions, refer to Private data source connect (PDC) and Configure PDC. Click Manage private data source connect to open your PDC connection page and view your configuration details.

After configuring your Amazon CloudWatch data source options, click Save & test at the bottom to test the connection. You should see a confirmation dialog box that says:

Note
To troubleshoot issues while setting up the CloudWatch data source, check the /var/log/grafana/grafana.log file. Common issues include invalid credentials, missing regions and metrics-only credentials.

IAM policy examples

To read CloudWatch metrics and EC2 tags, instances, regions, and alarms, you must grant Grafana permissions via IAM. You can attach these permissions to the IAM role or IAM user you configured in AWS authentication.

Metrics-only permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    }
  ]
}

Logs-only permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Metrics and logs permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Cross-account observability permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Note
Cross-account observability lets you retrieve metrics and logs across different accounts in a single region, but you can’t query EC2 Instance Attributes across accounts because those come from the EC2 API and not the CloudWatch API.

For more information on configuring authentication, refer to Configure AWS authentication.

CloudWatch Logs data protection

CloudWatch Logs can protect data by applying log group data protection policies. When data protection is enabled for a log group, any sensitive data that matches the identifiers you select is automatically masked. To view masked data, your IAM role or user must have the logs:Unmask permission. For more details, refer to the AWS guide on masking sensitive log data.

Configure the data source with grafana.ini

The Grafana configuration file includes an AWS section where you can configure data source options:

Configuration option	Description
`allowed_auth_providers`	Specifies which authentication providers are allowed for the CloudWatch data source. The following providers are enabled by default in open source Grafana: `default` (AWS SDK default), `keys` (Access and secret key), `credentials` (Credentials file), `ec2_IAM_role` (EC2 IAM role).
`assume_role_enabled`	Allows you to disable `assume role (ARN)` in the CloudWatch data source. The assume role (ARN) is enabled by default in open source Grafana.
`list_metrics_page_limit`	Sets the limit of List Metrics API pages. When a custom namespace is specified in the query editor, the List Metrics API populates the Metrics field and Dimension fields. The API is paginated and returns up to 500 results per page, and the data source also limits the number of pages to 500 by default. This setting customizes that limit.

Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning and available configuration options, refer to Provision Grafana.

Using AWS SDK (default):

apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      defaultRegion: eu-west-2

Using credentials’ profile name (non-default):

apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: credentials
      defaultRegion: eu-west-2
      customMetricsNamespaces: 'CWAgent,CustomNameSpace'
      profile: secondary

Using accessKey and secretKey:

apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: keys
      defaultRegion: eu-west-2
    secureJsonData:
      accessKey: '<your access key>'
      secretKey: '<your secret key>'

Using AWS SDK Default and ARN of IAM Role to Assume:

apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      assumeRoleArn: arn:aws:iam::123456789012:root
      defaultRegion: eu-west-2

Amazon CloudWatch query editor

Fri, 03 Apr 2026 12:35:46 -0500

Amazon CloudWatch query editor

Grafana provides a query editor for the CloudWatch data source, which allows you to query, visualize, and alert on logs and metrics stored in Amazon CloudWatch. It is located on the Explore page. For general documentation on querying data sources in Grafana, refer to Query and transform data.

Choose a query editing mode

The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor.

Select the API to query using the drop-down to the right of the Region setting.

CloudWatch Metrics query editor components

The following are the components of the CloudWatch query editor.

Setting	Description
Region	Select an AWS region if it differs from the default.
Namespace	The AWS service namespace. Examples: `AWS/EC2`, `AWS_Lambda`.
Metric name	The name of the metric you want to visualize. Example: `CPUUtilization`.
Statistic	Choose how to aggregate your data. Examples: `sum`, `average`, `maximum`.
Dimensions	Select dimensions from the drop-down. Examples: `InstanceId`, `FunctionName`, `latency`. You can add several dimensions to your query.
Match exact	Optional. When enabled, this option restricts query results to metrics that precisely match the specified dimensions and their values. All dimensions of the queried metric must be explicitly defined in the query to ensure an exact schema match. If disabled, the query will also return metrics that match the defined schema but possess additional dimensions.
ID	Optional. Unique identifier required by the GetMetricData API for referencing queries in math expressions. Must start with a lowercase letter and can include letters, numbers, and underscores. If not specified, Grafana generates an ID using the pattern `query[refId]` (for example, `queryA`for the first query row).
Period	The minimum time interval, in seconds, between data points. The default is `auto`. Valid values are 1, 5, 10, 30, or any multiple of 60. When set to auto or left blank, Grafana calculates the period using time range in seconds / 2000, then rounds up to the next value (60, 300, 900, 3600, 21600, 86400) based on the CloudWatch retention policy.
Label	Optional. Add a customized time series legend name. The label field overrides the default metric legend name using CloudWatch dynamic labels. Time-based dynamic labels like ${MIN_MAX_TIME_RANGE} derive legend values from the current timezone in the time range picker. For the full list of label patterns and limitations, refer to CloudWatch dynamic labels.

Use Builder mode

Create a query in Builder mode:

Browse and select a metric namespace, metric name, filter, group, and order options using information from the Metrics Insights keywords table.
For each of these keywords, choose from the list of available options.

Grafana constructs a SQL query based on your selections.

Use Code mode

You can also write your SQL query directly in a code editor by using Code mode.

The code editor includes a built-in autocomplete feature that suggests keywords, aggregations, namespaces, metrics, labels, and label values. Suggestions appear after typing a space, comma, or dollar ($) character, or by pressing CTRL+Space.

Note
Template variables in the code editor can interfere with autocompletion.

To run the query, click Run query above the code editor.

Query CloudWatch metrics

You can create two types of queries in the CloudWatch query editor:

Metric Search, which help you retrieve and filter available metrics.
Metric Query, which use the Metrics Insights feature to fetch time series data.

The query type you use depends on how you want to interact with AWS metrics. Use the drop-down in the upper middle of the query editor to select which type you want to create.

Metric Search queries

Metric search queries help you discover and filter available metrics. These queries use wildcards and filters to find metrics without needing to know exact metric names.

A valid metric query requires a specified namespace, metric name, and at least one statistic. Dimensions are optional, but if included, you must provide both a key and a value.

The Match Exact option controls how dimension filtering is applied to metric queries. When you enable match exact, the query returns only metrics whose dimensions precisely match the specified criteria.

This requires the following:

All dimensions present on the target metric must be explicitly specified.
Dimensions you don’t want to filter by must use a wildcard (*) filter.
The metric schema must match exactly as defined in the CloudWatch metric schema documentation.

When Match Exact is disabled, you can specify any subset of dimensions for filtering. The query returns metrics that:

Match the specified namespace and metric name.
Match all defined dimension filters.
May contain additional dimensions beyond those specified.

This mode provides more flexible querying but may return metrics with unexpected additional dimensions.

The data source returns up to 100 metrics matching your filter criteria.

Enhance metric queries using template variables to create dynamic, reusable dashboards.

Create dynamic queries with dimension wildcards

Use the asterisk (*) wildcard for dimension values to create dynamic queries that automatically monitor changing sets of AWS resources.

The query returns the average CPU utilization for all EC2 instances in the default region. With Match Exact disabled and InstanceId using a wildcard, the query retrieves metrics for any EC2 instance regardless of additional dimensions.

Auto-scaling events add new instances to the graph without manual instance ID tracking. This feature supports up to 100 metrics.

Click the Query inspector button and select Meta Data to see the search expression that’s automatically built to support wildcards.

To learn more about search expressions, refer to the CloudWatch documentation. The search expression is defined by default in such a way that the queried metrics must match the defined dimension names exactly. This means that in the example, the query returns only metrics with exactly one dimension containing the name InstanceId.

Disabling Match Exact includes metrics with additional dimensions and creates a search expression even without wildcards. Grafana searches for any metric matching at least the namespace, metric name, and all defined dimensions.

Use multi-value template variables

When defining dimension values based on multi-valued template variables, the data source uses a search expression to query for the matching metrics. This enables the use of multiple template variables in one query, and also lets you use template variables for queries that have the Match Exact option disabled.

Search expressions are limited to 1,024 characters, so your query might fail if you have a long list of values. We recommend using the asterisk (*) wildcard instead of the All option to query all metrics that have any value for a certain dimension name.

Multi-valued template variables are supported only for dimension values. Using multi-valued template variables for Region, Namespace, or Metric Name is not supported.

Use metric math expressions

Create new time series metrics using mathematical functions on CloudWatch metrics. This supports arithmetic operators, unary subtraction, and other functions. For available functions, refer to AWS Metric Math.

To apply arithmetic operations, assign a unique string ID to the raw metric, then reference this ID in the Expression field of the new metric.

Note
If you use the expression field to reference another query, such as queryA * 2, you can’t create an alert rule based on that query.

Query metrics across AWS monitoring accounts

When you select Builder mode within the Metric search editor, a new Account field is displayed. Use the Account field to specify which of the linked monitoring accounts to target for the given query. By default, the All option is specified, which will target all linked accounts.

While in Code mode, you can specify any math expression. If the Monitoring account badge displays in the query editor header, all SEARCH expressions entered in this field will be cross-account by default and can query metrics from linked accounts. Note that while queries run cross-account, the autocomplete feature currently doesn’t fetch cross-account resources, so you’ll need to manually specify resource names when writing cross-account queries. You can limit the search to one or a set of accounts, as documented in the AWS documentation.

Period macro

If you use a CloudWatch SEARCH expression, consider using the $__period_auto macro rather than specifying a period explicitly. The $__period_auto macro will resolve to a CloudWatch period that is suitable for the chosen time range.

Deep-link Grafana panels to the CloudWatch console

Left-clicking a time series in the panel displays a context menu with a link to View in CloudWatch console. Clicking the link opens a new tab that takes you to the CloudWatch console and displays all metrics for that query. If you’re not logged in to the CloudWatch console, the link forwards you to the login page. The link provided is valid for any account but displays the expected metrics only if you’re logged in to the account that corresponds to the selected data source in Grafana.

This feature is not available for metrics based on metric math expressions.

Use Metric Insights syntax

Metric Insights uses a dialect of SQL and this query syntax:

SELECT FUNCTION(MetricName)
FROM Namespace | SCHEMA(...)
[ WHERE labelKey OPERATOR labelValue [AND|...]]
[ GROUP BY labelKey [, ...]]
[ ORDER BY FUNCTION() [DESC | ASC] ]
[ LIMIT number]

For details about the Metrics Insights syntax, refer to the AWS reference documentation.

For information about Metrics Insights limits, refer to the AWS feature documentation.

You can also augment queries by using template variables.

Use Metrics Insights keywords

This table summarizes common Metrics Insights query keywords:

Keyword	Description
`FUNCTION`	Required. Specifies the aggregate function to use, and also specifies the name of the metric to be queried. Valid values are AVG, COUNT, MAX, MIN, and SUM.
`MetricName`	Required. For example, `CPUUtilization`.
`FROM`	Required. Specifies the metric’s source. You can specify either the metric namespace that contains the metric to be queried, or a SCHEMA table function. Namespace examples include `AWS/EC2`, `AWS/Lambda`.
`SCHEMA`	Optional. Narrows the query results to only the metrics that are an exact match, or to metrics that do not match.
`WHERE`	Optional. Filters the query results to only the metrics that match your specified expression. For example, `WHERE InstanceType != 'c3.4xlarge'`.
`GROUP BY`	Optional. Groups the query results into multiple time series. For example, `GROUP BY ServiceName`.
`ORDER BY`	Optional. Specifies the order in which time series are returned. Options are `ASC`, `DESC`.
`LIMIT`	Optional. Limits the number of time series returned.

Query CloudWatch Logs

The logs query editor helps you write CloudWatch Logs queries across a selected region and set of log groups.

You can query CloudWatch Logs using three supported query language options:

Logs Insights QL - The AWS native query language specifically designed for CloudWatch Logs. It uses a SQL-like syntax with commands like fields, filter, stats, and sort. It’s optimized for the CloudWatch log structure and offers built-in functions for parsing timestamps, extracting fields from JSON logs, and performing aggregations.
OpenSearch PPL - The OpenSearch query language is based on Elasticsearch’s query DSL (Domain Specific Language). It uses a pipe-based syntax similar to Unix command-line tools or the Splunk search language, and supports complex boolean logic, range queries, wildcard matching, and full-text search capabilities.
OpenSearch SQL - OpenSearch SQL is a query language that uses a SQL-like syntax for querying data in OpenSearch. It supports standard SQL queries and is designed for users familiar with SQL.

Create a CloudWatch Logs query:

Select a region.
Select CloudWatch Logs from the query type drop-down.
Use the Logs Mode selector to choose between Logs Insights and Log Anomalies queries.

Log Anomalies

Anomaly detection uses machine-learning and pattern recognition to establish baselines of typical log content.
The Log Anomalies query editor fetches the list of anomalies detected in your CloudWatch service. In order to query log anomalies in the editor, a log anomaly detector must be created in the AWS CloudWatch console first. The log trend cell shows the number of occurrences of the pattern over the selected query time range. The table shows 50 log anomalies at a time. If you would like to narrow down the list, you can filter anomalies by their ARN and suppressed state.

In addition to this, you can use the Logs Insights QL editor and the anomaly command together with the patterns command to define and display log anomalies in real time. See the CloudWatch Logs Insights documentation for more info.

Logs Insights

The Query language drop-down is available only when Logs Mode is set to Logs Insights.

Select the query language you want to use in the Query language drop-down.
Specify the log groups you want to query.
Use the main input area to write your logs query. Amazon CloudWatch only supports a subset of OpenSearch SQL and PPL commands. To find out more about the syntax supported, consult Amazon CloudWatch Logs documentation

Specify log groups to query

You must specify a set of log groups to target for your query using one of the following methods:

Log group name: Select specific log groups. This is the default behavior. You can select up to 50 log groups. Click Select log groups to choose the target log groups for your query. When the Monitoring account badge appears in the query editor header, you can search and select log groups across multiple accounts. Use the Accounts filter to narrow results by account. For large lists, use prefix search to narrow the selection.
Name prefix: Select the Name prefix query scope and provide up to five log group name prefixes. The query will run against log groups that have names that start with the specified prefixes. Each prefix must be at least three characters and must not include *. This option is only available for Logs Insights QL.
All log groups: Select the All log groups query scope. This queries all log groups in the selected region. This option is only available for Logs Insights QL.

You can also apply optional filters when specifying log groups with the Name prefix or All log groups query scopes:

Class: Filter by log group class. Choose Standard or Infrequent Access.
Accounts: Filter by AWS account. This option is only available when the Monitoring account badge appears in the query editor header. You can select up to 20 accounts.

Note
You must specify the region and log groups when querying with Logs Insights QL, OpenSearch PPL, and OpenSearch SQL. In OpenSearch SQL, you can specify log groups in multiple ways. For details, refer to the Query log groups with OpenSearch SQL section.

Click View in CloudWatch console to interactively view, search, and analyze your log data in the CloudWatch Logs Insights console. If you’re not logged in to the CloudWatch console, the link forwards you to the login page.

Query log groups with OpenSearch SQL

When querying log groups with OpenSearch SQL, you can use the $__logGroups macro to automatically reference log groups selected in the query editor’s log group selector. This is the recommended approach as it allows you to manage log groups through the UI.

SELECT window.start, COUNT(*) AS exceptionCount
FROM `$__logGroups`
WHERE `@message` LIKE '%Exception%'

The $__logGroups macro expands to the proper logGroups(logGroupIdentifier: [...]) syntax with the log groups you’ve selected in the UI.

Alternatively, you can manually specify a single log group directly in the FROM clause:

SELECT window.start, COUNT(*) AS exceptionCount
FROM `log_group`
WHERE `@message` LIKE '%Exception%'

When querying multiple log groups you must use the logGroups(logGroupIdentifier: [...]) syntax:

SELECT window.start, COUNT(*) AS exceptionCount
FROM `logGroups( logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`
WHERE `@message` LIKE '%Exception%'

To reference log groups in a monitoring account, use ARNs instead of LogGroup names.

You can also write queries returning time series data by using the stats command. When making stats queries in Explore, ensure you are in Metrics Explore mode.

Create queries for alerting

Alerting requires queries that return numeric data, which CloudWatch Logs supports. For example, you can enable alerts through the use of the stats command.

The following is a valid query for alerting on messages that include the text “Exception”:

filter @message like /Exception/
    | stats count(*) as exceptionCount by bin(1h)
    | sort exceptionCount desc

Note
If you receive an error like input data must be a wide series but got ... when trying to alert on a query, make sure that your query returns valid numeric data that can be output to a Time series panel.

For more information on Grafana alerts, refer to Alerting.

Cross-account observability

The CloudWatch plugin monitors and troubleshoots applications that span multiple accounts within a region. Cross-account observability enables seamless searching, visualization, and analysis of metrics and logs across account boundaries.

To enable cross-account observability, complete the following steps:

Go to the Amazon CloudWatch documentation and follow the instructions for enabling cross-account observability.
Add two API actions to the IAM policy attached to the role/user running the plugin.

Cross-account querying is available in the plugin through the Logs, Metric search, and Metric Insights modes. After you have configured it, you’ll see a Monitoring account badge in the query editor header.

Query caching

When you enable query and resource caching, Grafana temporarily stores the results of data source queries and resource requests. Query caching is available in CloudWatch Metrics in Grafana Cloud and Grafana Enterprise. It is not available in CloudWatch Logs Insights due to how query results are polled from AWS.

CloudWatch template variables

Fri, 03 Apr 2026 12:35:46 -0500

CloudWatch template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard, and they are called template variables

For an introduction to templating and template variables, refer to Templating and Add and manage variables.

Use query variables

You can specify these CloudWatch data source queries in the Variable edit view’s Query Type field. Use them to fill a variable’s options list with values like regions, namespaces, metric names, and dimension keys/values.

Name	List returned
Regions	All AWS regions.
Namespaces	All namespaces CloudWatch supports.
Metrics	Metrics in the namespace. (Specify region or use “default” for custom metrics.)
Dimension Keys	Dimension keys in the namespace.
Dimension Values	Dimension values matching the specified `region`, `namespace`, `metric`, and `dimension_key`. Use dimension `filters` for more specific results.
EBS Volume IDs	Volume ids matching the specified `region` and `instance_id`.
EC2 Instance Attributes	Attributes matching the specified `region`, `attribute_name`, and `filters`.
Resource ARNs	ARNs matching the specified `region`, `resource_type`, and `tags`.
Statistics	All standard statistics.
LogGroups	All log groups matching the specified `region`.

For details on the available dimensions, refer to the CloudWatch Metrics and Dimensions Reference.

For details about the metrics CloudWatch provides, refer to the CloudWatch documentation.

Use variables in queries

Use the Grafana variable syntax to include variables in queries. A query variable in dynamically retrieves values from your data source using a query. For details, refer to the variable syntax documentation.

Use ec2_instance_attribute

The ec2_instance_attribute function in template variables allows Grafana to retrieve certain instance metadata from the EC2 metadata service, including Instance ID and region.

Filters

The ec2_instance_attribute query takes a filters parameter, where each key is a filter name (such as a tag or instance property), and each value is a comma-separated list of matching values.

You can specify pre-defined filters of ec2:DescribeInstances.

Select attributes

A query returns only one attribute per instance. You can select any attribute that has a single value and isn’t an object or array, also known as a flat attribute:

AmiLaunchIndex
Architecture
ClientToken
EbsOptimized
EnaSupport
Hypervisor
IamInstanceProfile
ImageId
InstanceId
InstanceLifecycle
InstanceType
KernelId
KeyName
LaunchTime
Platform
PrivateDnsName
PrivateIpAddress
PublicDnsName
PublicIpAddress
RamdiskId
RootDeviceName
RootDeviceType
SourceDestCheck
SpotInstanceRequestId
SriovNetSupport
SubnetId
VirtualizationType
VpcId

You can select tags by prepending the tag name with Tags.. For example, select the tag Name by using Tags.Name.

Configure AWS authentication

Fri, 03 Apr 2026 12:35:46 -0500

Configure AWS authentication

Grafana data source plugins make requests to AWS on behalf of an AWS Identity and Access Management (IAM) role or IAM user. The IAM user or IAM role must have the associated policies to perform certain API actions for querying data source data. Since these policies are specific to each data source, refer to individual data source plugin documentation for details.

The Grafana backend performs all AWS API requests server-side using the official AWS SDK.

This document explores the following topics:

Select an authentication method

Available authentication methods depend on your configuration and the environment where Grafana runs.

Open source Grafana enables the AWS SDK Default, Credentials file, and Access and secret key methods by default. Cloud Grafana enables only Access and secret key by default. Users with server configuration access can enable or disable specific auth providers as needed. For more information, refer to the allowed_auth_providers documentation.

AWS SDK Default uses the default provider from the AWS SDK for Go without custom configuration. This method requires configuring AWS credentials outside Grafana through the CLI, or by attaching credentials directly to an EC2 instance, in an ECS task, or for a Service Account in a Kubernetes cluster. You can attach permissions directly to the data source with AWS SDK Default or combine it with the optional Assume Role ARN field.
Credentials file maps to the SharedCredentialsProvider provider in the AWS SDK for Go. This method reads the AWS shared credentials file for a specified profile. Unlike AWS SDK Default which also reads the shared credentials file, this option lets you specify a profile directly without environment variables. This option provides no fallback to other credential providers and fails if the file credentials are invalid.
Access and secret key corresponds to the StaticProvider and authenticates using a specified access key ID and secret key pair. This method doesn’t provide fallback authentication and fails if the key pair is invalid. Grafana Cloud uses this as the primary authentication method.
Grafana Assume Role - With this authentication method, Grafana Cloud users create an AWS IAM role that has a trust relationship with Grafana’s AWS account. Grafana uses STS to generate temporary credentials on its behalf. This method eliminates the need to generate secret and access keys for users Refer to Use Grafana Assume Role for more information.
Workspace IAM role corresponds to the EC2RoleProvider. The EC2RoleProvider retrieves credentials from a role attached to the EC2 instance running Grafana. While AWS SDK Default can achieve similar results, this option provides no fallback authentication. Amazon Managed Grafana enables this option by default.

Assume a role

Specify an IAM role to assume in the Assume Role ARN field.

When you configure Assume Role ARN, Grafana uses the provided credentials to perform an sts:AssumeRole call. The primary authentication method only needs permission to assume the role, while the assumed role requires CloudWatch access permissions.

For example, you might use one set of long-term credentials across all AWS data sources but want to limit each data source’s AWS access (such as separating staging and production data access). You could create separate credentials for each data source with distinct permissions, but this approach requires managing and rotating multiple secret and access keys across many data source instances.

Instead, assume role functionality lets you use one set of AWS credentials across all AWS data sources with a single permission: the ability to assume roles through STS. You then create separate IAM roles for each data source that specify temporary permissions. Since IAM roles are not credentials, they require no rotation and simplify management.

If the Assume Role ARN field is left empty, Grafana uses the provided credentials from the selected authentication method directly, and permissions to AWS data must be attached directly to those credentials. The Assume Role ARN field is optional for all authentication methods except for Grafana Assume Role.

To disable this feature in open source Grafana or Grafana Enterprise, refer to assume_role_enabled.

Use an external ID

Note
You cannot use an external ID for the Grafana Assume Role authentication provider.

To assume a role in another account created with an external ID, specify the external ID in the External ID field.

For more information, refer to the AWS documentation on external ID.

Use a custom endpoint

Use the Endpoint field to specify a custom endpoint URL that overrides the default AWS service API endpoint. Leave this field blank to use the default generated endpoint.

For more information about using service endpoints, refer to the AWS service endpoints documentation.

Use an AWS credentials file

Create a file at ~/.aws/credentials, the HOME path for the user running the grafana-server service.

Note
If the credentials file appears to be in the correct location but isn’t working, move your aws file to /usr/share/grafana/ and set the credentials file permissions to 0644.

Credentials file example:

[default]
aws_access_key_id = asdsadasdasdasd
aws_secret_access_key = dasdasdsadasdasdasdsa
region = us-west-2

Use EKS IAM roles for service accounts

EKS IAM roles for service accounts (IRSA) are an AWS EKS feature that allows pods to assume IAM roles without storing long-term credentials. When you configure IRSA in your EKS cluster, AWS injects temporary credentials into your pod as projected volume mounts.

In Grafana containers, the process runs as user 472 (“grafana”). By default, Kubernetes mounts the projected credentials with permissions for the root user only.

To grant user 472 permission to access the credentials, and prevent fallback to the IAM role attached to the EC2 instance, set a security context for your pod.

Security context example:

securityContext:
  fsGroup: 472
  runAsUser: 472
  runAsGroup: 472

Use Grafana Assume Role

Note
Grafana Assume Role is only available in Grafana Cloud for Amazon CloudWatch and Athena data sources.

The Grafana Assume Role authentication provider lets you access AWS without creating or managing long-term AWS IAM users or rotating access keys. Instead, you can create an IAM role that has permissions to access CloudWatch and trusts a Grafana AWS account.

The Grafana AWS account then makes a Security Token Service (STS) request to generate temporary credentials for your AWS data. This request includes an externalID unique to your Grafana Cloud account, which ensures users can access only their own AWS resources.

For more information, refer to the AWS documentation on external ID.

To use the Grafana Assume Role:

Create a new CloudWatch data source (or update an existing one) and select Grafana Assume Role as an authentication provider.
In the AWS Console, create a new IAM role, and under Trusted entity type, select Another AWS account as the trusted Entity.
Enter the Grafana account id (displayed in the instructions box on the Settings tab of the CloudWatch data source configuration) and check the Require external ID box.
Enter the external ID specified in the instructions box on the Settings tab of the CloudWatch data source configuration in Grafana. This external ID will be unique to your Grafana instance.
Attach any required permissions you would like Grafana to be able to access on your behalf (for example, CloudWatch Logs and CloudWatch Metrics policies).
Give the role a name and description, and click Create role.
Copy the ARN of the role you just created and paste it into the Assume Role ARN field on the Settings tab of CloudWatch data source configuration in Grafana.

Sample Trust Relationship for an IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": {Grafana's AWS Account}
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": {External ID unique to your account}
                }
            }
        }
    ]
}

Troubleshoot Amazon CloudWatch data source issues

Fri, 03 Apr 2026 12:35:46 -0500

Troubleshoot Amazon CloudWatch data source issues

This document provides solutions to common issues you may encounter when configuring or using the Amazon CloudWatch data source. For configuration instructions, refer to Configure CloudWatch.

Note
The data source health check validates both metrics and logs permissions. If your IAM policy only grants access to one of these (for example, metrics-only or logs-only), the health check displays a red status. However, the service you have permissions for is still usable—you can query metrics or logs based on whichever permissions are configured.

Authentication errors

These errors occur when AWS credentials are invalid, missing, or don’t have the required permissions.

“Access Denied” or “Not authorized to perform this operation”

Symptoms:

Save & test fails with “Access Denied”
Queries return authorization errors
Namespaces, metrics, or dimensions don’t load

Possible causes and solutions:

Cause	Solution
IAM policy missing required permissions	Attach the appropriate IAM policy to your user or role. For metrics, you need `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, and related permissions. For logs, you need `logs:DescribeLogGroups`, `logs:StartQuery`, `logs:GetQueryResults`, and related permissions. Refer to Configure CloudWatch for complete policy examples.
Incorrect access key or secret key	Verify the credentials in the AWS Console under IAM > Users > your user > Security credentials. Generate new credentials if necessary.
Credentials have expired	For temporary credentials, generate new ones. For access keys, verify they haven’t been deactivated or deleted.
Wrong AWS region	Verify the default region in the data source configuration matches where your resources are located.
Assume Role ARN is incorrect	Verify the role ARN format: `arn:aws:iam::<account-id>:role/<role-name>`. Check that the role exists in the AWS Console.

“Unable to assume role”

Symptoms:

Authentication fails when using Assume Role ARN
Error message references STS or AssumeRole

Solutions:

Verify the trust relationship on the IAM role allows the Grafana credentials to assume it.
Check the trust policy includes the correct principal (the user or role running Grafana).
If using an external ID, ensure it matches exactly in both the role’s trust policy and the Grafana data source configuration.
Verify the base credentials have the sts:AssumeRole permission.
Check that the role ARN is correct and the role exists.

Example trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<your-account-id>:user/<grafana-user>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<your-external-id>"
        }
      }
    }
  ]
}

AWS SDK Default authentication not working

Symptoms:

Data source test fails when using AWS SDK Default
Works locally but fails in production

Solutions:

Verify AWS credentials are configured in the environment where Grafana runs.
Check for credentials in the default locations:
- Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
- Shared credentials file (~/.aws/credentials)
- EC2 instance metadata (if running on EC2)
- ECS task role (if running in ECS)
- EKS service account (if running in EKS)
Ensure the Grafana process has permission to read the credentials file.
For EKS with IRSA, set the pod’s security context to allow user 472 (grafana) to access the projected token. Refer to AWS authentication for details.

Credentials file not found

Symptoms:

Error indicates credentials file cannot be read
Authentication fails with “Credentials file” option

Solutions:

Create the credentials file at ~/.aws/credentials for the user running the grafana-server service.
Verify the file has correct permissions (0644).
If the file exists but isn’t working, move it to /usr/share/grafana/ and set permissions to 0644.
Ensure the profile name in the data source configuration matches a profile in the credentials file.

Connection errors

These errors occur when Grafana cannot reach AWS CloudWatch endpoints.

“Request timed out” or connection failures

Symptoms:

Data source test times out
Queries fail with timeout errors
Intermittent connection issues

Solutions:

Verify network connectivity from the Grafana server to AWS endpoints.
Check firewall rules allow outbound HTTPS (port 443) to AWS services.
If using a VPC, ensure proper NAT gateway or VPC endpoint configuration.
For Grafana Cloud connecting to private resources, configure Private data source connect.
Check if the default region is correct—incorrect regions may cause longer timeouts.
Increase the timeout settings if queries involve large data volumes.

Custom endpoint configuration issues

Symptoms:

Connection fails when using a custom endpoint
Endpoint URL rejected

Solutions:

Verify the endpoint URL format is correct.
Ensure the endpoint is accessible from the Grafana server.
Check that the endpoint supports the required AWS APIs.
For VPC endpoints, verify the endpoint policy allows the required actions.

CloudWatch Metrics query errors

These errors occur when querying CloudWatch Metrics.

“No data” or empty results

Symptoms:

Query executes without error but returns no data
Charts show “No data” message

Possible causes and solutions:

Cause	Solution
Time range doesn’t contain data	Expand the dashboard time range. CloudWatch metrics have different retention periods based on resolution.
Wrong namespace or metric name	Verify the namespace (for example, `AWS/EC2`) and metric name (for example, `CPUUtilization`) are correct.
Incorrect dimensions	Ensure dimension names and values match your AWS resources exactly.
Match Exact enabled incorrectly	When Match Exact is enabled, all dimensions must be specified. Try disabling it to see if metrics appear.
Period too large	Reduce the period setting or set it to “auto” to ensure data points are returned for your time range.
Custom metrics not configured	Add custom metric namespaces in the data source configuration under Namespaces of Custom Metrics.

“Metric not found” or metrics don’t appear in drop-down

Symptoms:

Expected metrics don’t appear in the query editor
Metric drop-down is empty for a namespace

Solutions:

Verify the metric exists in the selected region.
For custom metrics, add the namespace to Namespaces of Custom Metrics in the data source configuration.
Check that the IAM policy includes cloudwatch:ListMetrics permission.
CloudWatch limits ListMetrics to 500 results per page. To retrieve more metrics, increase the list_metrics_page_limit setting in the Grafana configuration file.
Use the Query Inspector to verify the API request and response.

Dimension values not loading

Symptoms:

Dimension value drop-down doesn’t populate
Wildcard searches return no results

Solutions:

Verify the IAM policy includes cloudwatch:ListMetrics permission.
Check that the namespace and metric are selected before dimension values can load.
For EC2 dimensions, ensure ec2:DescribeTags and ec2:DescribeInstances permissions are granted.
Dimension values require existing metrics—if no metrics match, no values appear.

“Too many data points” or API throttling

Symptoms:

Queries fail with throttling errors
Performance degrades with multiple panels

Solutions:

Increase the period setting to reduce the number of data points.
Reduce the time range of your queries.
Use fewer dimensions or wildcard queries per panel.
Request a quota increase for GetMetricData requests per second in the AWS Service Quotas console.
Enable query caching in Grafana to reduce API calls.

Metric math expression errors

Symptoms:

Expression returns errors
Referenced metrics not found

Solutions:

Verify each referenced metric has a unique ID set.
Check that metric IDs start with a lowercase letter and contain only letters, numbers, and underscores.
Ensure all referenced metrics are in the same query.
Verify the expression syntax follows AWS Metric Math documentation.
Metric math expressions can’t be used with Grafana alerting if they reference other query rows.

CloudWatch Logs query errors

These errors occur when querying CloudWatch Logs.

“Query failed” or logs don’t appear

Symptoms:

Log queries return errors
No log data is displayed

Solutions:

Verify log group names are correct and exist in the selected region.
Check the IAM policy includes logs:StartQuery, logs:GetQueryResults, and logs:DescribeLogGroups permissions.
Ensure the time range contains log data.
Verify the query syntax is valid. For CloudWatch Logs Insights QL, test the query in the AWS Console.
Select the correct query language (Logs Insights QL, OpenSearch PPL, or OpenSearch SQL) based on your query syntax.

Log query timeout

Symptoms:

Query runs for a long time then fails
Error mentions timeout

Solutions:

Increase the Query timeout result setting in the data source configuration (default is 30 minutes).
Narrow the time range to reduce the amount of data scanned.
Add filters to your query to limit results.
Break complex queries into smaller, more focused queries.
For alerting, the timeout defined in the Grafana configuration file takes precedence.

Log groups not appearing in selector

Symptoms:

Log group selector is empty
Can’t find expected log groups

Solutions:

Verify the IAM policy includes logs:DescribeLogGroups permission.
Check that log groups exist in the selected region.
For cross-account observability, ensure proper IAM permissions for oam:ListSinks and oam:ListAttachedLinks.
Use prefix search to filter log groups if you have many groups.
Verify the selected account (for cross-account) contains the expected log groups.

OpenSearch SQL query errors

Symptoms:

OpenSearch SQL queries fail
Syntax errors with SQL queries

Solutions:

Specify the log group identifier or ARN in the FROM clause:
SQL
```
SELECT * FROM `log_group_name` WHERE `@message` LIKE '%error%'
```

For multiple log groups, use the logGroups function:

SELECT * FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`

Amazon CloudWatch supports only a subset of OpenSearch SQL commands. Refer to the CloudWatch Logs documentation for supported syntax.

Template variable errors

These errors occur when using template variables with the CloudWatch data source.

Variables return no values

Symptoms:

Variable drop-down is empty
Dashboard fails to load with variable errors

Solutions:

Verify the data source connection is working.
Check that the IAM policy includes permissions for the variable query type:
- Regions: No additional permissions needed.
- Namespaces: No additional permissions needed.
- Metrics: Requires cloudwatch:ListMetrics.
- Dimension Values: Requires cloudwatch:ListMetrics.
- EC2 Instance Attributes: Requires ec2:DescribeInstances.
- EBS Volume IDs: Requires ec2:DescribeVolumes.
- Resource ARNs: Requires tag:GetResources.
- Log Groups: Requires logs:DescribeLogGroups.
For dependent variables, ensure parent variables have valid selections.
Verify the region is set correctly (use “default” for the data source’s default region).

For more information on template variables, refer to CloudWatch template variables.

Multi-value template variables cause query failures

Symptoms:

Queries fail when selecting multiple dimension values
Error about search expression limits

Solutions:

Search expressions are limited to 1,024 characters. Reduce the number of selected values.
Use the asterisk (*) wildcard instead of selecting “All” to query all metrics for a dimension.
Multi-valued template variables are only supported for dimension values—not for Region, Namespace, or Metric Name.

Cross-account observability errors

These errors occur when using CloudWatch cross-account observability features.

Cross-account queries fail

Symptoms:

Can’t query metrics or logs from linked accounts
Monitoring account badge doesn’t appear

Solutions:

Verify cross-account observability is configured in the AWS CloudWatch console.

Add the required IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Check that the monitoring account and source accounts are properly linked in AWS.
Cross-account observability works within a single region—verify all accounts are in the same region.
EC2 Instance Attributes can’t be queried across accounts because they use the EC2 API, not the CloudWatch API.

Quota and pricing issues

These issues relate to AWS service quotas and cost management.

API throttling errors

Symptoms:

“Rate exceeded” errors
Dashboard panels intermittently fail to load

Solutions:

Reduce the frequency of dashboard refreshes.
Increase the period setting to reduce GetMetricData requests.
Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud).
Request a quota increase in the AWS Service Quotas console.
Consider consolidating similar queries using metric math.

Unexpectedly high CloudWatch costs

Symptoms:

AWS CloudWatch costs are higher than expected
Frequent API calls from Grafana

Solutions:

The GetMetricData API doesn’t qualify for the CloudWatch API free tier.
Reduce dashboard auto-refresh frequency.
Increase the period setting to reduce data points returned.
Use query caching to reduce repeated API calls.
Review variable query settings—set variable refresh to “On dashboard load” instead of “On time range change.”
Avoid using wildcards in dimensions when possible, as they generate search expressions with multiple API calls.

Other common issues

These issues don’t produce specific error messages but are commonly encountered.

Custom metrics don’t appear

Symptoms:

Custom metrics from applications or agents don’t show in the namespace drop-down
Only standard AWS namespaces are visible

Solutions:

Add your custom metric namespace to the Namespaces of Custom Metrics field in the data source configuration.
Separate multiple namespaces with commas (for example, CWAgent,CustomNamespace).
Verify custom metrics have been published to CloudWatch in the selected region.

Pre-configured dashboards not working

Symptoms:

Imported dashboards show no data
Dashboard variables don’t load

Solutions:

Verify the data source name in the dashboard matches your CloudWatch data source.
Check that the dashboard’s AWS region setting matches where your resources are located.
Ensure the IAM policy grants access to the required services (EC2, Lambda, RDS, etc.).
Verify resources exist and are emitting metrics in the selected region.

X-Ray trace links not appearing

Symptoms:

Log entries don’t show X-Ray trace links
@xrayTraceId field not appearing

Solutions:

Verify an X-Ray data source is configured and linked in the CloudWatch data source settings.
Ensure your logs contain the @xrayTraceId field.
Update log queries to include @xrayTraceId in the fields, for example: fields @message, @xrayTraceId.
Configure your application to log X-Ray trace IDs. Refer to the AWS X-Ray documentation.

Enable debug logging

To capture detailed error information for troubleshooting:

Set the Grafana log level to debug in the configuration file:
ini
```
[log]
level = debug
```
Review logs in /var/log/grafana/grafana.log (or your configured log location).
Look for CloudWatch-specific entries that include request and response details.
Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried the solutions above and still encounter issues:

Check the Grafana community forums for similar issues.
Review the CloudWatch plugin GitHub issues for known bugs.
Consult the AWS CloudWatch documentation for service-specific guidance.
Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Contracted user.
When reporting issues, include:
- Grafana version
- AWS region
- Error messages (redact sensitive information)
- Steps to reproduce
- Query configuration (redact credentials and account IDs)