Data sources on Grafana Labs

Add data source

Sat, 04 Apr 2026 12:26:57 +0000

Add a data source

Before you can create your first dashboard, you need to add your data source.

Note: Only users with the organization Admin role can add data sources.

To add a data source:

Move your cursor to the cog icon on the side menu which will show the configuration options.
Click on Data sources. The data sources page opens showing a list of previously configured data sources for the Grafana instance.
Click Add data source to see a list of all supported data sources.
Search for a specific data source by entering the name in the search dialog. Or you can scroll through supported data sources grouped into time series, logging, tracing and other categories.
Move the cursor over the data source you want to add.
Click Select. The data source configuration page opens.
Configure the data source following instructions specific to that data source. See Data sources for links to configuration instructions for all supported data sources.

Alertmanager

Sat, 04 Apr 2026 12:26:57 +0000

Alertmanager data source

Grafana includes built-in support for Prometheus Alertmanager. It is presently in alpha and not accessible unless alpha plugins are enabled in Grafana settings. Once you add it as a data source, you can use the Grafana alerting UI to manage silences, contact points as well as notification policies. A drop-down option in these pages allows you to switch between Grafana and any configured Alertmanager data sources.

Alertmanager implementations

Prometheus and Cortex (default) implementations of Alertmanager are supported. You can specify implementation in the data source settings page. In case of Prometheus contact points and notification policies are read-only in the Grafana alerting UI, as it does not support updating configuration via HTTP API.

Provision the Alertmanager data source

Configure the Alertmanager data sources by updating Grafana’s configuration files. For more information on how it works and the settings available, refer to the provisioning docs page.

Here is an example for provisioning the Alertmanager data source:

apiVersion: 1

datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://localhost:9093
    access: proxy
    jsonData:
      implementation: 'prometheus' # alternatively 'cortex'
    # optionally
    basicAuth: true
    basicAuthUser: my_user
    basicAuthPassword: test_password

AWS CloudWatch

Sat, 04 Apr 2026 12:26:57 +0000

AWS CloudWatch data source

Grafana ships with built-in support for CloudWatch. This topic describes queries, templates, variables, and other configuration specific to the CloudWatch data source. For instructions on how to add a data source to Grafana, refer to Add a data source. Only users with the organization admin role can add data sources.

Once you have added the Cloudwatch data source, you can build dashboards or use Explore with CloudWatch metrics and CloudWatch Logs.

Note: For troubleshooting issues when setting up the Cloudwatch data source, check the /var/log/grafana/grafana.log file.

Configure the CloudWatch data source

To access data source settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and then click the AWS Cloudwatch data source.

For authentication options and configuration details, see AWS authentication topic.

CloudWatch specific data source configuration

IAM policies

Grafana needs permissions granted via IAM to be able to read CloudWatch metrics and EC2 tags/instances/regions/alarms. You can attach these permissions to the IAM role or IAM user configured in the previous step.

Metrics only example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Logs only example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Metrics and Logs example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Namespaces of Custom Metrics

Grafana is not able to load custom namespaces through the GetMetricData API. If you still want your custom metrics to show up in the fields in the query editor, you can specify the names of the namespaces containing the custom metrics in the Namespaces of Custom Metrics field. The field accepts a multiple namespaces, separated by a comma.

Timeout

Timeout specifically, for CloudWatch Logs queries. Log queries don’t recognize standard Grafana query timeout as they don’t keep a single request open and instead periodically poll for results. Because of limits on concurrently running queries in CloudWatch they can also take a longer time to finish.

X-Ray trace links

Link an X-Ray data source in the “X-Ray trace link” section of the configuration page to automatically add links in your logs when the log contains @xrayTraceId field.

The data source select will contain only existing data source instances of type X-Ray so in order to use this feature you need to have existing X-Ray data source already configured, see X-Ray docs for details.

The X-Ray link will then appear in the log details section which is accessible by clicking on the log row either in Explore or in dashboard Logs panel. To log the @xrayTraceId in your logs see the AWS X-Ray documentation. To provide the field to Grafana your log queries also have to contain the @xrayTraceId field, for example using query fields @message, @xrayTraceId.

CloudWatch query editor

The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor. You select which API you want to query with using the query mode switch on top of the editor.

Metrics query editor

The metrics query editor allows you to build two types of queries - Metric Search and Metric Query.

Using the Metric Search option

To create a valid Metric Search query specify the namespace, metric name and at least one statistic.

If Match Exact is enabled, you also need to specify all the dimensions of the metric you’re querying, so that the metric schema matches exactly. If Match Exact is disabled, you can specify any number of dimensions by which you’d like to filter. Up to 100 metrics matching your filter criteria will be returned.

Dynamic queries using dimension wildcards

You can monitor a dynamic list of metrics by using the asterisk (*) wildcard for one or more dimension values.

In this example, the query returns all metrics in the namespace AWS/EC2 with a metric name of CPUUtilization and ANY value for the InstanceId dimension are queried. This can help you monitor metrics for AWS resources, like EC2 instances or containers. When new instances are created as part of an auto scaling event, they will automatically appear in the graph without you having to track the new instance IDs. This capability is currently limited to retrieving up to 100 metrics.

You can expand the Query inspector button and click Meta Data to see the search expression that is automatically built to support wildcards. To learn more about search expressions, visit the CloudWatch documentation. By default, the search expression is defined in such a way that the queried metrics must match the defined dimension names exactly. This means that in the example only metrics with exactly one dimension with the name ‘InstanceId’ will be returned.

You can disable Match Exact to include metrics that have other dimensions defined. Disabling Match Exact also creates a search expression even if you don’t use wildcards. We simply search for any metric that matches at least the namespace, metric name, and all defined dimensions.

Multi-value template variables

When defining dimension values based on multi-valued template variables, a search expression is used to query for the matching metrics. This enables the use of multiple template variables in one query and also allows you to use template variables for queries that have the Match Exact option disabled.

Search expressions are currently limited to 1024 characters, so your query may fail if you have a long list of values. We recommend using the asterisk (*) wildcard instead of the All option if you want to query all metrics that have any value for a certain dimension name.

The use of multi-valued template variables is only supported for dimension values. Using multi-valued template variables for Region, Namespace, or Metric Name is not supported.

Metric math expressions

You can create new time series metrics by operating on top of CloudWatch metrics using mathematical functions. Arithmetic operators, unary subtraction and other functions are supported and can be applied to CloudWatch metrics. More details on the available functions can be found on AWS Metric Math

As an example, if you want to apply arithmetic operations on a metric, you can do it by giving an id (a unique string) to the raw metric as shown below. You can then use this id and apply arithmetic operations to it in the Expression field of the new metric.

Please note that in the case you use the expression field to reference another query, like queryA * 2, it will not be possible to create an alert rule based on that query.

Deep linking from Grafana panels to the CloudWatch console

Left clicking a time series in the panel shows a context menu with a link to View in CloudWatch console. Clicking that link will open a new tab that will take you to the CloudWatch console and display all the metrics for that query. If you’re not currently logged in to the CloudWatch console, the link will forward you to the login page. The provided link is valid for any account but will only display the right metrics if you’re logged in to the account that corresponds to the selected data source in Grafana.

This feature is not available for metrics that are based on metric math expressions.

Using the Metric Query option

Note: This query option is available in Grafana 8.3 and higher versions only.

Metrics Query in the CloudWatch plugin is what is referred to as Metric Insights in the AWS console. It’s a fast, flexible, SQL-based query engine that enables you to identify trends and patterns across millions of operational metrics in real time. It uses a dialect of SQL. The query syntax is as follows.


SELECT FUNCTION(MetricName)
FROM Namespace | SCHEMA(...)
[ WHERE labelKey OPERATOR labelValue [AND|...]]
[ GROUP BY labelKey [, ...]]
[ ORDER BY FUNCTION() [DESC | ASC] ]
[ LIMIT number]

The following table provides basic explanation of the query keywords. For details about the Metrics Insights syntax, refer to the AWS documentation.

Keyword	Description
`FUNCTION`	Required. Specifies the aggregate function to use, and also specifies the name of the metric that is to be queried. Valid values are AVG, COUNT, MAX, MIN, and SUM
`MetricName`	Required. For example, `CPUUtilization`.
`FROM`	Required. Specifies the source of the metric. You can specify either the metric namespace that contains the metric that is to be queried, or a SCHEMA table function. Some namespace examples are 1`AWS/EC2`, `AWS/Lambda`.
`SCHEMA`	Optional. Allows you to narrow down the query results to only the metrics that is an exact match or to metrics that do noy match.
`WHERE`	Optional. Filters the results to only those metrics that match your specified expression. For example, `WHERE InstanceType != 'c3.4xlarge'`.
`GROUP BY`	Optional. Groups the query results into multiple time series. For example, `GROUP BY ServiceName`.
`ORDER BY`	Optional. Specifies the order of time series that are returned. Options are `ASC`, `DESC`.
`LIMIT`	Optional. Limits the number of time series returned.

For information about limits for the Metrics Insights, please refer to the AWS documentation.

Builder mode

To create a query in Builder mode:

Browse and select a metric namespace, metric name, filter, group, and order options using information from the table above.
For each of these options, choose from the list of possible options.

Grafana automatically constructs a SQL query based on your selections.

Code mode

To create a query in the Code mode:

Write your SQL query.
To run the query, click the Run query above the code editor.

The code editor has a built in autocomplete feature that gives suggestions for keywords, aggregations, namespaces, metrics, labels and label values. The suggestions are shown when hitting space, comma or dollar character. You can also use the keyboard combination CTRL+Space.

Note: Usage of template variables in the code editor might interfere the autocompletion.

Common metric query editor fields

At the bottom of the metric query editor, you’ll find three fields that are common to both Metric Search and Metric Query.

Id

The GetMetricData API requires that all queries have a unique ID. Use this field to specify an ID of choice. The ID can include numbers, letters, and underscore, and must start with a lowercase letter. If no ID is specified, grafana will generate an ID using the following pattern query[refId of the current query row], e.g queryA for the first query row in the panel editor.

The ID can be used to reference queries in Metric Math expressions.

Period

A period is the length of time associated with a specific Amazon CloudWatch statistic. Periods are defined in numbers of seconds, and valid values for period are 1, 5, 10, 30, or any multiple of 60.

If the period field is left blank or set to auto, then it calculates automatically based on the time range and cloudwatch’s retention policy. The formula used is time range in seconds / 2000, and then it snaps to the next higher value in an array of predefined periods [60, 300, 900, 3600, 21600, 86400] after removing periods based on retention. By clicking Show Query Preview in the query editor, you can see what period Grafana used.

Alias

The alias field allows you to override the default name of the metric legend.

Alias patterns

Alias Pattern	Description	Example Result
`{{region}}`	returns the region	`us-east-1`
`{{period}}`	returns the period	`3000`
`{{metric}}`	returns the metric	`CPUUtilization`
`{{label}}`	returns the label returned by the API (only in Metric Search)	`i-01343`
`{{namespace}}`	returns the namespace (only in Metric Search)	`AWS/EC2`
`{{stat}}`	returns the statistic (only in Metric Search)	`Average`
`{{[dimension name]}}`	returns the dimension name (only in Metric Search)	`i-01343`

Using the Logs query editor

To query CloudWatch Logs:

Select the region and up to 20 log groups which you want to query.
Use the main input area to write your query in CloudWatch Logs Query Language.

You can also write queries returning time series data by using the stats command. When making stats queries in Explore, you have to make sure you are in Metrics Explore mode.

Deep linking from Grafana panels to the CloudWatch console

If you’d like to view your query in the CloudWatch Logs Insights console, simply click the CloudWatch Logs Insights button next to the query editor. If you’re not currently logged in to the CloudWatch console, the link will forward you to the login page. The provided link is valid for any account but will only display the right metrics if you’re logged in to the account that corresponds to the selected data source in Grafana.

Alerting

Since CloudWatch Logs queries can return numeric data, for example through the use of the stats command, alerts are supported. For more information on Grafana alerts, refer to Alerting documentation.

Configure CloudWatch with grafana.ini

The Grafana configuration file includes an AWS section where you can customize the data source.

Configuration option	Description
`allowed_auth_providers`	Specifies which authentication providers are allowed for the CloudWatch data source. The following providers are enabled by default in OSS Grafana: `default` (AWS SDK default), keys (Access and secret key), credentials (Credentials file), ec2_IAM_role (EC2 IAM role).
`assume_role_enabled`	Allows you to disable `assume role (ARN)` in the CloudWatch data source. By default, assume role (ARN) is enabled for OSS Grafana.
`list_metrics_page_limit`	When a custom namespace is specified in the query editor, the List Metrics API is used to populate the Metrics field and the Dimension fields. The API is paginated and returns up to 500 results per page. The CloudWatch data source also limits the number of pages to 500. However, you can change this limit using the `list_metrics_page_limit` variable in the grafana configuration file.

Pricing

The Amazon CloudWatch data source for Grafana uses the ListMetrics and GetMetricData CloudWatch API calls to list and retrieve metrics. Pricing for CloudWatch Logs is based on the amount of data ingested, archived, and analyzed via CloudWatch Logs Insights queries. Every time you pick a dimension in the query editor Grafana will issue a ListMetrics request. Whenever you make a change to the queries in the query editor, one new request to GetMetricData will be issued.

In Grafana version 6.5 or higher, all API requests to GetMetricStatistics have been replaced with calls to GetMetricData to provide better support for CloudWatch metric math and enables the automatic generation of search expressions when using wildcards or disabling the Match Exact option. While GetMetricStatistics qualified for the CloudWatch API free tier, this is not the case for GetMetricData calls.

For more information, please refer to the CloudWatch pricing page.

Service quotas

AWS defines quotas, or limits, for resources, actions, and items in your AWS account. Depending on the number of queries in your dashboard and the number of users accessing the dashboard, you may reach the usage limits for various CloudWatch and CloudWatch Logs resources. Note that quotas are defined per account and per region. If you’re using multiple regions or have set up more than one CloudWatch data source to query against multiple accounts, you need to request a quota increase for each account and each region in which you hit the limit.

To request a quota increase, visit the AWS Service Quotas console. For more information, refer to the AWS documentation for Service Quotas and CloudWatch limits.

Azure Monitor

Sat, 04 Apr 2026 12:26:57 +0000

Azure Monitor data source

Grafana includes built-in support for Azure Monitor, the Azure service to maximize the availability and performance of your applications and services in the Azure Cloud. The Azure Monitor data source supports visualizing data from three Azure services:

Azure Monitor Metrics to collect numeric data from resources in your Azure account.
Azure Monitor Logs to collect log and performance data from your Azure account, and query using the powerful Kusto Language.
Azure Resource Graph to quickly query your Azure resources across subscriptions.

This topic explains configuring, querying, and other options specific to the Azure Monitor data source. Refer to Add a data source for instructions on how to add a data source to Grafana.

Azure Monitor configuration

To access Azure Monitor configuration, hover your mouse over the Configuration (gear) icon, click Data Sources, and then select the Azure Monitor data source. If you haven’t already, you’ll need to add the Azure Monitor data source.

You must create an app registration and service principal in Azure AD to authenticate the data source. See the Azure documentation for configuration details. Alternatively, if you are hosting Grafana in Azure (e.g. App Service, or Azure Virtual Machines) you can configure the Azure Monitor data source to use Managed Identity to securely authenticate without entering credentials into Grafana. Refer to Configuring using Managed Identity for more details.

Name	Description
Authentication	Enables Managed Identity. Selecting Managed Identity will hide many of the fields below. See Configuring using Managed Identity for more details.
Azure Cloud	The national cloud for your Azure account. For most users, this is the default “Azure”. For more information, see the Azure documentation.
Directory (tenant) ID	The directory/tenant ID for the Azure AD app registration to use for authentication. See Get tenant and app ID values for signing in from the Azure documentation.
Application (client) ID	The application/client ID for the Azure AD app registration to use for authentication.
Client secret	The application client secret for the Azure AD app registration to use for authentication. See Create a new application secret from the Azure documentation.
Default subscription	(optional) Sets a default subscription for template variables to use
Default workspace	(optional) Sets a default workspace for Log Analytics-based template variable queries to use

Azure Monitor query editor

The Azure Monitor data source has three different modes depending on which Azure service you wish to query:

Querying Azure Monitor Metrics

Azure Monitor Metrics collects numeric data from supported resources and allows you to query them to investigate the health and utilization of your resources to maximise availability and performance.

Metrics are a lightweight format that only stores simple numeric data in a particular structure. Metrics is capable for supporting near real-time scenarios making it useful for fast detection of issues. Azure Monitor Logs can store a variety of different data types each with their own structure.

Your first Azure Monitor Metrics query

Select the Metrics service
Select a resource to pull metrics from using the subscription, resource group, resource type, and resource fields.
Some resources, such as storage accounts, organise metrics under multiple metric namespaces. Grafana will pick a default namespace, but change this to see which other metrics are available.
Select a metric from the Metric field.

Optionally, you can apply further aggregations or filter by dimensions for further analysis.

Change the aggregation from the default average to show minimum, maximum or total values.
Set a specific custom time grain. By default Grafana will automatically select a time grain interval based on your selected time range.
For metrics that have multiple dimensions, you can split and filter further the returned metrics. For example, the Application Insights dependency calls metric supports returning multiple time series for successful vs unsuccessful calls.

The options available will change depending on what is most relevant to the selected metric.

Legend alias formatting

The legend label for Metrics can be changed using aliases. In the Legend Format field, you can combine aliases defined below any way you want e.g

Blob Type: {{ blobtype }} becomes Blob Type: PageBlob, Blob Type: BlockBlob
{{ resourcegroup }} - {{ resourcename }} becomes production - web_server

Alias pattern	Description
`{{ resourcegroup }}`	Replaced with the the resource group
`{{ namespace }}`	Replaced with the resource type / namespace (e.g. Microsoft.Compute/virtualMachines)
`{{ resourcename }}`	Replaced with the resource name
`{{ metric }}`	Replaced with the metric name (e.g. Percentage CPU)
`{{ arbitaryDimensionID }}`	Replaced with the value of the specified dimension. (e.g. {{ blobtype }} becomes BlockBlob)
`{{ dimensionname }}`	(Legacy for backwards compatibility) Replaced with the name of the first dimension
`{{ dimensionvalue }}`	(Legacy for backwards compatibility) Replaced with the value of the first dimension

Supported Azure Monitor metrics

Not all metrics returned by the Azure Monitor Metrics API have values. To make it easier for you when building a query, the Grafana data source has a list of supported metrics and ignores metrics which will never have values. This list is updated regularly as new services and metrics are added to the Azure cloud. For more information about the list of metrics, refer to current supported namespaces.

Querying Azure Monitor Logs

Azure Monitor Logs collects and organises log and performance data from supported resources and makes many sources of data available to query together with the sophisticated Kusto Query Language (KQL).

While Azure Monitor Metrics only stores simplified numerical data, Logs can store different data types each with their own structure and can perform complexe analysis of data using KQL.

Your first Azure Monitor Logs query

Select the Logs service
Select a resource to query. Alternatively, you can dynamically query all resources under a single resource group or subscription.
Enter in your KQL query. See below for examples.

Kusto Query Language

Azure Monitor Logs queries are written using the Kusto Query Language (KQL), a rich language designed to be easy to read and write, which should be familiar to those know who SQL. The Azure documentation has plenty of resource to help with learning KQL:

Here is an example query that returns a virtual machine’s CPU performance, averaged over 5m time grains

Perf
# $__timeFilter is a special Grafana macro that filters the results to the time span of the dashboard
| where $__timeFilter(TimeGenerated)
| where CounterName == "% Processor Time"
| summarize avg(CounterValue) by bin(TimeGenerated, 5m), Computer
| order by TimeGenerated asc

Time series queries are for values that change over time, usually for graph visualisations such as the Time series panel. Each query should return at least a datetime column and a numeric value column. The result must also be sorted in ascending order by the datetime column.

A query can also have one or more non-numeric/non-datetime columns, and those columns are considered dimensions and become labels in the response. For example, a query that returns the aggregated count grouped by hour, Computer, and the CounterName:

Perf
| where $__timeFilter(TimeGenerated)
| summarize count() by bin(TimeGenerated, 1h), Computer, CounterName
| order by TimeGenerated asc

You can also select additional number value columns (with, or without multiple dimensions). For example, getting a count and average value by hour, Computer, CounterName, and InstanceName:

Perf
| where $__timeFilter(TimeGenerated)
| summarize Samples=count(), ["Avg Value"]=avg(CounterValue)
    by bin(TimeGenerated, $__interval), Computer, CounterName, InstanceName
| order by TimeGenerated asc

Table queries are mainly used in the Table panel and show a list of columns and rows. This example query returns rows with the six specified columns:

AzureActivity
| where $__timeFilter()
| project TimeGenerated, ResourceGroup, Category, OperationName, ActivityStatus, Caller
| order by TimeGenerated desc

Logs macros

To make writing queries easier there are several Grafana macros that can be used in the where clause of a query:

Macro	Description
`$__timeFilter()`	Used to filter the results to the time range of the dashboard. Example: `TimeGenerated >= datetime(2018-06-05T18:09:58.907Z) and TimeGenerated <= datetime(2018-06-05T20:09:58.907Z)`.
`$__timeFilter(datetimeColumn)`	Like `$__timeFilter()`, but specifies a custom field to filter on.
`$__timeFrom()`	Expands to the start of the dashboard time range. Example: `datetime(2018-06-05T18:09:58.907Z)`.
`$__timeTo()`	Expands to the end of the dashboard time range. Example: `datetime(2018-06-05T20:09:58.907Z)`.
`$__escapeMulti($myVar)`	Used with multi-value template variables that contain illegal characters. If `$myVar` has the following two values as a string `'\\grafana-vm\Network(eth0)\Total','\\hello!'`, then it expands to `@'\\grafana-vm\Network(eth0)\Total', @'\\hello!'`. If using single value variables there is no need for this macro, simply escape the variable inline instead - `@'\$myVar'`.
`$__contains(colName, $myVar)`	Used with multi-value template variables. If `$myVar` has the value `'value1','value2'`, it expands to: `colName in ('value1','value2')`. If using the `All` option, then check the `Include All Option` checkbox and in the `Custom all value` field type in the value `all`. If `$myVar` has value `all` then the macro will instead expand to `1 == 1`. For template variables with a lot of options, this will increase the query performance by not building a large “where..in” clause.

Additionally, Grafana has the built-in $__interval macro

Querying Azure Resource Graph

Azure Resource Graph (ARG) is a service in Azure that is designed to extend Azure Resource Management by providing efficient and performant resource exploration, with the ability to query at scale across a given set of subscriptions so that you can effectively govern your environment. By querying ARG, you can query resources with complex filtering, iteratively explore resources based on governance requirements, and assess the impact of applying policies in a vast cloud environment.

Your first Azure Resource Graph query

ARG queries are written in a variant of the Kusto Query Language, but not all Kusto language features are available in ARG. An Azure Resource Graph query is formatted as table data.

If your credentials give you access to multiple subscriptions, then you can choose multiple subscriptions before entering queries.

Sort results by resource properties

Here is an example query that returns all resources in the selected subscriptions, but only the name, type, and location properties:

Resources
| project name, type, location
| order by name asc

The query uses order by to sort the properties by the name property in ascending (asc) order. You can change what property to sort by and the order (asc or desc). The query uses project to show only the listed properties in the results. You can add or remove properties.

Query resources with complex filtering

Filtering for Azure resources with a tag name of environment that have a value of Internal. You can change these to any desired tag key and value. The =~ in the type match tells Resource Graph to be case insensitive. You can project by other properties or add/remove more.

For example, a query that returns a list of resources with an environment tag value of Internal:

Resources
| where tags.environment=~'internal'
| project name

Group and aggregate the values by property

You can also use summarize and count to define how to group and aggregate the values by property. For example, returning count of healthy, unhealthy, and not applicable resources per recommendation:

securityresources
| where type == 'microsoft.security/assessments'
| extend resourceId=id,
    recommendationId=name,
    resourceType=type,
    recommendationName=properties.displayName,
    source=properties.resourceDetails.Source,
    recommendationState=properties.status.code,
    description=properties.metadata.description,
    assessmentType=properties.metadata.assessmentType,
    remediationDescription=properties.metadata.remediationDescription,
    policyDefinitionId=properties.metadata.policyDefinitionId,
    implementationEffort=properties.metadata.implementationEffort,
    recommendationSeverity=properties.metadata.severity,
    category=properties.metadata.categories,
    userImpact=properties.metadata.userImpact,
    threats=properties.metadata.threats,
    portalLink=properties.links.azurePortal
| summarize numberOfResources=count(resourceId) by tostring(recommendationName), tostring(recommendationState)

In Azure Resource Graph many nested properties (properties.displayName) are of a dynamic type, and should be cast to a string with tostring() to operate on them.

The Azure documentation also hosts many sample queries to help you get started

Azure Resource Graph macros

You can use Grafana macros when constructing a query. Use the macros in the where clause of a query:

$__timeFilter() - Expands to timestamp ≥ datetime(2018-06-05T18:09:58.907Z) and timestamp ≤ datetime(2018-06-05T20:09:58.907Z) where the from and to datetimes are from the Grafana time picker.
$__timeFilter(datetimeColumn) - Expands to datetimeColumn ≥ datetime(2018-06-05T18:09:58.907Z) and datetimeColumn ≤ datetime(2018-06-05T20:09:58.907Z) where the from and to datetimes are from the Grafana time picker.
$__timeFrom() - Returns the From datetime from the Grafana picker. Example: datetime(2018-06-05T18:09:58.907Z).
$__timeTo() - Returns the To datetime from the Grafana picker. Example: datetime(2018-06-05T20:09:58.907Z).
$__escapeMulti($myVar) - Use with multi-value template variables that contain illegal characters. If $myVar has the two values as a string '\\grafana-vm\Network(eth0)\Total','\\hello!', then it expands to: @'\\grafana-vm\Network(eth0)\Total', @'\\hello!'. If you are using single value variables, then there is no need for this macro, simply escape the variable inline instead - @'\$myVar'.
$__contains(colName, $myVar) - Use with multi-value template variables. If $myVar has the value 'value1','value2', the it expands to: colName in ('value1','value2').

If using the All option, then check the Include All Option checkbox and in the Custom all value field type in the following value: all. If $myVar has value all then the macro will instead expand to 1 == 1. For template variables with a lot of options, this will increase the query performance by not building a large “where..in” clause.

Going further with Azure Monitor

See the following topics to learn more about the Azure Monitor data source:

Azure Monitor template variables for more interactive, dynamic, and reusable dashboards.
Provisioning Azure Monitor for configuring the Azure Monitor data source using YAML files
Deprecating Application Insights and migrating to Metrics and Logs queries

Configuring using Managed Identity

Customers who host Grafana in Azure (e.g. App Service, Azure Virtual Machines) and have managed identity enabled on their VM, will now be able to use the managed identity to configure Azure Monitor in Grafana. This will simplify the data source configuration, requiring the data source to be securely authenticated without having to manually configure credentials via Azure AD App Registrations for each data source. For more details on Azure managed identities, refer to the Azure documentation.

To enable managed identity for Grafana, set the managed_identity_enabled flag in the [azure] section of the Grafana server config.

[azure]
managed_identity_enabled = true

Then, in the Azure Monitor data source configuration and set Authentication to Managed Identity. The directory ID, application ID and client secret fields will be hidden and the data source will use managed identity for authenticating to Azure Monitor Metrics, Logs, and Azure Resource Graph.

Elasticsearch

Sat, 04 Apr 2026 12:26:57 +0000

Using Elasticsearch in Grafana

Grafana ships with advanced support for Elasticsearch. You can do many types of simple or complex Elasticsearch queries to visualize logs or metrics stored in Elasticsearch. You can also annotate your graphs with log events stored in Elasticsearch.

Adding the data source

Open the side menu by clicking the Grafana icon in the top header.
In the side menu under the Dashboards link you should find a link named Data Sources.
Click the + Add data source button in the top header.
Select Elasticsearch from the Type dropdown.

Note: If you’re not seeing the Data Sources link in your side menu it means that your current user does not have the Admin role for the current organization.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source means that it will be pre-selected for new panels.
`Url`	The HTTP protocol, IP, and port of your Elasticsearch server.
`Access`	Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. Note: Browser (direct) access is deprecated and will be removed in a future release.

Access mode controls how requests to the data source will be handled. Server should be the preferred way if nothing else stated.

Server access mode (Default)

All requests will be made from the browser to Grafana backend/server which in turn will forward the requests to the data source and by that circumvent possible Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the grafana backend/server if you select this access mode.

Browser (Direct) access

Warning: Browser (Direct) access is deprecated and will be removed in a future release.

All requests will be made from the browser directly to the data source and may be subject to Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the browser if you select this access mode.

If you select Browser access you must update your Elasticsearch configuration to allow other domains to access Elasticsearch from the browser. You do this by specifying these two options in your elasticsearch.yml config file.

http.cors.enabled: true
http.cors.allow-origin: "*"

Index settings

Here you can specify a default for the time field and specify the name of your Elasticsearch index. You can use a time pattern for the index name or a wildcard.

Elasticsearch version

Select the version of your Elasticsearch data source from the version selection dropdown. Different query compositions and functionalities are available in the query editor for different versions. Available Elasticsearch versions are 2.x, 5.x, 5.6+, 6.0+, 7.0+, 7.7+ and 7.10+. Select the option that best matches your data source version.

Grafana assumes that you are running the lowest possible version for a specified range. This ensures that new features or breaking changes in a future Elasticsearch release will not affect your configuration.

For example, suppose you are running Elasticsearch 7.6.1 and you selected 7.0+. If a new feature is made available for Elasticsearch 7.5.0 or newer releases, then a 7.5+ option will be available. However, your configuration will not be affected until you explicitly select the new 7.5+ option in your settings.

Min time interval

A lower limit for the auto group by time interval. Recommended to be set to write frequency, for example 1m if your data is written every minute. This option can also be overridden/configured in a dashboard panel under data source options. It’s important to note that this value needs to be formatted as a number followed by a valid time identifier, e.g. 1m (1 minute) or 30s (30 seconds). The following time identifiers are supported:

Identifier	Description
`y`	year
`M`	month
`w`	week
`d`	day
`h`	hour
`m`	minute
`s`	second
`ms`	millisecond

X-Pack enabled

Enables X-Pack specific features and options, providing the query editor with additional aggregations such as Rate and Top Metrics.

Include frozen indices

When X-Pack enabled is active and the configured Elasticsearch version is higher than 6.6.0, you can configure Grafana to not ignore frozen indices when performing search requests.

Logs

There are two parameters, Message field name and Level field name, that can optionally be configured from the data source settings page that determine which fields will be used for log messages and log levels when visualizing logs in Explore.

For example, if you’re using a default setup of Filebeat for shipping logs to Elasticsearch the following configuration should work:

Message field name: message
Level field name: fields.level

Data links

Data links create a link from a specified field that can be accessed in logs view in Explore.

Each data link configuration consists of:

Field - Name of the field used by the data link.
URL/query - If the link is external, then enter the full link URL. If the link is internal link, then this input serves as query for the target data source. In both cases, you can interpolate the value from the field with ${__value.raw } macro.
URL Label - (Optional) Set a custom display label for the link. The link label defaults to the full external URL or name of the linked internal data source and is overridden by this setting.
Internal link - Select if the link is internal or external. In case of internal link, a data source selector allows you to select the target data source. Only tracing data sources are supported.

Metric Query editor

The Elasticsearch query editor allows you to select multiple metrics and group by multiple terms or filters. Use the plus and minus icons to the right to add/remove metrics or group by clauses. Some metrics and group by clauses haves options, click the option text to expand the row to view and edit metric or group by options.

Series naming and alias patterns

You can control the name for time series via the Alias input field.

Pattern	Description
`{{term fieldname}}`	replaced with value of a term group by
`{{metric}}`	replaced with metric name (ex. Average, Min, Max)
`{{field}}`	replaced with the metric field name

Pipeline metrics

Some metric aggregations are called Pipeline aggregations, for example, Moving Average and Derivative. Elasticsearch pipeline metrics require another metric to be based on. Use the eye icon next to the metric to hide metrics from appearing in the graph. This is useful for metrics you only have in the query for use in a pipeline metric.

Templating

Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query variable

The Elasticsearch data source supports two types of queries you can use in the Query field of Query variables. The query is written using a custom JSON string. The field should be mapped as a keyword in the Elasticsearch index mapping. If it is multi-field with both a text and keyword type, then use "field":"fieldname.keyword"(sometimesfieldname.raw) to specify the keyword field in your query.

Query	Description
`{"find": "fields", "type": "keyword"}`	Returns a list of field names with the index type `keyword`.
`{"find": "terms", "field": "hostname.keyword", "size": 1000}`	Returns a list of values for a keyword using term aggregation. Query will use current dashboard time range as time range query.
`{"find": "terms", "field": "hostname", "query": '<lucene query>'}`	Returns a list of values for a keyword field using term aggregation and a specified lucene query filter. Query will use current dashboard time range as time range for query.

There is a default size limit of 500 on terms queries. Set the size property in your query to set a custom limit. You can use other variables inside the query. Example query definition for a variable named $host.

{"find": "terms", "field": "@hostname", "query": "@source:$source"}

In the above example, we use another variable named $source inside the query definition. Whenever you change, via the dropdown, the current value of the $source variable, it will trigger an update of the $host variable so it now only contains hostnames filtered by in this case the @source document property.

These queries by default return results in term order (which can then be sorted alphabetically or numerically as for any variable). To produce a list of terms sorted by doc count (a top-N values list), add an orderBy property of “doc_count”. This automatically selects a descending sort; using “asc” with doc_count (a bottom-N list) can be done by setting order: "asc" but is discouraged as it “increases the error on document counts”. To keep terms in the doc count order, set the variable’s Sort dropdown to Disabled; you might alternatively still want to use e.g. Alphabetical to re-sort them.

{"find": "terms", "field": "@hostname", "orderBy": "doc_count"}

Using variables in queries

There are two syntaxes:

$<varname> Example: @hostname:$hostname
[[varname]] Example: @hostname:[[hostname]]

Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the Multi-value or Include all value options are enabled, Grafana converts the labels from plain text to a lucene compatible condition.

In the above example, we have a lucene query that filters documents based on the @hostname property using a variable named $hostname. It is also using a variable in the Terms group by field input box. This allows you to use a variable to quickly change how the data is grouped.

Example dashboard: Elasticsearch Templated Dashboard

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view. Grafana can query any Elasticsearch index for annotation events.

Name	Description
`Query`	You can leave the search query blank or specify a lucene query.
`Time`	The name of the time field, needs to be date field.
`Time End`	Optional name of the time end field needs to be date field. If set, then annotations will be marked as a region between time and time-end.
`Text`	Event description field.
`Tags`	Optional field name to use for event tags (can be an array or a CSV string).

Querying Logs

Querying and displaying log data from Elasticsearch is available in Explore, and in the logs panel in dashboards. Select the Elasticsearch data source, and then optionally enter a lucene query to display your logs.

When switching from a Prometheus or Loki data source in Explore, your query is translated to an Elasticsearch log query with a correct Lucene filter.

Log Queries

Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.

Note that the fields used for log message and level is based on an optional data source configuration.

Filter Log Messages

Optionally enter a lucene query into the query field to filter the log messages. For example, using a default Filebeat setup you should be able to use fields.level:error to only show error log messages.

Configure the data source with provisioning

It’s now possible to configure data sources using config files with Grafana’s provisioning system. You can read more about how it works and all the settings you can set for data sources on the provisioning docs page

Here are some provisioning examples for this data source.

apiVersion: 1

datasources:
  - name: Elastic
    type: elasticsearch
    access: proxy
    database: '[metrics-]YYYY.MM.DD'
    url: http://localhost:9200
    jsonData:
      interval: Daily
      timeField: '@timestamp'

or, for logs:

apiVersion: 1

datasources:
  - name: elasticsearch-v7-filebeat
    type: elasticsearch
    access: proxy
    database: '[filebeat-]YYYY.MM.DD'
    url: http://localhost:9200
    jsonData:
      interval: Daily
      timeField: '@timestamp'
      esVersion: '7.0.0'
      logMessageField: message
      logLevelField: fields.level
      dataLinks:
        - datasourceUid: my_jaeger_uid # Target UID needs to be known
          field: traceID
          url: '$${__value.raw}' # Careful about the double "$$" because of env var expansion

Amazon Elasticsearch Service

AWS users using Amazon’s Elasticsearch Service can use Grafana’s Elasticsearch data source to visualize Elasticsearch data. If you are using an AWS Identity and Access Management (IAM) policy to control access to your Amazon Elasticsearch Service domain, then you must use AWS Signature Version 4 (AWS SigV4) to sign all requests to that domain. For more details on AWS SigV4, refer to the AWS documentation.

AWS Signature Version 4 authentication

Note: Only available in Grafana v7.3+.

In order to sign requests to your Amazon Elasticsearch Service domain, SigV4 can be enabled in the Grafana configuration.

Once AWS SigV4 is enabled, it can be configured on the Elasticsearch data source configuration page. Refer to Cloudwatch authentication for more information about authentication options.

Google Cloud Monitoring

Sat, 04 Apr 2026 12:26:57 +0000

Using Google Cloud Monitoring in Grafana

Grafana ships with built-in support for Google Cloud Monitoring. Add it as a data source to build dashboards for your Google Cloud Monitoring metrics. For instructions on how to add a data source, refer to Add a data source. Only users with the organization admin role can add data sources.

Note Before Grafana v7.1, Google Cloud Monitoring was referred to as Google Stackdriver.

Configure the Google Cloud Monitoring data source

To access Google Cloud Monitoring settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and click Add data source, then click the Google Cloud Monitoring data source.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source means that it is pre-selected for new panels.

For authentication options and configuration details, see the Google authentication documentation.

Google Cloud Monitoring specific data source configuration

The following APIs need to be enabled first:

Click on the links above and click the Enable button:

Using GCP Service Account Key File

The GCP Service Account must have the Monitoring Viewer role as shown in the image below:

Using GCE Default Service Account

If Grafana is running on a Google Compute Engine (GCE) virtual machine, the service account in use must have access to the Cloud Monitoring API scope.

Using the Query Editor

The Google Cloud Monitoring query editor allows you to build two types of queries - Metric and Service Level Objective (SLO). Both types return time series data.

Metric Queries

The metric query editor allows you to select metrics, group/aggregate by labels and by time, and use filters to specify which time series you want in the results.

To create a metric query, follow these steps:

Choose the option Metrics in the Query Type dropdown
Choose a project from the Project dropdown
Choose a Google Cloud Platform service from the Service dropdown
Choose a metric from the Metric dropdown.
Use the plus and minus icons in the filter and group by sections to add/remove filters or group by clauses. This step is optional.

Google Cloud Monitoring supports different kinds of metrics like GAUGE, DELTA, and CUMULATIVE. They support different aggregation options, for example, reducers and aligners. The Grafana query editor displays the list of available aggregation methods for a selected metric and sets a default reducer and aligner when you select the metric.

Filter

To add a filter, click the plus icon and choose a field to filter by and enter a filter value e.g. instance_name = grafana-1. You can remove the filter by clicking on the trash icon.

Simple wildcards

When the operator is set to = or != it is possible to add wildcards to the filter value field. E.g us-* will capture all values that starts with “us-” and *central-a will capture all values that ends with “central-a”. *-central-* captures all values that has the substring of -central-. Simple wildcards are less expensive than regular expressions.

Regular expressions

When the operator is set to =~ or !=~ it is possible to add regular expressions to the filter value field. E.g us-central[1-3]-[af] would match all values that starts with “us-central”, is followed by a number in the range of 1 to 3, a dash and then either an “a” or an “f”. Leading and trailing slashes are not needed when creating regular expressions.

Pre-processing

Preprocessing options are displayed in the UI when the selected metric has a metric kind of delta or cumulative. If the selected metric has a metric kind of gauge, no pre-processing option will be displayed.

If you select Rate, data points are aligned and converted to a rate per time series. If you select Delta, data points are aligned by their delta (difference) per time series.

Grouping

You can reduce the amount of data returned for a metric by combining different time series. To combine multiple time series, specify a grouping and a function.

Group by

Group by resource or metric labels to reduce the number of time series and to aggregate the results by a group. For example, group by instance_name to view an aggregated metric for a Compute instance.

Metadata labels

Resource metadata labels contain information that can uniquely identify a resource in Google Cloud. Metadata labels are only returned in the time series response if they’re part of the Group By segment in the time series request.

There’s no API for retrieving metadata labels. As a result, you cannot populate the group by list with the metadata labels that are available for the selected service and metric. However, the Group By field list comes with a pre-defined set of common system labels.

User labels cannot be predefined, but you can enter them manually in the Group By field. If a metadata label, user label, or system label is included in the Group By segment, then you can create filters based on it and expand its value on the Alias field.

Group by function

Select a grouping function to combine the time series in the group into a single time series.

Alignment

The process of alignment consists of collecting all data points received in a fixed length of time, applying a function to combine those data points, and assigning a timestamp to the result.

Alignment function

During alignment, all data points are received in a fixed interval. Within each interval (determined by the alignment period) and for each time series, the data is aggregated into a single point. The value of that point is determined by the type of alignment function used. For more information on alignment functions, refer to alignment metric selector.

Alignment period

The Alignment Period groups a metric by time if an aggregation is chosen. The default is to use the GCP Google Cloud Monitoring default groupings (which allows you to compare graphs in Grafana with graphs in the Google Cloud Monitoring UI). The option is called cloud monitoring auto and the defaults are:

1m for time ranges < 23 hours
5m for time ranges >= 23 hours and < 6 days
1h for time ranges >= 6 days

The other automatic option is grafana auto. This will automatically set the group by time depending on the time range chosen and the width of the time series panel. For more information about grafana auto, refer to the interval variable.

You can also choose fixed time intervals to group by, like 1h or 1d.

Alias patterns

The Alias By field allows you to control the format of the legend keys. The default is to show the metric name and labels. This can be long and hard to read. Using the following patterns in the alias field, you can format the legend key the way you want it.

Metric type patterns

Alias Pattern	Description	Example Result
`{{metric.type}}`	returns the full Metric Type	`compute.googleapis.com/instance/cpu/utilization`
`{{metric.name}}`	returns the metric name part	`instance/cpu/utilization`
`{{metric.service}}`	returns the service part	`compute`

Label patterns

In the Group By dropdown, you can see a list of metric and resource labels for a metric. These can be included in the legend key using alias patterns.

Alias Pattern Format	Description	Alias Pattern Example	Example Result
`{{metric.label.xxx}}`	returns the metric label value	`{{metric.label.instance_name}}`	`grafana-1-prod`
`{{resource.label.xxx}}`	returns the resource label value	`{{resource.label.zone}}`	`us-east1-b`
`{{metadata.system_labels.xxx}}`	returns the meta data system label value	`{{metadata.system_labels.name}}`	`grafana`
`{{metadata.user_labels.xxx}}`	returns the meta data user label value	`{{metadata.user_labels.tag}}`	`production`

Example Alias By: {{metric.type}} - {{metric.label.instance_name}}

Example Result: compute.googleapis.com/instance/cpu/usage_time - server1-prod

It is also possible to resolve the name of the Monitored Resource Type.

Alias Pattern Format	Description	Example Result
`{{resource.type}}`	returns the name of the monitored resource type	`gce_instance`

Example Alias By: {{resource.type}} - {{metric.type}}

Example Result: gce_instance - compute.googleapis.com/instance/cpu/usage_time

Deep linking from Grafana panels to the Metrics Explorer in Google Cloud Console

Note: Available in Grafana v7.1 and later versions.

Click on a time series in the panel to see a context menu with a link to View in Metrics Explorer in Google Cloud Console. Clicking that link opens the Metrics Explorer in the Google Cloud Console and runs the query from the Grafana panel there. The link navigates the user first to the Google Account Chooser and after successfully selecting an account, the user is redirected to the Metrics Explorer. The provided link is valid for any account, but it only displays the query if your account has access to the GCP project specified in the query.

Automatic unit detection

Grafana issues one query to the Cloud Monitoring API per query editor row, and each API response includes a unit. Grafana will attempt to convert the returned unit into a unit that is understood by the Grafana time series panel. If the conversion was successful, then the unit will be displayed on the Y-axis on the panel. If the query editor rows returned different units, then the unit from the last query editor row is used in the time series panel.

SLO (Service Level Objective) queries

Note: Available in Grafana v7.0 and later versions.

The SLO query builder in the Google Cloud Monitoring data source allows you to display SLO data in time series format. To get an understanding of the basic concepts in service monitoring, please refer to Google Cloud Monitoring’s official docs.

How to create an SLO query

To create an SLO query, follow these steps:

Choose the option Service Level Objectives (SLO) in the Query Type dropdown.
Choose a project from the Project dropdown.
Choose an SLO service from the Service dropdown.
Choose an SLO from the SLO dropdown.
Choose a time series selector from the Selector dropdown.

The friendly names for the time series selectors are shown in Grafana. Here is the mapping from the friendly name to the system name that is used in the Service Monitoring documentation:

Selector dropdown value	Corresponding time series selector used
SLI Value	select_slo_health
SLO Compliance	select_slo_compliance
SLO Error Budget Remaining	select_slo_budget_fraction

Alias patterns for SLO queries

The Alias By field allows you to control the format of the legend keys for SLO queries too.

Alias Pattern	Description	Example Result
`{{project}}`	returns the GCP project name	`myProject`
`{{service}}`	returns the service name	`myService`
`{{slo}}`	returns the SLO	`latency-slo`
`{{selector}}`	returns the selector	`select_slo_health`

Alignment period/group by time for SLO queries

SLO queries use the same alignment period functionality as metric queries.

MQL (Monitoring Query Language) queries

Note: Available in Grafana v7.4 and later versions.

The MQL query builder in the Google Cloud Monitoring data source allows you to display MQL results in time series format. To get an understanding of the basic concepts in MQL, refer to Introduction to Monitoring Query Language.

Create an MQL query

To create an MQL query, follow these steps:

In the Query Type list, select Metrics.
Click <> Edit MQL right next to the Query Type field. This will toggle the metric query builder mode so that raw MQL queries can be used.
Choose a project from the Project list.
Add the MQL query of your choice in the text area.

Alias patterns for MQL queries

MQL queries use the same alias patterns as metric queries.

{{metric.service}} is not supported. {{metric.type}} and {{metric.name}} show the time series key in the response.

Templating

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query Variable

Variable of the type Query allows you to query Google Cloud Monitoring for various types of data. The Google Cloud Monitoring data source plugin provides the following Query Types.

Name	Description
`Metric Types`	Returns a list of metric type names that are available for the specified service.
`Labels Keys`	Returns a list of keys for `metric label` and `resource label` in the specified metric.
`Labels Values`	Returns a list of values for the label in the specified metric.
`Resource Types`	Returns a list of resource types for the specified metric.
`Aggregations`	Returns a list of aggregations (cross series reducers) for the specified metric.
`Aligners`	Returns a list of aligners (per series aligners) for the specified metric.
`Alignment periods`	Returns a list of all alignment periods that are available in Google Cloud Monitoring query editor in Grafana
`Selectors`	Returns a list of selectors that can be used in SLO (Service Level Objectives) queries
`SLO Services`	Returns a list of Service Monitoring services that can be used in SLO queries
`Service Level Objectives (SLO)`	Returns a list of SLO’s for the specified SLO service

Using variables in queries

Refer to the variable syntax documentation.

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view. Annotation rendering is expensive so it is important to limit the number of rows returned. There is no support for showing Google Cloud Monitoring annotations and events yet but it works well with custom metrics in Google Cloud Monitoring.

With the query editor for annotations, you can select a metric and filters. The Title and Text fields support templating and can use data returned from the query. For example, the Title field could have the following text:

{{metric.type}} has value: {{metric.value}}

Example Result: monitoring.googleapis.com/uptime_check/http_status has this value: 502

Patterns for the Annotation Query Editor

Alias Pattern Format	Description	Alias Pattern Example	Example Result
`{{metric.value}}`	value of the metric/point	`{{metric.value}}`	`555`
`{{metric.type}}`	returns the full Metric Type	`{{metric.type}}`	`compute.googleapis.com/instance/cpu/utilization`
`{{metric.name}}`	returns the metric name part	`{{metric.name}}`	`instance/cpu/utilization`
`{{metric.service}}`	returns the service part	`{{metric.service}}`	`compute`
`{{metric.label.xxx}}`	returns the metric label value	`{{metric.label.instance_name}}`	`grafana-1-prod`
`{{resource.label.xxx}}`	returns the resource label value	`{{resource.label.zone}}`	`us-east1-b`

Configure the data source with provisioning

You can provision CloudWatch data source by modifying Grafana’s configuration files. To learn more about provisioning and all the settings you can set, see the provisioning documentation

Here is a provisioning example using the JWT (Service Account key file) authentication type.

apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      tokenUri: https://oauth2.googleapis.com/token
      clientEmail: stackdriver@myproject.iam.gserviceaccount.com
      authenticationType: jwt
      defaultProject: my-project-name
    secureJsonData:
      privateKey: |
        -----BEGIN PRIVATE KEY-----
        POSEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCb1u1Srw8ICYHS
        ...
        yA+23427282348234=
        -----END PRIVATE KEY-----

Here is a provisioning example using GCE Default Service Account authentication.

apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      authenticationType: gce

Graphite

Sat, 04 Apr 2026 12:26:57 +0000

Using Graphite in Grafana

Grafana has an advanced Graphite query editor that lets you quickly navigate the metric space, add functions, change function parameters and much more. The editor can handle all types of graphite queries. It can even handle complex nested queries through the use of query references.

Refer to Add a data source for instructions on how to add a data source to Grafana. Only organization admins can add data sources. To learn more about the Graphite data source, refer to Graphite’s product documentation.

Graphite settings

To access Graphite settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and then click the Graphite data source.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source means that it will be pre-selected for new panels.
`URL`	The HTTP protocol, IP, and port of your graphite-web or graphite-api install.
`Auth`	Refer to Authentication for more information.
`Basic Auth`	Enable basic authentication to the data source.
`User`	User name for basic authentication.
`Password`	Password for basic authentication.
`Custom HTTP Headers`	Click Add header to add a custom HTTP header.
`Header`	Enter the custom header name.
`Value`	Enter the custom header value.
`Graphite details`
`Version`	Select your version of Graphite.
`Type`	Select your type of Graphite.

Graphite query editor

Grafana includes a Graphite-specific query editor to help you build your queries.

To see the raw text of the query that is sent to Graphite, click the Toggle text edit mode (pencil) icon.

Choose metrics to query

Click Select metric to start navigating the metric space. Once you start, you can continue using the mouse or keyboard arrow keys. You can select a wildcard and still continue.

Functions

Click the plus icon next to Function to add a function. You can search for the function or select it from the menu. Once a function is selected, it will be added and your focus will be in the text box of the first parameter.

To edit or change a parameter, click on it and it will turn into a text box.
To delete a function, click the function name followed by the x icon.

Some functions like aliasByNode support an optional second argument. To add an argument, hover your mouse over the first argument and then click the + symbol that appears. To remove the second optional parameter, click on it and leave it blank and the editor will remove it.

To learn more, refer to Graphite’s documentation on functions.

Sort labels

If you want consistent ordering, use sortByName. This can be particularly annoying when you have the same labels on multiple graphs, and they are both sorted differently and using different colors. To fix this, use sortByName().

Nested queries

You can reference queries by the row “letter” that they’re on (similar to Microsoft Excel). If you add a second query to a graph, you can reference the first query simply by typing in #A. This provides an easy and convenient way to build compounded queries.

Avoiding many queries by using wildcards

Occasionally one would like to see multiple time series plotted on the same graph. For example we might want to see how the CPU is being utilized on a machine. You might initially create the graph by adding a query for each time series, such as cpu.percent.user.g, cpu.percent.system.g, and so on. This results in n queries made to the data source, which is inefficient.

To be more efficient one can use wildcards in your search, returning all the time series in one query. For example, cpu.percent.*.g.

Modify the metric name in my tables or charts

Use alias functions to change metric names on Grafana tables or graphs For example aliasByNode() or aliasSub().

Point consolidation

All Graphite metrics are consolidated so that Graphite doesn’t return more data points than there are pixels in the graph. By default, this consolidation is done using avg function. You can control how Graphite consolidates metrics by adding the Graphite consolidateBy function.

Note: This means that legend summary values (max, min, total) cannot all be correct at the same time. They are calculated client-side by Grafana. And depending on your consolidation function, only one or two can be correct at the same time.

Combine time series

To combine time series, click Combine in the Functions list.

Data exploration and tags

In Graphite, everything is a tag.

When exploring data, previously-selected tags are used to filter the remaining result set. To select data, you use the seriesByTag function, which takes tag expressions (=, !=, =~, !=~) to filter timeseries.

The Grafana query builder does this for you automatically when you select a tag.

Tip: The regular expression search can be quite slow on high-cardinality tags, so try to use other tags to reduce the scope first. Starting off with a particular name/namespace can help reduce the results.

Template variables

Instead of hard-coding things like server, application, and sensor name in your metric queries, you can use variables in their place. Variables are shown as drop-down select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.

For more information, refer to Variables and templates.

Graphite 1.1 introduced tags and Grafana added support for Graphite queries with tags in version 5.0. To create a variable using tag values, use the Grafana functions tags and tag_values.

Query	Description
`tags()`	Returns all tags.
`tags(server=~backend\*)`	Returns only tags that occur in series matching the filter expression.
`tag_values(server)`	Return tag values for the specified tag.
`tag_values(server, server=~backend\*)`	Returns filtered tag values that occur for the specified tag in series matching those expressions.
`tag_values(server, server=~backend\*, app=~${apps:regex})`	Multiple filter expressions and expressions can contain other variables.

For more details, see the Graphite docs on the autocomplete API for tags.

Query variable

The query you specify in the query field should be a metric find type of query. For example, a query like prod.servers.* fills the variable with all possible values that exist in the wildcard position.

The results contain all possible values occurring only at the last level of the query. To get full metric names matching the query use expand function (expand(*.servers.*)).

Comparison between expanded and non-expanded metric search results

The expanded query returns the full names of matching metrics. In combination with regex, it can extract any part of the metric name. By contrast, a non-expanded query only returns the last part of the metric name. It does not allow you to extract other parts of metric names.

Here are some example metrics:

prod.servers.001.cpu
prod.servers.002.cpu
test.servers.001.cpu

The following examples show how expanded and non-expanded queries can be used to fetch specific parts of the metrics name.

non-expanded query	results	expanded query	expanded results
`*`	prod, test	`expand(*)`	prod, test
`*.servers`	servers	`expand(*.servers)`	prod.servers, test.servers
`test.servers`	servers	`expand(test.servers)`	test.servers
`.servers.`	001,002	`expand(.servers.)`	prod.servers.001, prod.servers.002, test.servers.001
`test.servers.*`	001	`expand(test.servers.*)`	test.servers.001
`.servers..cpu`	cpu	`expand(.servers..cpu)`	prod.servers.001.cpu, prod.servers.002.cpu, test.servers.001.cpu

As you can see from the results, the non-expanded query is the same as an expanded query with a regex matching the last part of the name.

You can also create nested variables that use other variables in their definition. For example apps.$app.servers.* uses the variable $app in its query definition.

Using `__searchFilter` to filter query variable results

Available from Grafana 6.5 and above

Using __searchFilter in the query field will filter the query result based on what the user types in the dropdown select box. When nothing has been entered by the user the default value for __searchFilter is * and `` when used as part of a regular expression.

The example below shows how to use __searchFilter as part of the query field to enable searching for server while the user types in the dropdown select box.

Query

apps.$app.servers.$__searchFilter

TagValues

tag_values(server, server=~${__searchFilter:regex})

Variable usage

You can use a variable in a metric node path or as a parameter to a function.

There are two syntaxes:

$<varname> Example: apps.frontend.$server.requests.count
${varname} Example: apps.frontend.${server}.requests.count

Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. Use the second syntax in expressions like my.server${serverNumber}.count.

Example: Graphite Templated Dashboard

Variable usage in tag queries

Multi-value variables in tag queries use the advanced formatting syntax introduced in Grafana 5.0 for variables: {var:regex}. Non-tag queries will use the default glob formatting for multi-value variables.

Example of a tag expression with regex formatting and using the Equal Tilde operator, =~:

server=~${servers:regex}

For more information, refer to Advanced variable format options.

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.

Graphite supports two ways to query annotations. A regular metric query, for this you use the Graphite query textbox. A Graphite events query, use the Graphite event tags textbox, specify a tag or wildcard (leave empty should also work)

Get Grafana metrics into Graphite

Grafana exposes metrics for Graphite on the /metrics endpoint. For detailed instructions, refer to Internal Grafana metrics.

Configure the data source with provisioning

Here are some provisioning examples for this data source.

apiVersion: 1

datasources:
  - name: Graphite
    type: graphite
    access: proxy
    url: http://localhost:8080
    jsonData:
      graphiteVersion: '1.1'

Integration with Loki

Graphite queries get converted to Loki queries when the data source selection changes in Explore. Loki label names and values are extracted from the Graphite queries according to mappings information provided in Graphite data source configuration. Queries using tags with seriesByTags() are also transformed without any additional setup.

Refer to the Graphite data source settings for more details.

InfluxDB data source

Sat, 04 Apr 2026 12:26:57 +0000

InfluxDB data source

InfluxDB is an open-source time series database (TSDB) developed by InfluxData. It is optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, IoT sensor data, and real-time analytics.

This topic explains options, variables, querying, and other options specific to this data source. Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.

Data source options

To access data source settings, hover your mouse over the Configuration (gear) icon, then click Data sources, and then click the data source.

InfluxDB data source options differ depending on which query language you select: InfluxQL or Flux.

Note: Though not required, it’s a good practice to append the language choice to the data source name. For example:

InfluxDB-InfluxQL
InfluxDB-Flux

InfluxQL (classic InfluxDB query)

These options apply if you are using the InfluxQL query language. If you are using Flux, refer to Flux support in Grafana.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries. We recommend something like `InfluxDB-InfluxQL`.
`Default`	Default data source means that it will be pre-selected for new panels.
`URL`	The HTTP protocol, IP address and port of your InfluxDB API. InfluxDB API port is by default 8086.
`Access`	Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. Note: Browser (direct) access is deprecated and will be removed in a future release.
`Allowed cookies`	Cookies that will be forwarded to the data source. All other cookies will be deleted.
`Database`	The ID of the bucket you want to query from, copied from the Buckets page of the InfluxDB UI.
`User`	The username you use to sign into InfluxDB.
`Password`	The token you use to query the bucket above, copied from the Tokens page of the InfluxDB UI.
`HTTP mode`	How to query the database (`GET` or `POST` HTTP verb). The `POST` verb allows heavy queries that would return an error using the `GET` verb. Default is `GET`.
`Min time interval`	(Optional) Refer to Min time interval.
`Max series`	(Optional) Limits the number of series/tables that Grafana processes. Lower this number to prevent abuse, and increase it if you have lots of small time series and not all are shown. Defaults to 1000.

Flux

For information on data source settings and using Flux in Grafana, refer to Flux support in Grafana.

Min time interval

A lower limit for the auto group by time interval. Recommended to be set to write frequency, for example 1m if your data is written every minute. This option can also be overridden/configured in a dashboard panel under data source options. It’s important to note that this value must be formatted as a number followed by a valid time identifier, e.g. 1m (1 minute) or 30s (30 seconds). The following time identifiers are supported:

Identifier	Description
`y`	year
`M`	month
`w`	week
`d`	day
`h`	hour
`m`	minute
`s`	second
`ms`	millisecond

Query languages

You can query InfluxDB using InfluxQL or Flux:

InfluxQL is a SQL-like language for querying InfluxDB, with statements such as SELECT, FROM, WHERE, and GROUP BY that are familiar to SQL users. InfluxQL is available in InfluxDB 1.0 onwards.
Flux provides significantly broader functionality than InfluxQL, supporting not only queries, but built-in functions for data shaping, string manipulation, joining to non-InfluxDB data sources and more, but also processing time-series data. It’s more similar to JavaScript with a functional style.

To help you choose the best language for your needs, here’s a comparison of Flux vs InfluxQL, and why InfluxData created Flux.

InfluxQL query editor

Enter edit mode by clicking the panel title and clicking Edit. The editor allows you to select metrics and tags.

Filter data (WHERE)

To add a tag filter, click the plus icon to the right of the WHERE condition. You can remove tag filters by clicking on the tag key and then selecting --remove tag filter--.

Regex matching

You can type in regex patterns for metric names or tag filter values. Be sure to wrap the regex pattern in forward slashes (/). Grafana automatically adjusts the filter tag condition to use the InfluxDB regex match condition operator (=~).

Field and Aggregation functions

In the SELECT row you can specify what fields and functions you want to use. If you have a group by time you need an aggregation function. Some functions like derivative require an aggregation function. The editor tries to simplify and unify this part of the query. For example:

The above generates the following InfluxDB SELECT clause:

SELECT derivative(mean("value"), 10s) /10 AS "REQ/s" FROM ....

Select multiple fields

Use the plus button and select Field > field to add another SELECT clause. You can also specify an asterix * to select all fields.

Group By

To group by a tag, click the plus icon at the end of the GROUP BY row. Pick a tag from the dropdown that appears. You can remove the “Group By” by clicking on the tag and then click on the x icon.

Text Editor Mode (RAW)

You can switch to raw query mode by clicking hamburger icon and then Switch editor mode.

If you use Raw Query be sure your query at minimum have WHERE $timeFilter. Also, always have a group by time and an aggregation function, otherwise InfluxDB can easily return hundreds of thousands of data points that will hang the browser.

Alias patterns

$m = replaced with measurement name
$measurement = replaced with measurement name
$1 - $9 = replaced with part of measurement name (if you separate your measurement name with dots)
$col = replaced with column name
$tag_exampletag = replaced with the value of the exampletag tag. The syntax is $tag*yourTagName(must start with$tag*). To use your tag as an alias in the ALIAS BY field then the tag must be used to group by in the query.
You can also use [[tag_hostname]] pattern replacement syntax. For example, in the ALIAS BY field using this text Host: [[tag_hostname]] would substitute in the hostname tag value for each legend value and an example legend value would be: Host: server1.

Querying logs

Querying and displaying log data from InfluxDB is available in Explore, and in the logs panel in dashboards. Select the InfluxDB data source, and then enter a query to display your logs.

Log queries

The Logs Explorer (the Measurements/Fields button) next to the query field shows a list of measurements and fields. Choose the desired measurement that contains your log data and then choose which field Explore should use to display the log message.

Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.

Filter search

To add a filter, click the plus icon to the right of the Measurements/Fields button or a condition. You can remove tag filters by clicking on the first select and choosing --remove filter--.

Annotations

Annotations allows you to overlay rich event information on top of graphs. Add annotation queries using the Annotations view in the Dashboard menu.

An example query:

SELECT title, description from events WHERE $timeFilter ORDER BY time ASC

For InfluxDB, you need to enter a query like the one in the example above. The where $timeFilter component is required. If you only select one column, then you do not need to enter anything in the column mapping fields. The Tags field can be a comma-separated string.

Jaeger

Sat, 04 Apr 2026 12:26:57 +0000

Jaeger data source

Grafana ships with built-in support for Jaeger, which provides open source, end-to-end distributed tracing. Just add it as a data source and you are ready to query your traces in Explore.

Add data source

To access Jaeger settings, click the Configuration (gear) icon, then click Data Sources > Jaeger.

Name	Description
`Name`	The data source name in panels, queries, and Explore.
`Default`	The pre-selected data source for a new panel.
`URL`	The URL of the Jaeger instance. For example, `http://localhost:16686`.
`Basic Auth`	Enable basic authentication for the Jaeger data source.
`User`	Specify a user name for basic authentication.
`Password`	Specify a password for basic authentication.

Trace to logs

Note: This feature is available in Grafana 7.4+.

This is a configuration for the trace to logs feature. Select target data source (at this moment limited to Loki data sources) and select which tags will be used in the logs query.

Data source - Target data source.
Tags - The tags that will be used in the Loki query. Default is 'cluster', 'hostname', 'namespace', 'pod'.
Map tag names - When enabled, allows configuring how Jaeger tag names map to Loki label names. For example, map service.name to service.
Span start time shift - Shift in the start time for the Loki query based on the span start time. In order to extend to the past, you need to use a negative value. Use time interval units like 5s, 1m, 3h. The default is 0.
Span end time shift - Shift in the end time for the Loki query based on the span end time. Time units can be used here, for example, 5s, 1m, 3h. The default is 0.
Filter by Trace ID - Toggle to append the trace ID to the Loki query.
Filter by Span ID - Toggle to append the span ID to the Loki query.

Node Graph

This is a configuration for the beta Node Graph visualization. The Node Graph is shown after the trace view is loaded and is disabled by default.

– Enable Node Graph - Enables the Node Graph visualization.

Query traces

You can query and display traces from Jaeger via Explore.

You can query by trace ID or use the search form to find traces. To query by trace ID, select the TraceID from the Query type selector and insert the ID into the text input.

To perform a search, set the query type selector to Search, then use the following fields to find traces:

Service - Returns a list of services.
Operation - Field gets populated once you select a service. It then lists the operations related to the selected service. Select All option to query all operations.
Tags - Use values in the logfmt format. For example error=true db.statement="select * from User".
Min Duration - Filter all traces with a duration higher than the set value. Possible values are 1.2s, 100ms, 500us.
Max Duration - Filter all traces with a duration lower than the set value. Possible values are 1.2s, 100ms, 500us.
Limit - Limits the number of traces returned.

Upload JSON trace file

You can upload a JSON file that contains a single trace to visualize it. If the file has multiple traces then the first trace is used for visualization.

Here is an example JSON:

{
  "data": [
    {
      "traceID": "2ee9739529395e31",
      "spans": [
        {
          "traceID": "2ee9739529395e31",
          "spanID": "2ee9739529395e31",
          "flags": 1,
          "operationName": "CAS",
          "references": [],
          "startTime": 1616095319593196,
          "duration": 1004,
          "tags": [
            {
              "key": "sampler.type",
              "type": "string",
              "value": "const"
            }
          ],
          "logs": [],
          "processID": "p1",
          "warnings": null
        }
      ],
      "processes": {
        "p1": {
          "serviceName": "loki-all",
          "tags": [
            {
              "key": "jaeger.version",
              "type": "string",
              "value": "Go-2.25.0"
            }
          ]
        }
      },
      "warnings": null
    }
  ],
  "total": 0,
  "limit": 0,
  "offset": 0,
  "errors": null
}

Linking Trace ID from logs

You can link to Jaeger trace from logs in Loki by configuring a derived field with internal link. See the Derived fields section in the Loki data source documentation for details.

Configure the data source with provisioning

You can set up the data source via configuration files with Grafana’s provisioning system. Refer to provisioning docs page for more information on configuring various settings.

Here is an example with basic auth and trace-to-logs field.

apiVersion: 1

datasources:
  - name: Jaeger
    type: jaeger
    uid: jaeger-spectra
    access: proxy
    url: http://localhost:16686/
    basicAuth: true
    basicAuthUser: my_user
    editable: true
    isDefault: false
    jsonData:
      tracesToLogs:
        # Field with internal link pointing to a Loki data source in Grafana.
        # datasourceUid value must match the `datasourceUid` value of the Loki data source.
        datasourceUid: 'loki'
        tags: ['job', 'instance', 'pod', 'namespace']
        mappedTags: [{ key: 'service.name', value: 'service' }]
        mapTagNamesEnabled: false
        spanStartTimeShift: '1h'
        spanEndTimeShift: '1h'
        filterByTraceID: false
        filterBySpanID: false
        lokiSearch: true
    secureJsonData:
      basicAuthPassword: my_password

Loki

Sat, 04 Apr 2026 12:26:57 +0000

Using Loki in Grafana

Grafana ships with built-in support for Loki, an open source log aggregation system by Grafana Labs. This topic explains options, variables, querying, and other options specific to this data source.

Add it as a data source and you are ready to build dashboards or query your log data in Explore. Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.

Hosted Loki

You can run Loki on your own hardware or use Grafana Cloud. The free forever plan includes Grafana, 50 GB of Loki logs, 10K Prometheus series, and more. Create a free account to get started.

Loki settings

To access Loki settings, click the Configuration (gear) icon, then click Data Sources, and then click the Loki data source.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels, queries, and Explore.
`Default`	Default data source that is pre-selected for new panels.
`URL`	URL of the Loki instance, e.g., `http://localhost:3100`.
`Allowed cookies`	Grafana Proxy deletes forwarded cookies by default. Specify cookies by name that should be forwarded to the data source.
`Maximum lines`	Upper limit for the number of log lines returned by Loki (default is 1000). Lower this limit if your browser is sluggish when displaying logs in Explore.

Note: To troubleshoot configuration and other issues, check the log file located at /var/log/grafana/grafana.log on Unix systems or in <grafana_install_dir>/data/log on other platforms and manual installations.

Derived fields

The Derived Fields configuration allows you to:

Add fields parsed from the log message.
Add a link that uses the value of the field.

For example, you can use this functionality to link to your tracing backend directly from your logs, or link to a user profile page if a userId is present in the log line. These links appear in the log details.

Each derived field consists of:

Name - Shown in the log details as a label.
Regex - A Regex pattern that runs on the log message and captures part of it as the value of the new field. Can only contain a single capture group.
URL/query - If the link is external, then enter the full link URL. If the link is internal link, then this input serves as query for the target data source. In both cases, you can interpolate the value from the field with ${__value.raw} macro.
URL Label - (Optional) Set a custom display label for the link. The link label defaults to the full external URL or name of the linked internal data source and is overridden by this setting.
Internal link - Select if the link is internal or external. In case of internal link, a data source selector allows you to select the target data source. Only tracing data sources are supported.

You can use a debug section to see what your fields extract and how the URL is interpolated. Click Show example log message to show the text area where you can enter a log message.

The new field with the link shown in log details:

Loki query editor

You can use the Loki query editor to create log and metric queries.

Name	Description
`Query expression`	Loki query expression, refer to the LogQL documentation for more information.
`Query type`	Choose the type of query to run. The instant type queries against a single point in time. We are using “To” time from the time range. The range type queries over the selected range of time.
`Line limit`	Upper limit for number of log lines returned by query. The default is the Maximum lines limit set in Loki settings.
`Legend`	Available only in Dashboard. Controls the name of the time series, using name or pattern. For example `{{hostname}}` is replaced with the label value for the label `hostname`.

Log browser

With Loki log browser you can easily navigate through your list of labels and values and construct the query of your choice. Log browser has multi-step selection:

Choose the labels you would like to consider for your search.
Pick the values for selected labels. Log browser supports facetting and therefore it shows you only possible label combinations.
Choose the type of query - logs query or rate metrics query. Additionally, you can also validate selector.

Querying with Loki

There are two types of LogQL queries:

Log queries - Return the contents of log lines.
Metric queries - Extend log queries and calculate sample values based on the content of logs from a log query.

Log queries

Querying and displaying log data from Loki is available via Explore, and with the logs panel in dashboards. Select the Loki data source, and then enter a LogQL query to display your logs.

A log query consists of two parts: log stream selector, and a log pipeline. For performance reasons begin by choosing a log stream by selecting a log label.

Log context

When using a search expression as detailed above, you can retrieve the context surrounding your filtered results. By clicking the Show Context link on the filtered rows, you’ll be able to investigate the log messages that came before and after the log message you’re interested in.

Live tailing

Loki supports Live tailing which displays logs in real-time. This feature is supported in Explore.

Note that Live Tailing relies on two Websocket connections: one between the browser and the Grafana server, and another between the Grafana server and the Loki server. If you run any reverse proxies, please configure them accordingly. The following example for Apache2 can be used for proxying between the browser and the Grafana server:

ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"

The following example shows basic NGINX proxy configuration. It assumes that the Grafana server is available at http://localhost:3000/, Loki server is running locally without proxy, and your external site uses HTTPS. If you also host Loki behind NGINX proxy, then you might want to repeat the following configuration for Loki as well.

In the http section of NGINX configuration, add the following map definition:

  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }

In your server section, add the following configuration:

  location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) {
      proxy_pass          http://localhost:3000$request_uri;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
      proxy_set_header    Connection        $connection_upgrade;
      proxy_set_header    Upgrade           $http_upgrade;
  }

  location / {
      proxy_pass          http://localhost:3000/;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
  }

Note: This feature is only available in Grafana v6.3+.

Metric queries

LogQL supports wrapping a log query with functions that allow for creating metrics out of the logs. See LogQL documentation on how to create and use metrics queries.

Templating

Instead of hard-coding things like server, application and sensor name in your metric queries, you can use variables in their place. Variables are shown as drop-down select boxes at the top of the dashboard. These drop-down boxes make it easy to change the data being displayed in your dashboard.

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query variable

Variable of the type Query allows you to query Loki for a list labels or label values. The Loki data source plugin provides the following functions you can use in the Query input field.

Name	Description
`label_names()`	Returns a list of label names.
`label_values(label)`	Returns a list of label values for the `label`.
`label_values(log stream selector, label)`	Returns a list of label values for the `label` in the specified `log stream selector`.

Ad hoc filters variable

Loki supports the special ad hoc filters variable type. It allows you to specify any number of label/value filters on the fly. These filters are automatically applied to all your Loki queries.

Using interval and range variables

You can use some global built-in variables in query variables; $__interval, $__interval_ms, $__range, $__range_s and $__range_ms. For more information, refer to Global built-in variables.

Annotations

You can use any non-metric Loki query as a source for annotations. Log content will be used as annotation text and your log stream labels as tags, so there is no need for additional mapping.

Configure the data source with provisioning

You can set up the data source via config files with Grafana’s provisioning system. You can read more about how it works and all the settings you can set for data sources on the provisioning docs page

Here is an example:

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      maxLines: 1000

Here’s another with basic auth and derived field. Keep in mind that $ character needs to be escaped in YAML values as it is used to interpolate environment variables:

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    basicAuth: true
    basicAuthUser: my_user
    basicAuthPassword: test_password
    jsonData:
      maxLines: 1000
      derivedFields:
        # Field with internal link pointing to data source in Grafana.
        # Right now, Grafana supports only Jaeger and Zipkin data sources as link targets.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          matcherRegex: "traceID=(\\w+)"
          name: TraceID
          # url will be interpreted as query for the datasource
          url: '$${__value.raw}'

        # Field with external link.
        - matcherRegex: "traceID=(\\w+)"
          name: TraceID
          url: 'http://localhost:16686/trace/$${__value.raw}'

Here’s an example of a Jaeger data source corresponding to the above example. Note that the Jaeger uid value does match the Loki datasourceUid value.

datasources:
    - name: Jaeger
      type: jaeger
      url: http://jaeger-tracing-query:16686/
      access: proxy
      # UID should match the datasourceUid in dervidedFields.
      uid: my_jaeger_uid

Microsoft SQL Server

Sat, 04 Apr 2026 12:26:57 +0000

Using Microsoft SQL Server in Grafana

Grafana ships with a built-in Microsoft SQL Server (MS SQL) data source plugin that allows you to query and visualize data from any Microsoft SQL Server 2005 or newer, including Microsoft Azure SQL Database. This topic explains options, variables, querying, and other options specific to the MS SQL data source. Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.

Data source options

To access data source settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and then click the data source.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source means that it will be pre-selected for new panels.
`Host`	The IP address/hostname and optional port of your MS SQL instance. If you omit the port, then the driver default is used (0). You can specify multiple connection properties such as ApplicationIntent using ‘;’ character to separate each property.
`Database`	Name of your MS SQL database.
`Authentication`	Authentication mode. Either using SQL Server Authentication or Windows Authentication (single sign on for Windows users).
`User`	Database user’s login/username
`Password`	Database user’s password
`Encrypt`	This option determines whether or to which extent a secure SSL TCP/IP connection will be negotiated with the server, default `false`.
`Max open`	The maximum number of open connections to the database, default `unlimited`.
`Max idle`	The maximum number of connections in the idle connection pool, default `2`.
`Max lifetime`	The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours.

Min time interval

A lower limit for the $__interval and $__interval_ms variables. Recommended to be set to write frequency, for example 1m if your data is written every minute. This option can also be overridden/configured in a dashboard panel under data source options. It’s important to note that this value needs to be formatted as a number followed by a valid time identifier, e.g. 1m (1 minute) or 30s (30 seconds). The following time identifiers are supported:

Identifier	Description
`y`	year
`M`	month
`w`	week
`d`	day
`h`	hour
`m`	minute
`s`	second
`ms`	millisecond

Database user permissions

The database user you specify when you add the data source should only be granted SELECT permissions on the specified database and tables you want to query. Grafana does not validate that the query is safe. The query could include any SQL statement. For example, statements like DELETE FROM user; and DROP TABLE user; would be executed. To protect against this we highly recommend you create a specific MS SQL user with restricted permissions.

Example:

 CREATE USER grafanareader WITH PASSWORD 'password'
 GRANT SELECT ON dbo.YourTable3 TO grafanareader

Make sure the user does not get any unwanted privileges from the public role.

Known Issues

If you’re using an older version of Microsoft SQL Server like 2008 and 2008R2 you may need to disable encryption to be able to connect. If possible, we recommend you to use the latest service pack available for optimal compatibility.

Query Editor

You will find the MSSQL query editor in the metrics tab in Graph, Singlestat or Table panel’s edit mode. You enter edit mode by clicking the panel title, then edit. The editor allows you to define a SQL query to select data to be visualized.

Select Format as Time series (for use in Graph or Singlestat panel’s among others) or Table (for use in Table panel among others).
This is the actual editor where you write your SQL queries.
Show help section for MSSQL below the query editor.
Show actual executed SQL query. Will be available first after a successful query has been executed.
Add an additional query where an additional query editor will be displayed.

Macros

To simplify syntax and to allow for dynamic parts, like date range filters, the query can contain macros.

Macro example	Description
`$__time(dateColumn)`	Will be replaced by an expression to rename the column to time. For example, dateColumn as time
`$__timeEpoch(dateColumn)`	Will be replaced by an expression to convert a DATETIME column type to Unix timestamp and rename it to time. For example, DATEDIFF(second, ‘1970-01-01’, dateColumn) AS time
`$__timeFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name. For example, dateColumn BETWEEN ‘2017-04-21T05:01:17Z’ AND ‘2017-04-21T05:06:17Z’
`$__timeFrom()`	Will be replaced by the start of the currently active time selection. For example, ‘2017-04-21T05:01:17Z’
`$__timeTo()`	Will be replaced by the end of the currently active time selection. For example, ‘2017-04-21T05:06:17Z’
`$__timeGroup(dateColumn,'5m'[, fillvalue])`	Will be replaced by an expression usable in GROUP BY clause. Providing a fillValue of NULL or floating value will automatically fill empty series in timerange with that value. For example, CAST(ROUND(DATEDIFF(second, ‘1970-01-01’, time_column)/300.0, 0) as bigint)300*.
`$__timeGroup(dateColumn,'5m', 0)`	Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
`$__timeGroup(dateColumn,'5m', NULL)`	Same as above but NULL will be used as value for missing points.
`$__timeGroup(dateColumn,'5m', previous)`	Same as above but the previous value in that series will be used as fill value if no value has been seen yet NULL will be used (only available in Grafana 5.3+).
`$__timeGroupAlias(dateColumn,'5m')`	Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
`$__unixEpochFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name with times represented as Unix timestamp. For example, dateColumn > 1494410783 AND dateColumn < 1494497183
`$__unixEpochFrom()`	Will be replaced by the start of the currently active time selection as Unix timestamp. For example, 1494410783
`$__unixEpochTo()`	Will be replaced by the end of the currently active time selection as Unix timestamp. For example, 1494497183
`$__unixEpochNanoFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name with times represented as nanosecond timestamp. For example, dateColumn > 1494410783152415214 AND dateColumn < 1494497183142514872
`$__unixEpochNanoFrom()`	Will be replaced by the start of the currently active time selection as nanosecond timestamp. For example, 1494410783152415214
`$__unixEpochNanoTo()`	Will be replaced by the end of the currently active time selection as nanosecond timestamp. For example, 1494497183142514872
`$__unixEpochGroup(dateColumn,'5m', [fillmode])`	Same as $__timeGroup but for times stored as Unix timestamp (only available in Grafana 5.3+).
`$__unixEpochGroupAlias(dateColumn,'5m', [fillmode])`	Same as above but also adds a column alias (only available in Grafana 5.3+).

We plan to add many more macros. If you have suggestions for what macros you would like to see, please open an issue in our GitHub repo.

The query editor has a link named Generated SQL that shows up after a query has been executed, while in panel edit mode. Click on it and it will expand and show the raw interpolated SQL string that was executed.

Table queries

If the Format as query option is set to Table then you can basically do any type of SQL query. The table panel will automatically show the results of whatever columns and rows your query returns.

Example database table:

CREATE TABLE [event] (
  time_sec bigint,
  description nvarchar(100),
  tags nvarchar(100),
)

CREATE TABLE [mssql_types] (
  c_bit bit, c_tinyint tinyint, c_smallint smallint, c_int int, c_bigint bigint, c_money money, c_smallmoney smallmoney, c_numeric numeric(10,5),
  c_real real, c_decimal decimal(10,2), c_float float,
  c_char char(10), c_varchar varchar(10), c_text text,
  c_nchar nchar(12), c_nvarchar nvarchar(12), c_ntext ntext,
  c_datetime datetime,  c_datetime2 datetime2, c_smalldatetime smalldatetime, c_date date, c_time time, c_datetimeoffset datetimeoffset
)

INSERT INTO [mssql_types]
SELECT
  1, 5, 20020, 980300, 1420070400, '$20000.15', '£2.15', 12345.12,
  1.11, 2.22, 3.33,
  'char10', 'varchar10', 'text',
  N'☺nchar12☺', N'☺nvarchar12☺', N'☺text☺',
  GETDATE(), CAST(GETDATE() AS DATETIME2), CAST(GETDATE() AS SMALLDATETIME), CAST(GETDATE() AS DATE), CAST(GETDATE() AS TIME), SWITCHOFFSET(CAST(GETDATE() AS DATETIMEOFFSET), '-07:00')

Query editor with example query:

The query:

SELECT * FROM [mssql_types]

You can control the name of the Table panel columns by using regular AS SQL column selection syntax. Example:

SELECT
  c_bit as [column1], c_tinyint as [column2]
FROM
  [mssql_types]

The resulting table panel:

Time series queries

If you set Format as to Time series, then the query must have a column named time that returns either a SQL datetime or any numeric datatype representing Unix epoch in seconds. In addition, result sets of time series queries must be sorted by time for panels to properly visualize the result.

A time series query result is returned in a wide data frame format. Any column except time or of type string transforms into value fields in the data frame query result. Any string column transforms into field labels in the data frame query result.

For backward compatibility, there’s an exception to the above rule for queries that return three columns including a string column named metric. Instead of transforming the metric column into field labels, it becomes the field name, and then the series name is formatted as the value of the metric column. See the example with the metric column below.

You can optionally customize the default series name formatting using instructions in Reference: Standard field definitions.

Example with metric column:

SELECT
  $__timeGroup(time_date_time, '5m') as time,
  min("value_double"),
  'min' as metric
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY $__timeGroup(time_date_time, '5m')
ORDER BY 1

Data frame result:

+---------------------+-----------------+
| Name: time          | Name: min       |
| Labels:             | Labels:         |
| Type: []time.Time   | Type: []float64 |
+---------------------+-----------------+
| 2020-01-02 03:05:00 | 3               |
| 2020-01-02 03:10:00 | 6               |
+---------------------+-----------------+

Example using the fill parameter in the $__timeGroup macro to convert null values to be zero instead:

SELECT
  $__timeGroup(createdAt, '5m', 0) as time,
  sum(value) as value,
  hostname
FROM test_data
WHERE
  $__timeFilter(createdAt)
GROUP BY
  $__timeGroup(createdAt, '5m', 0),
  hostname
ORDER BY 1

Given the data frame result in the following example and using the graph panel, you will get two series named value 10.0.1.1 and value 10.0.1.2. To render the series with a name of 10.0.1.1 and 10.0.1.2 , use a Reference: Standard field definitions display name value of ${__field.labels.hostname}.

Data frame result:

+---------------------+---------------------------+---------------------------+
| Name: time          | Name: value               | Name: value               |
| Labels:             | Labels: hostname=10.0.1.1 | Labels: hostname=10.0.1.2 |
| Type: []time.Time   | Type: []float64           | Type: []float64           |
+---------------------+---------------------------+---------------------------+
| 2020-01-02 03:05:00 | 3                         | 4                         |
| 2020-01-02 03:10:00 | 6                         | 7                         |
+---------------------+---------------------------+---------------------------+

Example with multiple columns:

SELECT
  $__timeGroup(time_date_time, '5m'),
  min(value_double) as min_value,
  max(value_double) as max_value
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY $__timeGroup(time_date_time, '5m')
ORDER BY 1

Data frame result:

+---------------------+-----------------+-----------------+
| Name: time          | Name: min_value | Name: max_value |
| Labels:             | Labels:         | Labels:         |
| Type: []time.Time   | Type: []float64 | Type: []float64 |
+---------------------+-----------------+-----------------+
| 2020-01-02 03:04:00 | 3               | 4               |
| 2020-01-02 03:05:00 | 6               | 7               |
+---------------------+-----------------+-----------------+

Templating

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query variable

If you add a template variable of the type Query, you can write a MS SQL query that can return things like measurement names, key names or key values that are shown as a dropdown select box.

For example, you can have a variable that contains all values for the hostname column in a table if you specify a query like this in the templating variable Query setting.

SELECT hostname FROM host

A query can return multiple columns and Grafana will automatically create a list from them. For example, the query below will return a list with values from hostname and hostname2.

SELECT [host].[hostname], [other_host].[hostname2] FROM host JOIN other_host ON [host].[city] = [other_host].[city]

Another option is a query that can create a key/value variable. The query should return two columns that are named __text and __value. The __text column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allow you to have a friendly name as text and an id as the value. An example query with hostname as the text and id as the value:

SELECT hostname __text, id __value FROM host

You can also create nested variables. For example, if you had another variable named region. Then you could have the hosts variable only show hosts from the current selected region with a query like this (if region is a multi-value variable, then use the IN comparison operator rather than = to match against multiple values):

SELECT hostname FROM host WHERE region IN ($region)

Using Variables in Queries

From Grafana 4.3.0 to 4.6.0, template variables are always quoted automatically so if it is a string value do not wrap them in quotes in where clauses.

From Grafana 5.0.0, template variable values are only quoted when the template variable is a multi-value.

If the variable is a multi-value variable then use the IN comparison operator rather than = to match against multiple values.

There are two syntaxes:

$<varname> Example with a template variable named hostname:

SELECT
  atimestamp time,
  aint value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp

[[varname]] Example with a template variable named hostname:

SELECT
  atimestamp as time,
  aint as value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp

Disabling Quoting for Multi-value Variables

Grafana automatically creates a quoted, comma-separated string for multi-value variables. For example: if server01 and server02 are selected then it will be formatted as: 'server01', 'server02'. Do disable quoting, use the csv formatting option for variables:

${servers:csv}

Read more about variable formatting options in the Variables documentation.

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.

Columns:

Name	Description
`time`	The name of the date/time field. Could be a column with a native SQL date/time data type or epoch value.
`timeend`	Optional name of the end date/time field. Could be a column with a native SQL date/time data type or epoch value. (Grafana v6.6+)
`text`	Event description field.
`tags`	Optional field name to use for event tags as a comma separated string.

Example database tables:

CREATE TABLE [events] (
  time_sec bigint,
  description nvarchar(100),
  tags nvarchar(100),
)

We also use the database table defined in Time series queries.

Example query using time column with epoch values:

SELECT
  time_sec as time,
  description as [text],
  tags
FROM
  [events]
WHERE
  $__unixEpochFilter(time_sec)
ORDER BY 1

Example region query using time and timeend columns with epoch values:

Only available in Grafana v6.6+.

SELECT
  time_sec as time,
  time_end_sec as timeend,
  description as [text],
  tags
FROM
  [events]
WHERE
  $__unixEpochFilter(time_sec)
ORDER BY 1

Example query using time column of native SQL date/time data type:

SELECT
  time,
  measurement as text,
  convert(varchar, valueOne) + ',' + convert(varchar, valueTwo) as tags
FROM
  metric_values
WHERE
  $__timeFilter(time_column)
ORDER BY 1

Stored procedure support

Stored procedures have been verified to work. However, please note that we haven’t done anything special to support this, so there might be edge cases where it won’t work as you would expect. Stored procedures should be supported in table, time series and annotation queries as long as you use the same naming of columns and return data in the same format as describe above under respective section.

Please note that any macro function will not work inside a stored procedure.

Examples

For the following examples, the database table is defined in Time series queries. Let’s say that we want to visualize four series in a graph panel, such as all combinations of columns valueOne, valueTwo and measurement. Graph panel to the right visualizes what we want to achieve. To solve this, we need to use two queries:

First query:

SELECT
  $__timeGroup(time, '5m') as time,
  measurement + ' - value one' as metric,
  avg(valueOne) as valueOne
FROM
  metric_values
WHERE
  $__timeFilter(time)
GROUP BY
  $__timeGroup(time, '5m'),
  measurement
ORDER BY 1

Second query:

SELECT
  $__timeGroup(time, '5m') as time,
  measurement + ' - value two' as metric,
  avg(valueTwo) as valueTwo
FROM
  metric_values
GROUP BY
  $__timeGroup(time, '5m'),
  measurement
ORDER BY 1

Stored procedure using time in epoch format

We can define a stored procedure that will return all data we need to render 4 series in a graph panel like above. In this case the stored procedure accepts two parameters @from and @to of int data types which should be a timerange (from-to) in epoch format which will be used to filter the data to return from the stored procedure.

We’re mimicking the $__timeGroup(time, '5m') in the select and group by expressions, and that’s why there are a lot of lengthy expressions needed - these could be extracted to MS SQL functions, if wanted.

CREATE PROCEDURE sp_test_epoch(
  @from int,
  @to 	int
)	AS
BEGIN
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time,
    measurement + ' - value one' as metric,
    avg(valueOne) as value
  FROM
    metric_values
  WHERE
    time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01')
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int),
    measurement
  UNION ALL
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time,
    measurement + ' - value two' as metric,
    avg(valueTwo) as value
  FROM
    metric_values
  WHERE
    time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01')
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int),
    measurement
  ORDER BY 1
END

Then we can use the following query for our graph panel.

DECLARE
  @from int = $__unixEpochFrom(),
  @to int = $__unixEpochTo()

EXEC dbo.sp_test_epoch @from, @to

Stored procedure using time in datetime format

We can define a stored procedure that will return all data we need to render 4 series in a graph panel like above. In this case the stored procedure accepts two parameters @from and @to of datetime data types which should be a timerange (from-to) which will be used to filter the data to return from the stored procedure.

We’re mimicking the $__timeGroup(time, '5m') in the select and group by expressions and that’s why there’s a lot of lengthy expressions needed - these could be extracted to MS SQL functions, if wanted.

CREATE PROCEDURE sp_test_datetime(
  @from datetime,
  @to 	datetime
)	AS
BEGIN
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time,
    measurement + ' - value one' as metric,
    avg(valueOne) as value
  FROM
    metric_values
  WHERE
    time >= @from AND time <= @to
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int),
    measurement
  UNION ALL
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time,
    measurement + ' - value two' as metric,
    avg(valueTwo) as value
  FROM
    metric_values
  WHERE
    time >= @from AND time <= @to
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int),
    measurement
  ORDER BY 1
END

Then we can use the following query for our graph panel.

DECLARE
  @from datetime = $__timeFrom(),
  @to datetime = $__timeTo()

EXEC dbo.sp_test_datetime @from, @to

Alerting

Time series queries should work in alerting conditions. Table formatted queries are not yet supported in alert rule conditions.

Configure the data source with provisioning

Here are some provisioning examples for this data source.

apiVersion: 1

datasources:
  - name: MSSQL
    type: mssql
    url: localhost:1433
    database: grafana
    user: grafana
    jsonData:
      maxOpenConns: 0 # Grafana v5.4+
      maxIdleConns: 2 # Grafana v5.4+
      connMaxLifetime: 14400 # Grafana v5.4+
    secureJsonData:
      password: 'Password!'

MySQL

Sat, 04 Apr 2026 12:26:57 +0000

Using MySQL in Grafana

Starting from Grafana v5.1 you can name the time column time in addition to earlier supported time_sec. Usage of time_sec will eventually be deprecated.

Grafana ships with a built-in MySQL data source plugin that allows you to query and visualize data from a MySQL compatible database.

Adding the data source

Open the side menu by clicking the Grafana icon in the top header.
In the side menu under the Dashboards link you should find a link named Data Sources.
Click the + Add data source button in the top header.
Select MySQL from the Type dropdown.

Data source options

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source means that it will be pre-selected for new panels.
`Host`	The IP address/hostname and optional port of your MySQL instance.
`Database`	Name of your MySQL database.
`User`	Database user’s login/username
`Password`	Database user’s password
`Session Timezone`	Specify the time zone used in the database session, such as `Europe/Berlin` or `+02:00`. This is necessary, if the timezone of the database (or the host of the database) is set to something other than UTC. Set the value used in the session with `SET time_zone='...'`. If you leave this field empty, then the time zone is not updated. For more information, refer to the MySQL documentation.
`Max open`	The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
`Max idle`	The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
`Max lifetime`	The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours. This should always be lower than configured wait_timeout in MySQL (Grafana v5.4+).

Min time interval

Identifier	Description
`y`	year
`M`	month
`w`	week
`d`	day
`h`	hour
`m`	minute
`s`	second
`ms`	millisecond

Database User Permissions (Important!)

The database user you specify when you add the data source should only be granted SELECT permissions on the specified database and tables you want to query. Grafana does not validate that the query is safe. The query could include any SQL statement. For example, statements like USE otherdb; and DROP TABLE user; would be executed. To protect against this we Highly recommend you create a specific mysql user with restricted permissions.

Example:

 CREATE USER 'grafanaReader' IDENTIFIED BY 'password';
 GRANT SELECT ON mydatabase.mytable TO 'grafanaReader';

You can use wildcards (*) in place of database or table if you want to grant access to more databases and tables.

Query Editor

Only available in Grafana v5.4+.

You find the MySQL query editor in the metrics tab in a panel’s edit mode. You enter edit mode by clicking the panel title, then edit.

Select table, time column and metric column (FROM)

When you enter edit mode for the first time or add a new query Grafana will try to prefill the query builder with the first table that has a timestamp column and a numeric column.

In the FROM field, Grafana will suggest tables that are in the configured database. To select a table or view in another database that your database user has access to you can manually enter a fully qualified name (database.table) like otherDb.metrics.

The Time column field refers to the name of the column holding your time values. Selecting a value for the Metric column field is optional. If a value is selected, the Metric column field will be used as the series name.

The metric column suggestions will only contain columns with a text datatype (text, tinytext, mediumtext, longtext, varchar, char). If you want to use a column with a different datatype as metric column you may enter the column name with a cast: CAST(numericColumn as CHAR). You may also enter arbitrary SQL expressions in the metric column field that evaluate to a text datatype like CONCAT(column1, " ", CAST(numericColumn as CHAR)).

Columns and Aggregation functions (SELECT)

In the SELECT row you can specify what columns and functions you want to use. In the column field you may write arbitrary expressions instead of a column name like column1 * column2 / column3.

If you use aggregate functions you need to group your resultset. The editor will automatically add a GROUP BY time if you add an aggregate function.

You may add further value columns by clicking the plus button and selecting Column from the menu. Multiple value columns will be plotted as separate series in the graph panel.

Filter data (WHERE)

To add a filter click the plus icon to the right of the WHERE condition. You can remove filters by clicking on the filter and selecting Remove. A filter for the current selected timerange is automatically added to new queries.

Group By

To group by time or any other columns click the plus icon at the end of the GROUP BY row. The suggestion dropdown will only show text columns of your currently selected table but you may manually enter any column. You can remove the group by clicking on the item and then selecting Remove.

If you add any grouping, all selected columns need to have an aggregate function applied. The query builder will automatically add aggregate functions to all columns without aggregate functions when you add groupings.

Gap Filling

Grafana can fill in missing values when you group by time. The time function accepts two arguments. The first argument is the time window that you would like to group by, and the second argument is the value you want Grafana to fill missing items with.

Text Editor Mode (RAW)

You can switch to the raw query editor mode by clicking the hamburger icon and selecting Switch editor mode or by clicking Edit SQL below the query.

If you use the raw query editor, be sure your query at minimum has ORDER BY time and a filter on the returned time range.

Macros

To simplify syntax and to allow for dynamic parts, like date range filters, the query can contain macros.

Macro example	Description
`$__time(dateColumn)`	Will be replaced by an expression to convert to a UNIX timestamp and rename the column to `time_sec`. For example, UNIX_TIMESTAMP(dateColumn) as time_sec
`$__timeEpoch(dateColumn)`	Will be replaced by an expression to convert to a UNIX timestamp and rename the column to `time_sec`. For example, UNIX_TIMESTAMP(dateColumn) as time_sec
`$__timeFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name. For example, dateColumn BETWEEN FROM_UNIXTIME(1494410783) AND FROM_UNIXTIME(1494410983)
`$__timeFrom()`	Will be replaced by the start of the currently active time selection. For example, FROM_UNIXTIME(1494410783)
`$__timeTo()`	Will be replaced by the end of the currently active time selection. For example, FROM_UNIXTIME(1494410983)
`$__timeGroup(dateColumn,'5m')`	Will be replaced by an expression usable in GROUP BY clause. For example, cast(cast(UNIX_TIMESTAMP(dateColumn)/(300) as signed)300 as signed),*
`$__timeGroup(dateColumn,'5m', 0)`	Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
`$__timeGroup(dateColumn,'5m', NULL)`	Same as above but NULL will be used as value for missing points.
`$__timeGroup(dateColumn,'5m', previous)`	Same as above but the previous value in that series will be used as fill value if no value has been seen yet NULL will be used (only available in Grafana 5.3+).
`$__timeGroupAlias(dateColumn,'5m')`	Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
`$__unixEpochFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name with times represented as Unix timestamp. For example, dateColumn > 1494410783 AND dateColumn < 1494497183
`$__unixEpochFrom()`	Will be replaced by the start of the currently active time selection as Unix timestamp. For example, 1494410783
`$__unixEpochTo()`	Will be replaced by the end of the currently active time selection as Unix timestamp. For example, 1494497183
`$__unixEpochNanoFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name with times represented as nanosecond timestamp. For example, dateColumn > 1494410783152415214 AND dateColumn < 1494497183142514872
`$__unixEpochNanoFrom()`	Will be replaced by the start of the currently active time selection as nanosecond timestamp. For example, 1494410783152415214
`$__unixEpochNanoTo()`	Will be replaced by the end of the currently active time selection as nanosecond timestamp. For example, 1494497183142514872
`$__unixEpochGroup(dateColumn,'5m', [fillmode])`	Same as $__timeGroup but for times stored as Unix timestamp (only available in Grafana 5.3+).
`$__unixEpochGroupAlias(dateColumn,'5m', [fillmode])`	Same as above but also adds a column alias (only available in Grafana 5.3+).

We plan to add many more macros. If you have suggestions for what macros you would like to see, please open an issue in our GitHub repo.

Table queries

If the Format as query option is set to Table then you can basically do any type of SQL query. The table panel will automatically show the results of whatever columns and rows your query returns.

Query editor with example query:

The query:

SELECT
  title as 'Title',
  user.login as 'Created By' ,
  dashboard.created as 'Created On'
 FROM dashboard
INNER JOIN user on user.id = dashboard.created_by
WHERE $__timeFilter(dashboard.created)

You can control the name of the Table panel columns by using regular as SQL column selection syntax.

The resulting table panel:

Time series queries

For backward compatibility, there’s an exception to the above rule for queries that return three columns including a string column named metric. Instead of transforming the metric column into field labels, it becomes the field name, and then the series name is formatted as the value of the metric column. See the example with the metric column below.

You can optionally customize the default series name formatting using instructions in Reference: Standard field definitions.

Example with metric column:

SELECT
  $__timeGroup(time_date_time,'5m'),
  min(value_double),
  'min' as metric
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY time
ORDER BY time

Data frame result:

+---------------------+-----------------+
| Name: time          | Name: min       |
| Labels:             | Labels:         |
| Type: []time.Time   | Type: []float64 |
+---------------------+-----------------+
| 2020-01-02 03:05:00 | 3               |
| 2020-01-02 03:10:00 | 6               |
+---------------------+-----------------+

Example using the fill parameter in the $__timeGroup macro to convert null values to be zero instead:

SELECT
  $__timeGroup(createdAt,'5m',0),
  sum(value_double) as value,
  hostname
FROM test_data
WHERE
  $__timeFilter(createdAt)
GROUP BY time, hostname
ORDER BY time

Data frame result:

+---------------------+---------------------------+---------------------------+
| Name: time          | Name: value               | Name: value               |
| Labels:             | Labels: hostname=10.0.1.1 | Labels: hostname=10.0.1.2 |
| Type: []time.Time   | Type: []float64           | Type: []float64           |
+---------------------+---------------------------+---------------------------+
| 2020-01-02 03:05:00 | 3                         | 4                         |
| 2020-01-02 03:10:00 | 6                         | 7                         |
+---------------------+---------------------------+---------------------------+

Example with multiple columns:

SELECT
  $__timeGroup(time_date_time,'5m'),
  min(value_double) as min_value,
  max(value_double) as max_value
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY time
ORDER BY time

Data frame result:

+---------------------+-----------------+-----------------+
| Name: time          | Name: min_value | Name: max_value |
| Labels:             | Labels:         | Labels:         |
| Type: []time.Time   | Type: []float64 | Type: []float64 |
+---------------------+-----------------+-----------------+
| 2020-01-02 03:04:00 | 3               | 4               |
| 2020-01-02 03:05:00 | 6               | 7               |
+---------------------+-----------------+-----------------+

Currently, there is no support for a dynamic group by time based on time range and panel width. This is something we plan to add.

Templating

This feature is currently available in the nightly builds and will be included in the 5.0.0 release.

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query Variable

If you add a template variable of the type Query, you can write a MySQL query that can return things like measurement names, key names or key values that are shown as a dropdown select box.

For example, you can have a variable that contains all values for the hostname column in a table if you specify a query like this in the templating variable Query setting.

SELECT hostname FROM my_host

A query can return multiple columns and Grafana will automatically create a list from them. For example, the query below will return a list with values from hostname and hostname2.

SELECT my_host.hostname, my_other_host.hostname2 FROM my_host JOIN my_other_host ON my_host.city = my_other_host.city

To use time range dependent macros like $__timeFilter(column) in your query the refresh mode of the template variable needs to be set to On Time Range Change.

SELECT event_name FROM event_log WHERE $__timeFilter(time_column)

Another option is a query that can create a key/value variable. The query should return two columns that are named __text and __value. The __text column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value. An example query with hostname as the text and id as the value:

SELECT hostname AS __text, id AS __value FROM my_host

You can also create nested variables. For example if you had another variable named region. Then you could have the hosts variable only show hosts from the current selected region with a query like this (if region is a multi-value variable then use the IN comparison operator rather than = to match against multiple values):

SELECT hostname FROM my_host  WHERE region IN($region)

Using `__searchFilter` to filter results in Query Variable

Available from Grafana 6.5 and above

Important that you surround the __searchFilter expression with quotes as Grafana does not do this for you.

The example below shows how to use __searchFilter as part of the query field to enable searching for hostname while the user types in the dropdown select box.

Query

SELECT hostname FROM my_host  WHERE hostname LIKE '$__searchFilter'

Using Variables in Queries

From Grafana 4.3.0 to 4.6.0, template variables are always quoted automatically so if it is a string value do not wrap them in quotes in where clauses.

From Grafana 4.7.0, template variable values are only quoted when the template variable is a multi-value.

If the variable is a multi-value variable then use the IN comparison operator rather than = to match against multiple values.

There are two syntaxes:

$<varname> Example with a template variable named hostname:

SELECT
  UNIX_TIMESTAMP(atimestamp) as time,
  aint as value,
  avarchar as metric
FROM my_table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp ASC

[[varname]] Example with a template variable named hostname:

SELECT
  UNIX_TIMESTAMP(atimestamp) as time,
  aint as value,
  avarchar as metric
FROM my_table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp ASC

Disabling Quoting for Multi-value Variables

${servers:csv}

Read more about variable formatting options in the Variables documentation.

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.

Example query using time column with epoch values:

SELECT
  epoch_time as time,
  metric1 as text,
  CONCAT(tag1, ',', tag2) as tags
FROM
  public.test_data
WHERE
  $__unixEpochFilter(epoch_time)

Example region query using time and timeend columns with epoch values:

Only available in Grafana v6.6+.

SELECT
  epoch_time as time,
  epoch_timeend as timeend,
  metric1 as text,
  CONCAT(tag1, ',', tag2) as tags
FROM
  public.test_data
WHERE
  $__unixEpochFilter(epoch_time)

Example query using time column of native SQL date/time data type:

SELECT
  native_date_time as time,
  metric1 as text,
  CONCAT(tag1, ',', tag2) as tags
FROM
  public.test_data
WHERE
  $__timeFilter(native_date_time)

Name	Description
`time`	The name of the date/time field. Could be a column with a native SQL date/time data type or epoch value.
`timeend`	Optional name of the end date/time field. Could be a column with a native SQL date/time data type or epoch value. (Grafana v6.6+)
`text`	Event description field.
`tags`	Optional field name to use for event tags as a comma separated string.

Alerting

Time series queries should work in alerting conditions. Table formatted queries are not yet supported in alert rule conditions.

Configure the data source with provisioning

Here are some provisioning examples for this data source.

apiVersion: 1

datasources:
  - name: MySQL
    type: mysql
    url: localhost:3306
    database: grafana
    user: grafana
    jsonData:
      maxOpenConns: 0 # Grafana v5.4+
      maxIdleConns: 2 # Grafana v5.4+
      connMaxLifetime: 14400 # Grafana v5.4+
    secureJsonData:
      password: ${GRAFANA_MYSQL_PASSWORD}

OpenTSDB

Sat, 04 Apr 2026 12:26:57 +0000

Using OpenTSDB in Grafana

Grafana ships with advanced support for OpenTSDB. This topic explains options, variables, querying, and other options specific to the OpenTSDB data source. Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.

OpenTSDB settings

To access OpenTSDB settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and then click the OpenTSDB data source.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source means that it will be pre-selected for new panels.
`URL`	The HTTP protocol, IP, and port of your OpenTSDB server (default port is usually 4242)
`Allowed cookies`	List the names of cookies to forward to the data source.
`Version`	Version = opentsdb version, either <=2.1 or 2.2
`Resolution`	Metrics from opentsdb may have datapoints with either second or millisecond resolution.
`Lookup limit`	Default is 1000.

Query editor

Open a graph in edit mode by click the title. Query editor will differ if the data source has version <=2.1 or = 2.2. In the former version, only tags can be used to query OpenTSDB. But in the latter version, filters as well as tags can be used to query opentsdb. Fill Policy is also introduced in OpenTSDB 2.2.

Note: While using OpenTSDB 2.2 data source, make sure you use either Filters or Tags as they are mutually exclusive. If used together, might give you weird results.

Auto complete suggestions

As soon as you start typing metric names, tag names and tag values , you should see highlighted auto complete suggestions for them. The autocomplete only works if the OpenTSDB suggest API is enabled.

Templating queries

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query variable

Grafana’s OpenTSDB data source supports template variable queries. This means you can create template variables that fetch the values from OpenTSDB. For example, metric names, tag names, or tag values.

When using OpenTSDB with a template variable of query type you can use following syntax for lookup.

Query	Description
`metrics(prefix)`	Returns metric names with specific prefix (can be empty)
`tag_names(cpu)`	Returns tag names (i.e. keys) for a specific cpu metric
`tag_values(cpu, hostname)`	Returns tag values for metric cpu and tag key hostname
`suggest_tagk(prefix)`	Returns tag names (i.e. keys) for all metrics with specific prefix (can be empty)
`suggest_tagv(prefix)`	Returns tag values for all metrics with specific prefix (can be empty)

If you do not see template variables being populated in Preview of values section, you need to enable tsd.core.meta.enable_realtime_ts in the OpenTSDB server settings. Also, to populate metadata of the existing time series data in OpenTSDB, you need to run tsdb uid metasync on the OpenTSDB server.

Nested templating

One template variable can be used to filter tag values for another template variable. First parameter is the metric name, second parameter is the tag key for which you need to find tag values, and after that all other dependent template variables. Some examples are mentioned below to make nested template queries work successfully.

Query	Description
`tag_values(cpu, hostname, env=$env)`	Return tag values for cpu metric, selected env tag value and tag key hostname
`tag_values(cpu, hostname, env=$env, region=$region)`	Return tag values for cpu metric, selected env tag value, selected region tag value and tag key hostname

For details on OpenTSDB metric queries, check out the official OpenTSDB documentation

Configure the data source with provisioning

Here are some provisioning examples for this data source.

apiVersion: 1

datasources:
  - name: OpenTsdb
    type: opentsdb
    access: proxy
    url: http://localhost:4242
    jsonData:
      tsdbResolution: 1
      tsdbVersion: 1

PostgreSQL

Sat, 04 Apr 2026 12:26:57 +0000

PostgreSQL data source

Grafana ships with a built-in PostgreSQL data source plugin that allows you to query and visualize data from a PostgreSQL compatible database. This topic explains options, variables, querying, and other options specific to this data source. For instructions about how to add a data source to Grafana, refer to Add a data source. Only users with the organization admin role can add data sources.

PostgreSQL settings

To access PostgreSQL settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and then click the PostgreSQL data source.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source means that it will be pre-selected for new panels.
`Host`	The IP address/hostname and optional port of your PostgreSQL instance. Do not include the database name. The connection string for connecting to Postgres will not be correct and it may cause errors.
`Database`	Name of your PostgreSQL database.
`User`	Database user’s login/username
`Password`	Database user’s password
`SSL Mode`	Determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server. When SSL Mode is disabled, SSL Method and Auth Details would not be visible.
`SSL Auth Details Method`	Determines whether the SSL Auth details will be configured as a file path or file content. Grafana v7.5+
`SSL Auth Details Value`	File path or file content of SSL root certificate, client certificate and client key
`Max open`	The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
`Max idle`	The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
`Max lifetime`	The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
`Version`	Determines which functions are available in the query builder (only available in Grafana 5.3+).
`TimescaleDB`	A time-series database built as a PostgreSQL extension. When enabled, Grafana uses `time_bucket` in the `$__timeGroup` macro to display TimescaleDB specific aggregate functions in the query builder (only available in Grafana 5.3+).

Min time interval

Identifier	Description
`y`	year
`M`	month
`w`	week
`d`	day
`h`	hour
`m`	minute
`s`	second
`ms`	millisecond

Database user permissions (Important!)

The database user you specify when you add the data source should only be granted SELECT permissions on the specified database and tables you want to query. Grafana does not validate that the query is safe. The query could include any SQL statement. For example, statements like DELETE FROM user; and DROP TABLE user; would be executed. To protect against this we highly recommend you create a specific PostgreSQL user with restricted permissions.

Example:

 CREATE USER grafanareader WITH PASSWORD 'password';
 GRANT USAGE ON SCHEMA schema TO grafanareader;
 GRANT SELECT ON schema.table TO grafanareader;

Make sure the user does not get any unwanted privileges from the public role.

Query editor

You find the PostgreSQL query editor in the metrics tab in Graph or Singlestat panel’s edit mode. You enter edit mode by clicking the panel title, then edit.

Select table, time column and metric column (FROM)

When you enter edit mode for the first time or add a new query Grafana will try to prefill the query builder with the first table that has a timestamp column and a numeric column.

In the FROM field, Grafana will suggest tables that are in the search_path of the database user. To select a table or view not in your search_path you can manually enter a fully qualified name (schema.table) like public.metrics.

The metric column suggestions will only contain columns with a text datatype (char,varchar,text). If you want to use a column with a different datatype as metric column you may enter the column name with a cast: ip::text. You may also enter arbitrary SQL expressions in the metric column field that evaluate to a text datatype like hostname || ' ' || container_name.

Columns, window, and aggregation functions (SELECT)

In the SELECT row you can specify what columns and functions you want to use. In the column field you may write arbitrary expressions instead of a column name like column1 * column2 / column3.

The available functions in the query editor depend on the PostgreSQL version you selected when configuring the data source. If you use aggregate functions you need to group your resultset. The editor will automatically add a GROUP BY time if you add an aggregate function.

The editor tries to simplify and unify this part of the query. For example:

The above will generate the following PostgreSQL SELECT clause:

avg(tx_bytes) OVER (ORDER BY "time" ROWS 5 PRECEDING) AS "tx_bytes"

You may add further value columns by clicking the plus button and selecting Column from the menu. Multiple value columns will be plotted as separate series in the graph panel.

Filter data (WHERE)

Group by

Gap filling

Text editor mode (RAW)

You can switch to the raw query editor mode by clicking the hamburger icon and selecting Switch editor mode or by clicking Edit SQL below the query.

If you use the raw query editor, be sure your query at minimum has ORDER BY time and a filter on the returned time range.

Macros

Macros can be used within a query to simplify syntax and allow for dynamic parts.

Macro example	Description
`$__time(dateColumn)`	Will be replaced by an expression to convert to a UNIX timestamp and rename the column to `time_sec`. For example, UNIX_TIMESTAMP(dateColumn) as time_sec
`$__timeEpoch(dateColumn)`	Will be replaced by an expression to convert to a UNIX timestamp and rename the column to `time_sec`. For example, UNIX_TIMESTAMP(dateColumn) as time_sec
`$__timeFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name. For example, dateColumn BETWEEN FROM_UNIXTIME(1494410783) AND FROM_UNIXTIME(1494410983)
`$__timeFrom()`	Will be replaced by the start of the currently active time selection. For example, FROM_UNIXTIME(1494410783)
`$__timeTo()`	Will be replaced by the end of the currently active time selection. For example, FROM_UNIXTIME(1494410983)
`$__timeGroup(dateColumn,'5m')`	Will be replaced by an expression usable in GROUP BY clause. For example, cast(cast(UNIX_TIMESTAMP(dateColumn)/(300) as signed)300 as signed),*
`$__timeGroup(dateColumn,'5m', 0)`	Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
`$__timeGroup(dateColumn,'5m', NULL)`	Same as above but NULL will be used as value for missing points.
`$__timeGroup(dateColumn,'5m', previous)`	Same as above but the previous value in that series will be used as fill value if no value has been seen yet NULL will be used (only available in Grafana 5.3+).
`$__timeGroupAlias(dateColumn,'5m')`	Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
`$__unixEpochFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name with times represented as Unix timestamp. For example, dateColumn > 1494410783 AND dateColumn < 1494497183
`$__unixEpochFrom()`	Will be replaced by the start of the currently active time selection as Unix timestamp. For example, 1494410783
`$__unixEpochTo()`	Will be replaced by the end of the currently active time selection as Unix timestamp. For example, 1494497183
`$__unixEpochNanoFilter(dateColumn)`	Will be replaced by a time range filter using the specified column name with times represented as nanosecond timestamp. For example, dateColumn > 1494410783152415214 AND dateColumn < 1494497183142514872
`$__unixEpochNanoFrom()`	Will be replaced by the start of the currently active time selection as nanosecond timestamp. For example, 1494410783152415214
`$__unixEpochNanoTo()`	Will be replaced by the end of the currently active time selection as nanosecond timestamp. For example, 1494497183142514872
`$__unixEpochGroup(dateColumn,'5m', [fillmode])`	Same as $__timeGroup but for times stored as Unix timestamp (only available in Grafana 5.3+).
`$__unixEpochGroupAlias(dateColumn,'5m', [fillmode])`	Same as above but also adds a column alias (only available in Grafana 5.3+).

We plan to add many more macros. If you have suggestions for what macros you would like to see, please open an issue in our GitHub repo.

Table queries

If the Format as query option is set to Table then you can basically do any type of SQL query. The table panel will automatically show the results of whatever columns and rows your query returns.

Query editor with example query:

The query:

SELECT
  title as "Title",
  "user".login as "Created By",
  dashboard.created as "Created On"
FROM dashboard
INNER JOIN "user" on "user".id = dashboard.created_by
WHERE $__timeFilter(dashboard.created)

You can control the name of the Table panel columns by using regular as SQL column selection syntax.

The resulting table panel:

Time series queries

For backward compatibility, there’s an exception to the above rule for queries that return three columns including a string column named metric. Instead of transforming the metric column into field labels, it becomes the field name, and then the series name is formatted as the value of the metric column. See the example with the metric column below.

You can optionally customize the default series name formatting using instructions in Reference: Standard field definitions.

Example with metric column:

SELECT
  $__timeGroup("time_date_time",'5m'),
  min("value_double"),
  'min' as metric
FROM test_data
WHERE $__timeFilter("time_date_time")
GROUP BY time
ORDER BY time

Data frame result:

+---------------------+-----------------+
| Name: time          | Name: min       |
| Labels:             | Labels:         |
| Type: []time.Time   | Type: []float64 |
+---------------------+-----------------+
| 2020-01-02 03:05:00 | 3               |
| 2020-01-02 03:10:00 | 6               |
+---------------------+-----------------+

Example using the fill parameter in the $__timeGroup macro to convert null values to be zero instead:

SELECT
  $__timeGroup("createdAt",'5m',0),
  sum(value) as value,
  hostname
FROM test_data
WHERE
  $__timeFilter("createdAt")
GROUP BY time, hostname
ORDER BY time

Data frame result:

+---------------------+---------------------------+---------------------------+
| Name: time          | Name: value               | Name: value               |
| Labels:             | Labels: hostname=10.0.1.1 | Labels: hostname=10.0.1.2 |
| Type: []time.Time   | Type: []float64           | Type: []float64           |
+---------------------+---------------------------+---------------------------+
| 2020-01-02 03:05:00 | 3                         | 4                         |
| 2020-01-02 03:10:00 | 6                         | 7                         |
+---------------------+---------------------------+---------------------------+

Example with multiple columns:

SELECT
  $__timeGroup("time_date_time",'5m'),
  min("value_double") as "min_value",
  max("value_double") as "max_value"
FROM test_data
WHERE $__timeFilter("time_date_time")
GROUP BY time
ORDER BY time

Data frame result:

+---------------------+-----------------+-----------------+
| Name: time          | Name: min_value | Name: max_value |
| Labels:             | Labels:         | Labels:         |
| Type: []time.Time   | Type: []float64 | Type: []float64 |
+---------------------+-----------------+-----------------+
| 2020-01-02 03:04:00 | 3               | 4               |
| 2020-01-02 03:05:00 | 6               | 7               |
+---------------------+-----------------+-----------------+

Templating

Refer to Templates and variables for an introduction to the templating feature and the different types of template variables.

Query variable

If you add a template variable of the type Query, you can write a PostgreSQL query that can return things like measurement names, key names or key values that are shown as a dropdown select box.

For example, you can have a variable that contains all values for the hostname column in a table if you specify a query like this in the templating variable Query setting.

SELECT hostname FROM host

A query can return multiple columns and Grafana will automatically create a list from them. For example, the query below will return a list with values from hostname and hostname2.

SELECT host.hostname, other_host.hostname2 FROM host JOIN other_host ON host.city = other_host.city

To use time range dependent macros like $__timeFilter(column) in your query the refresh mode of the template variable needs to be set to On Time Range Change.

SELECT event_name FROM event_log WHERE $__timeFilter(time_column)

Another option is a query that can create a key/value variable. The query should return two columns that are named __text and __value. The __text column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value. An example query with hostname as the text and id as the value:

SELECT hostname AS __text, id AS __value FROM host

You can also create nested variables. Using a variable named region, you could have the hosts variable only show hosts from the current selected region with a query like this (if region is a multi-value variable then use the IN comparison operator rather than = to match against multiple values):

SELECT hostname FROM host  WHERE region IN($region)

Using `__searchFilter` to filter results in Query Variable

Available from Grafana 6.5 and above

Important that you surround the __searchFilter expression with quotes as Grafana does not do this for you.

The example below shows how to use __searchFilter as part of the query field to enable searching for hostname while the user types in the dropdown select box.

Query

SELECT hostname FROM my_host  WHERE hostname LIKE '$__searchFilter'

Using Variables in Queries

From Grafana 4.3.0 to 4.6.0, template variables are always quoted automatically. If your template variables are strings, do not wrap them in quotes in where clauses.

From Grafana 4.7.0, template variable values are only quoted when the template variable is a multi-value.

If the variable is a multi-value variable then use the IN comparison operator rather than = to match against multiple values.

There are two syntaxes:

$<varname> Example with a template variable named hostname:

SELECT
  atimestamp as time,
  aint as value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp ASC

[[varname]] Example with a template variable named hostname:

SELECT
  atimestamp as time,
  aint as value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp ASC

Disabling quoting for multi-value variables

Grafana automatically creates a quoted, comma-separated string for multi-value variables. For example: if server01 and server02 are selected then it will be formatted as: 'server01', 'server02'. To disable quoting, use the csv formatting option for variables:

${servers:csv}

Read more about variable formatting options in the Variables documentation.

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.

Example query using time column with epoch values:

SELECT
  epoch_time as time,
  metric1 as text,
  concat_ws(', ', metric1::text, metric2::text) as tags
FROM
  public.test_data
WHERE
  $__unixEpochFilter(epoch_time)

Example region query using time and timeend columns with epoch values:

Only available in Grafana v6.6+.

SELECT
  epoch_time as time,
  epoch_time_end as timeend,
  metric1 as text,
  concat_ws(', ', metric1::text, metric2::text) as tags
FROM
  public.test_data
WHERE
  $__unixEpochFilter(epoch_time)

Example query using time column of native SQL date/time data type:

SELECT
  native_date_time as time,
  metric1 as text,
  concat_ws(', ', metric1::text, metric2::text) as tags
FROM
  public.test_data
WHERE
  $__timeFilter(native_date_time)

Name	Description
`time`	The name of the date/time field. Could be a column with a native SQL date/time data type or epoch value.
`timeend`	Optional name of the end date/time field. Could be a column with a native SQL date/time data type or epoch value. (Grafana v6.6+)
`text`	Event description field.
`tags`	Optional field name to use for event tags as a comma separated string.

Alerting

Time series queries should work in alerting conditions. Table formatted queries are not yet supported in alert rule conditions.

Configure the data source with provisioning

Here are some provisioning examples for this data source.

apiVersion: 1

datasources:
  - name: Postgres
    type: postgres
    url: localhost:5432
    database: grafana
    user: grafana
    secureJsonData:
      password: 'Password!'
    jsonData:
      sslmode: 'disable' # disable/require/verify-ca/verify-full
      maxOpenConns: 0 # Grafana v5.4+
      maxIdleConns: 2 # Grafana v5.4+
      connMaxLifetime: 14400 # Grafana v5.4+
      postgresVersion: 903 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10
      timescaledb: false

Note: In the above code, the postgresVersion value of 10 refers to version PotgreSQL 10 and above.

If you encounter metric request errors or other issues:

Make sure your data source YAML file parameters exactly match the example. This includes parameter names and use of quotation marks.
Make sure the database name is not included in the url.

Prometheus

Sat, 04 Apr 2026 12:26:57 +0000

Prometheus data source

Grafana includes built-in support for Prometheus. This topic explains options, variables, querying, and other options specific to the Prometheus data source. Refer to Add a data source for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.

Note: You can use Grafana Cloud to avoid the overhead of installing, maintaining, and scaling your observability stack. The free forever plan includes Grafana, 10K Prometheus series, 50 GB logs, and more.Create a free account to get started.

Prometheus settings

To access Prometheus settings, hover your mouse over the Configuration (gear) icon, then click Data Sources, and then click the Prometheus data source.

Name	Description
`Name`	The data source name. This is how you refer to the data source in panels and queries.
`Default`	Default data source that is pre-selected for new panels.
`Url`	The URL of your Prometheus server, for example, `http://prometheus.example.org:9090`.
`Access`	Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. Note: Browser (direct) access is deprecated and will be removed in a future release.
`Basic Auth`	Enable basic authentication to the Prometheus data source.
`User`	User name for basic authentication.
`Password`	Password for basic authentication.
`Scrape interval`	Set this to the typical scrape and evaluation interval configured in Prometheus. Defaults to 15s.
`HTTP method`	Use either POST or GET HTTP method to query your data source. POST is the recommended and pre-selected method as it allows bigger queries. Change this to GET if you have a Prometheus version older than 2.1 or if POST requests are restricted in your network.
`Disable metrics lookup`	Checking this option will disable the metrics chooser and metric/label support in the query field’s autocomplete. This helps if you have performance issues with bigger Prometheus instances.
`Custom Query Parameters`	Add custom parameters to the Prometheus query URL. For example `timeout`, `partial_response`, `dedup`, or `max_source_resolution`. Multiple parameters should be concatenated together with an ‘&’.
Exemplars configuration
`Internal link`	Enable this option is you have an internal link. When you enable this option, you will see a data source selector. Select the backend tracing data store for your exemplar data.
`Data source`	You will see this option only if you enable `Internal link` option. Select the backend tracing data store for your exemplar data.
`URL`	You will see this option only if the `Internal link` option is disabled. Enter the full URL of the external link. You can interpolate the value from the field with `${__value.raw }` macro.
`URL Label`	(Optional) add a custom display label to override the value of the `Label name` field.
`Label name`	Add a name for the exemplar traceID property.

Prometheus query editor

Below you can find information and options for Prometheus query editor in dashboard and in Explore.

Query editor in dashboards

Open a graph in edit mode by clicking the title > Edit (or by pressing e key while hovering over panel).

Name	Description
`Query expression`	Prometheus query expression. For more information, refer to the Prometheus documentation.
`Legend format`	Controls the name of the time series, using name or pattern. For example, `{{hostname}}` is replaced by the label value for the label `hostname`.
`Step`	Use ‘Minimum’ or ‘Maximum’ step mode to set the lower or upper bounds respectively on the interval between data points. For example, set “minimum 1h” to hint that measurements are not frequent (taken hourly). Use the ‘Exact’ step mode to set a precise interval between data points. `$__interval` and `$__rate_interval` are supported.
`Resolution`	`1/1` sets both the `$__interval` variable and the `step` parameter of Prometheus range queries such that each pixel corresponds to one data point. For better performance, you can pick lower resolutions. `1/2` only retrieves a data point for every other pixel, and `1/10` retrieves one data point per 10 pixels. Both Min time interval and Step limit the final value of `$__interval` and `step`.
`Metric lookup`	Search for metric names in this input field.
`Format as`	You can switch between `Table` `Time series` or `Heatmap` options. The `Table` option works only in the Table panel. `Heatmap` displays metrics of the Histogram type on a Heatmap panel. Under the hood, it converts cumulative histograms to regular ones and sorts series by the bucket bound.
`Instant`	Perform an “instant” query to return only the latest value that Prometheus has scraped for the requested time series. Instant queries can return results much faster than normal range queries. Use them to look up label sets.
`Min time interval`	This value multiplied by the denominator from the Resolution setting sets a lower limit to both the `$__interval` variable and the `step` parameter of Prometheus range queries. Defaults to Scrape interval as specified in the data source options.
`Exemplars`	Run and show exemplars in the graph.

Note: Grafana modifies the request dates for queries to align them with the dynamically calculated step. This ensures consistent display of metrics data, but it can result in a small gap of data at the right edge of a graph.

Instant queries in dashboards

The Prometheus data source allows you to run “instant” queries, which query only the latest value. You can visualize the results in a table panel to see all available labels of a timeseries.

Instant query results are made up only of one data point per series but can be shown in the graph panel with the help of series overrides. To show them in the graph as a latest value point, add a series override and select Points > true. To show a horizontal line across the whole graph, add a series override and select Transform > constant.

Support for constant series overrides is available from Grafana v6.4

Query editor in Explore

Name	Description
`Query expression`	Prometheus query expression, check out the Prometheus documentation.
`Step`	`Step` parameter of Prometheus range queries. Time units can be used here, for example: 5s, 1m, 3h, 1d, 1y. Default unit if no unit specified is `s` (seconds).
`Query type`	`Range`, `Instant`, or `Both`. When running Range query, the result of the query is displayed in graph and table. Instant query returns only the latest value that Prometheus has scraped for the requested time series and it is displayed in the table. When Both is selected, both instant query and range query is run. Result of range query is displayed in graph and the result of instant query is displayed in the table.
`Exemplars`	Run and show exemplars in the graph.

Metrics browser

The metrics browser allows you to quickly find metrics and select relevant labels to build basic queries. When you open the browser you will see all available metrics and labels. If supported by your Prometheus instance, each metric will show its HELP and TYPE as a tooltip.

When you select a metric, the browser narrows down the available labels to show only the ones applicable to the metric. You can then select one or more labels for which the available label values are shown in lists in the bottom section. Select one or more values for each label to tighten your query scope.

Note: If you do not remember a metric name to start with, you can also select a few labels first, to narrow down the list and then find relevant label values.

All lists in the metrics browser have a search field above them to quickly filter for metrics or labels that match a certain string. The values section only has one search field. Its filtering applies to all labels to help you find values across labels once they have been selected, for example, among your labels app, job, job_name only one might with the value you are looking for.

Once you are satisfied with your query, click “Use query” to run the query. The button “Use as rate query” adds a rate(...)[$__interval] around your query to help write queries for counter metrics. The “Validate selector” button will check with Prometheus how many time series are available for that selector.

Limitations

The metrics browser has a hard limit of 10,000 labels (keys) and 50,000 label values (including metric names). If your Prometheus instance returns more results, the browser will continue functioning. However, the result sets will be cut off above those maximum limits.

Templating

Instead of hard-coding things like server, application and sensor name in your metric queries, you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query variable

Variable of the type Query allows you to query Prometheus for a list of metrics, labels or label values. The Prometheus data source plugin provides the following functions you can use in the Query input field.

Name	Description	Used API endpoints
`label_names()`	Returns a list of label names.	/api/v1/labels
`label_values(label)`	Returns a list of label values for the `label` in every metric.	/api/v1/label/`label`/values
`label_values(metric, label)`	Returns a list of label values for the `label` in the specified metric.	/api/v1/series
`metrics(metric)`	Returns a list of metrics matching the specified `metric` regex.	/api/v1/label/__name__/values
`query_result(query)`	Returns a list of Prometheus query result for the `query`.	/api/v1/query

For details of what metric names, label names and label values are please refer to the Prometheus documentation.

Using interval and range variables

Support for $__range, $__range_s and $__range_ms only available from Grafana v5.3

You can use some global built-in variables in query variables, for example, $__interval, $__interval_ms, $__range, $__range_s and $__range_ms. See Global built-in variables for more information. They are convenient to use in conjunction with the query_result function when you need to filter variable queries since the label_values function doesn’t support queries.

Make sure to set the variable’s refresh trigger to be On Time Range Change to get the correct instances when changing the time range on the dashboard.

Example usage:

Populate a variable with the busiest 5 request instances based on average QPS over the time range shown in the dashboard:

Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
Regex: /"([^"]+)"/

Populate a variable with the instances having a certain state over the time range shown in the dashboard, using $__range_s:

Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
Regex:

Using `$__rate_interval`

Note: Available in Grafana 7.2 and above

$__rate_interval is the recommended interval to use in the rate and increase functions. It will “just work” in most cases, avoiding most of the pitfalls that can occur when using a fixed interval or $__interval.

OK:       rate(http_requests_total[5m])
Better:   rate(http_requests_total[$__rate_interval])

Details: $__rate_interval is defined as max($__interval + Scrape interval, 4 * Scrape interval), where Scrape interval is the Min step setting (AKA queryinterval, a setting per PromQL query) if any is set. Otherwise, the Scrape interval setting in the Prometheus data source is used. (The Min interval setting in the panel is modified by the resolution setting and therefore doesn’t have any effect on _Scrape interval.) This article contains additional details.

Using variables in queries

There are two syntaxes:

$<varname> Example: rate(http_requests_total{job=~"$job"}[5m])
[[varname]] Example: rate(http_requests_total{job=~"[[job]]"}[5m])

Ad hoc filters variable

Prometheus supports the special ad hoc filters variable type. It allows you to specify any number of label/value filters on the fly. These filters are automatically applied to all your Prometheus queries.

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.

Prometheus supports two ways to query annotations.

A regular metric query
A Prometheus query for pending and firing alerts (for details see Inspecting alerts during runtime)

The step option is useful to limit the number of events returned from your query.

Get Grafana metrics into Prometheus

Grafana exposes metrics for Prometheus on the /metrics endpoint. We also bundle a dashboard within Grafana so you can get started viewing your metrics faster. You can import the bundled dashboard by going to the data source edit page and click the dashboard tab. There you can find a dashboard for Grafana and one for Prometheus. Import and start viewing all the metrics!

For detailed instructions, refer to Internal Grafana metrics.

Prometheus API

The Prometheus data source works with other projects that implement the Prometheus query API including:

For more information on how to query other Prometheus-compatible projects from Grafana, refer to the specific project documentation.

Provision the Prometheus data source

You can configure data sources using config files with Grafana’s provisioning system. Read more about how it works and all the settings you can set for data sources on the provisioning docs page.

Here are some provisioning examples for this data source:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    # Access mode - proxy (server in the UI) or direct (browser in the UI).
    access: proxy
    url: http://localhost:9090
    jsonData:
      httpMethod: POST
      exemplarTraceIdDestinations:
        # Field with internal link pointing to data source in Grafana.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          name: traceID

        # Field with external link.
        - name: traceID
          url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'

Amazon Managed Service for Prometheus

The Prometheus data source works with Amazon Managed Service for Prometheus. If you are using an AWS Identity and Access Management (IAM) policy to control access to your Amazon Managed Service for Prometheus domain, then you must use AWS Signature Version 4 (AWS SigV4) to sign all requests to that domain. For more details on AWS SigV4, refer to the AWS documentation.

Note: Grafana version 7.3.5 or higher is required to use SigV4 authentication.

To connect the Prometheus data source to Amazon Managed Service for Prometheus using SigV4 authentication, refer to the AWS guide to Set up Grafana open source or Grafana Enterprise for use with AMP.

If you are running Grafana in an Amazon EKS cluster, follow the AWS guide to Query using Grafana running in an Amazon EKS cluster.

Configuring exemplars

Note: This feature is available in Prometheus 2.26+ and Grafana 7.4+.

Grafana 7.4 and later versions have the capability to show exemplars data alongside a metric both in Explore and Dashboards. Exemplars are a way to associate higher cardinality metadata from a specific event with traditional timeseries data.

Configure Exemplars in the data source settings by adding external or internal links.

Tempo

Sat, 04 Apr 2026 12:26:57 +0000

Tempo data source

Grafana ships with built-in support for Tempo a high volume, minimal dependency trace storage, OSS tracing solution from Grafana Labs. Add it as a data source, and you are ready to query your traces in Explore.

Add data source

To access Tempo settings, click the Configuration (gear) icon, then click Data Sources > Tempo.

Name	Description
`Name`	The name using which you will refer to the data source in panels, queries, and Explore.
`Default`	The default data source will be pre-selected for new panels.
`URL`	The URL of the Tempo instance, e.g., `http://tempo`
`Basic Auth`	Enable basic authentication to the Tempo data source.
`User`	User name for basic authentication.
`Password`	Password for basic authentication.

Trace to logs

Note: This feature is available in Grafana 7.4+.

This is a configuration for the trace to logs feature. Select target data source (at this moment limited to Loki data sources) and select which tags will be used in the logs query.

Data source - Target data source.
Tags - The tags that will be used in the Loki query. Default is 'cluster', 'hostname', 'namespace', 'pod'.
Map tag names - When enabled, allows configuring how Tempo tag names map to Loki label names. For example, map service.name to service.
Span start time shift - A shift in the start time for the Loki query based on the start time for the span. To extend the time to the past, use a negative value. You can use time units, for example, 5s, 1m, 3h. The default is 0.
Span end time shift - Shift in the end time for the Loki query based on the span end time. Time units can be used here, for example, 5s, 1m, 3h. The default is 0.
Filter by Trace ID - Toggle to append the trace ID to the Loki query.
Filter by Span ID - Toggle to append the span ID to the Loki query.

Service Graph

This is a configuration for the Service Graph feature.

– Data source - Prometheus instance where the Service Graph data is stored.

Search

This is a configuration for Tempo search.

– Hide search - Optionally, hide the search query option in Explore in cases where search is not configured in the Tempo instance.

Node Graph

This is a configuration for the beta Node Graph visualization. The Node Graph is shown after the trace view is loaded and is disabled by default.

– Enable Node Graph - Enables the Node Graph visualization.

Query traces

You can query and display traces from Tempo via Explore. You can search for traces if you set up the trace to logs setting in the data source configuration page. To find traces to visualize, use the Loki query editor. To get search results, you must have derived fields configured, which point to this data source.

To query a particular trace, select the TraceID query type, and then put the ID into the Trace ID field.

Upload JSON trace file

You can upload a JSON file that contains a single trace to visualize it. If the file has multiple traces then the first trace is used for visualization.

Here is an example JSON:

{
  "batches": [
    {
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "db" } },
          { "key": "job", "value": { "stringValue": "tns/db" } },
          { "key": "opencensus.exporterversion", "value": { "stringValue": "Jaeger-Go-2.22.1" } },
          { "key": "host.name", "value": { "stringValue": "63d16772b4a2" } },
          { "key": "ip", "value": { "stringValue": "0.0.0.0" } },
          { "key": "client-uuid", "value": { "stringValue": "39fb01637a579639" } }
        ]
      },
      "instrumentationLibrarySpans": [
        {
          "instrumentationLibrary": {},
          "spans": [
            {
              "traceId": "AAAAAAAAAABguiq7RPE+rg==",
              "spanId": "cmteMBAvwNA=",
              "parentSpanId": "OY8PIaPbma4=",
              "name": "HTTP GET - root",
              "kind": "SPAN_KIND_SERVER",
              "startTimeUnixNano": "1627471657255809000",
              "endTimeUnixNano": "1627471657256268000",
              "attributes": [
                { "key": "http.status_code", "value": { "intValue": "200" } },
                { "key": "http.method", "value": { "stringValue": "GET" } },
                { "key": "http.url", "value": { "stringValue": "/" } },
                { "key": "component", "value": { "stringValue": "net/http" } }
              ],
              "status": {}
            }
          ]
        }
      ]
    }
  ]
}

Service Graph

A service graph is a visual representation of the relationships between services. Each node on the graph represents a service such as an API or database. With this graph, customers can easily detect performance issues, increases in error, fault, or throttle rates in any of their services, and dive deep into corresponding traces and root causes.

To display the service graph:

Configure the Grafana Agent to generate service graph data
Link a Prometheus datasource in the Tempo datasource settings.
Navigate to Explore
Select the Tempo datasource
Select the Service Graph query type and run the query
Optionally, filter by service name

You can pan and zoom the view with buttons or you mouse. For details about the visualization, refer to Node graph panel.

Each service in the graph is represented as a circle. Numbers on the inside shows average time per request and request per second.

The color of each circle represents the percentage of requests in each of the following states:

green = success
red = fault
yellow = errors
purple = throttled responses

Click on the service to see a context menu with additional links for quick navigation to other relevant information.

Linking Trace ID from logs

You can link to Tempo trace from logs in Loki or Elastic by configuring an internal link. See the Derived fields section in the Loki data source or Data links section in the Elastic data source for configuration instructions.

Provision the Tempo data source

You can modify the Grafana configuration files to provision the Tempo data source. Read more about how it works and all the settings you can set for data sources on the provisioning topic.

Here is an example config:

apiVersion: 1

datasources:
  - name: Tempo
    type: tempo
    # Access mode - proxy (server in the UI) or direct (browser in the UI).
    access: proxy
    url: http://localhost:3200
    jsonData:
      httpMethod: GET
      tracesToLogs:
        datasourceUid: 'loki'
        tags: ['job', 'instance', 'pod', 'namespace']
        mappedTags: [{ key: 'service.name', value: 'service' }]
        mapTagNamesEnabled: false
        spanStartTimeShift: '1h'
        spanEndTimeShift: '1h'
        filterByTraceID: false
        filterBySpanID: false
        lokiSearch: true
      serviceMap:
        datasourceUid: 'prometheus'
      search:
        hide: false
      nodeGraph:
        enabled: true

TestData

Sat, 04 Apr 2026 12:26:57 +0000

Grafana TestData DB

The purpose of this data source is to make it easier to create fake data for any panel. Using TestData DB you can build your own time series and have any panel render it. This makes it much easier to verify functionality since the data can be shared very easily.

Enable

The TestData DB data source is not enabled by default. To enable it:

In the Configuration menu (small gear on the left side of the screen), click Data Sources.
Click Add Data Source.
Search and click TestData DB.
Click Save & Test to enable it.

Create mock data.

Once TestData DB is enabled, you can use it as a data source in any metric panel.

CSV

The comma separated values scenario is the most powerful one since it lets you create any kind of graph you like. Once you provided the numbers, TestData DB distributes them evenly based on the time range of your query.

Dashboards

TestData DB also contains some dashboards with examples.

Click Configuration > Data Sources > TestData DB > Dashboards.
Import the Simple Streaming Example dashboard.

Commit updates to the dashboards

To submit a change to one of the current dashboards bundled with TestData DB, update the revision property. Otherwise the dashboard will not be updated automatically for other Grafana users.

Using test data in issues

If you post an issue on Github regarding time series data or rendering of time series data, we strongly advise you to use this data source to replicate the data. That makes it much easier for the developers to replicate and solve the issue you have.

Zipkin

Sat, 04 Apr 2026 12:26:57 +0000

Zipkin data source

Grafana ships with built-in support for Zipkin, an open source, distributed tracing system. Just add it as a data source and you are ready to query your traces in Explore.

Adding the data source

To access Zipkin settings, click the Configuration (gear) icon, then click Data Sources > Zipkin.

Name	Description
`Name`	The data source name in panels, queries, and Explore.
`Default`	The pre-selected data source for a new panel.
`URL`	The URL of the Zipkin instance. For example, `http://localhost:9411`.
`Basic Auth`	Enable basic authentication for the Zipkin data source.
`User`	Specify a user name for basic authentication.
`Password`	Specify a password for basic authentication.

Trace to logs

Note: This feature is available in Grafana 7.4+.

This is a configuration for the trace to logs feature. Select target data source (at this moment limited to Loki data sources) and select which tags will be used in the logs query.

Data source - Target data source.
Tags - The tags that will be used in the Loki query. Default is 'cluster', 'hostname', 'namespace', 'pod'.
Map tag names - When enabled, allows configuring how Zipkin tag names map to Loki label names. For example, map service.name to service.
Span start time shift - Shift in the start time for the Loki query based on the span start time. In order to extend to the past, you need to use a negative value. Use time interval units like 5s, 1m, 3h. The default is 0.
Span end time shift - Shift in the end time for the Loki query based on the span end time. Time units can be used here, for example, 5s, 1m, 3h. The default is 0.
Filter by Trace ID - Toggle to append the trace ID to the Loki query.
Filter by Span ID - Toggle to append the span ID to the Loki query.

Node Graph

This is a configuration for the beta Node Graph visualization. The Node Graph is shown after the trace view is loaded and is disabled by default.

– Enable Node Graph - Enables the Node Graph visualization.

Query traces

Querying and displaying traces from Zipkin is available via Explore.

The Zipkin query editor allows you to query by trace ID directly or selecting a trace from trace selector. To query by trace ID, insert the ID into the text input.

Use the trace selector to pick particular trace from all traces logged in the time range you have selected in Explore. The trace selector has three levels of nesting:

The service you are interested in.
Particular operation is part of the selected service
Specific trace in which the selected operation occurred, represented by the root operation name and trace duration.

Data mapping in the trace UI

Zipkin annotations are shown in the trace view as logs with annotation value shown under annotation key.

Upload JSON trace file

You can upload a JSON file that contains a single trace to visualize it.

Here is an example JSON:

[
  {
    "traceId": "efe9cb8857f68c8f",
    "parentId": "efe9cb8857f68c8f",
    "id": "8608dc6ce5cafe8e",
    "kind": "SERVER",
    "name": "get /api",
    "timestamp": 1627975249601797,
    "duration": 23457,
    "localEndpoint": { "serviceName": "backend", "ipv4": "127.0.0.1", "port": 9000 },
    "tags": {
      "http.method": "GET",
      "http.path": "/api",
      "jaxrs.resource.class": "Resource",
      "jaxrs.resource.method": "printDate"
    },
    "shared": true
  }
]

Linking Trace ID from logs

You can link to Zipkin trace from logs in Loki by configuring a derived field with internal link. See Loki documentation for details.