Prometheus data source on Grafana Labs

Configure the Prometheus data source

Fri, 03 Apr 2026 19:43:06 +0000

Configure the Prometheus data source

This document provides instructions for configuring the Prometheus data source and explains the available configuration options. Grafana includes built-in support for Prometheus, so you don’t need to install a plugin. For general information on adding a data source to Grafana, refer to Add a data source.

Before you begin

You need the Organization administrator role to configure the data source. You can also configure it via YAML using Grafana provisioning.
Grafana includes a built-in Prometheus data source; no plugin installation is required.
Know which type of Prometheus-compatible database you’re connecting to (Prometheus, Mimir, Cortex, or Thanos), as the configuration options vary by type.
Have your Prometheus server URL ready. If using TLS/SSL, gather any necessary security certificates and client keys.
If using Basic authentication, have your username and password ready.

Configure the data source using the UI

To add the Prometheus data source, complete the following steps:

Click Connections in the left-side menu.
Under Connections, click Add new connection.
Enter Prometheus in the search bar.
Click Prometheus data source.
Click Add new data source in the upper right.

Grafana takes you to the Settings tab where you will set up your Prometheus configuration.

Configuration options

Following is a list of configuration options for Prometheus.

Name - The data source name. Sets the name you use to refer to the data source in panels and queries. Examples: prometheus-1, prom-metrics.
Default - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source.

Connection:

Prometheus server URL - The URL of your Prometheus server.
If Prometheus is running locally, use http://localhost:9090. If it’s hosted on a networked server, provide the server’s URL along with the port where Prometheus is running. Example: http://prometheus.example.orgname:9090.

Note
When running Grafana and Prometheus in separate containers, localhost refers to each container’s own network namespace. This means that localhost:9090 points to port 9090 inside the Grafana container, not on the host machine.

Use the IP address of the Prometheus container, or the hostname if you are using Docker Compose. Alternatively, you can use http://host.docker.internal:9090 to reference the host machine.

Authentication:

There are three authentication options for the Prometheus data source.

Basic authentication - The most common authentication method.
- User - The username you use to connect to the data source.
- Password - The password you use to connect to the data source.
Forward OAuth identity - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.
No authentication - Allows access to the data source without any authentication.

TLS settings:

Note
Use TLS (Transport Layer Security) for an additional layer of security when working with Prometheus. For information on setting up TLS encryption with Prometheus refer to Securing Prometheus API and UI Endpoints Using TLS Encryption. You must add TLS settings to your Prometheus configuration file prior to setting these options in Grafana.

Add self-signed certificate - Check the box to authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file. Required for verifying self-signed TLS certificates.
- CA certificate - Add your certificate.
TLS client authentication - Check the box to enable TLS client authentication.
- Server name - Add the server name, which is used to verify the hostname on the returned certificate.
- Client certificate - The client certificate is generated from a Certificate Authority or its self-signed. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
- Client key - Add your client key, which can also be generated from a Certificate Authority (CA) or be self-signed. The client key encrypts data between the client and server.
Skip TLS verify - Toggle on to bypass TLS certificate validation. Skipping TLS certificate validation is not recommended unless absolutely necessary or for testing purposes.

HTTP headers:

Pass along additional information and metadata about the request or response.

Header - Add a custom header. This allows custom headers to be passed based on the needs of your Prometheus instance.
Value - The value of the header.

Advanced settings:

Following are optional configuration settings you can configure for more control over your data source.

Advanced HTTP settings:
- Allowed cookies - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
- Timeout - The HTTP request timeout, must be in seconds.

Alerting:

Manage alerts via Alerting UI -Toggled on by default. This enables data source-managed rules in Grafana Alerting for this data source. For Mimir, it enables managing data source-managed rules and alerts. For Prometheus, it only supports viewing existing rules and alerts, which are displayed as data source-managed. Change this by setting the default_manage_alerts_ui_toggle option in the grafana.ini configuration file.
Allow as recording rules target - Toggled on by default. This allows the data source to be selected as a target destination for writing Grafana-managed recording rules. When enabled, this data source will appear in the target data source list when creating or importing recording rules. When disabled, the data source will be filtered out from recording rules target selection. Change this by setting the default_allow_recording_rules_target_alerts_ui_toggle option in the grafana.ini configuration file.

Interval behavior:

Scrape interval - Sets the standard scrape and evaluation interval in Prometheus. The default is 15s. This interval determines how often Prometheus scrapes targets. Set it to match the typical scrape and evaluation interval in your Prometheus configuration file. If you set a higher value than your Prometheus configuration, Grafana will evaluate data at this interval, resulting in fewer data points.
Query timeout - Sets the Prometheus query timeout. The default is 60s. Without a timeout, complex or inefficient queries can run indefinitely, consuming CPU and memory resources.

Query editor:

Default editor - Sets the default query editor. Options are Builder or Code. Builder mode helps you build queries using a visual interface. Code mode is geared for the experienced Prometheus user with prior expertise in PromQL. For more details on editor types refer to Prometheus query editor. You can switch easily code editors in the Query editor UI.
Disable metrics lookup - Toggle on to disable the metrics chooser and metric and label support in the query field’s autocomplete. This can improve performance for large Prometheus instances.

Performance:

Prometheus type - Select the type of your Prometheus-compatible database, such as Prometheus, Cortex, Mimir, or Thanos. Changing this setting will save your current configuration. Different database types support different APIs. For example, some allow regex matching for label queries to improve performance, while others provide a metadata API. Setting this incorrectly may cause unexpected behavior when querying metrics and labels. Refer to your Prometheus documentation to ensure you select the correct type.
Cache level - Sets the browser caching level for editor queries. There are four options: Low, Medium, High, or None. Higher cache settings are recommended for high cardinality data sources.
Incremental querying (beta) - Toggle on to enable incremental querying. Enabling this feature changes the default behavior of relative queries. Instead of always requesting fresh data from the Prometheus instance, Grafana will cache query results and only fetch new records. This helps reduce database and network load.
- Query overlap window - If you are using incremental querying, specify a duration (e.g., 10m, 120s, or 0s). The default is 10m. This is a buffer of time added to incremental queries and this value is added to the duration of each incremental request.
Disable recording rules (beta) - Toggle on to disable the recording rules. When recording rules are disabled, Grafana won’t fetch and parse recording rules from Prometheus, improving dashboard performance by reducing processing overhead..

Other settings:

Custom query parameters - Add custom parameters to the Prometheus query URL, which allow for more control over how queries are executed. Examples: timeout, partial_response, dedup, or max_source_resolution. Multiple parameters should be joined using &.
HTTP method - Select either the POST or GET HTTP method to query your data source. POSTis recommended and selected by default, as it supports larger queries. Select GET if you’re using Prometheus version 2.1 or older, or if your network restricts POST requests. Toggle on
Series limit - Number of maximum returned series. The limit applies to all resources (metrics, labels, and values) for both endpoints (series and labels). Leave the field empty to use the default limit (40000). Set to 0 to disable the limit and fetch everything — this may cause performance issues. Default limit is 40000.
Use series endpoint - Enabling this option makes Grafana use the series endpoint (/api/v1/series) with the match[] parameter instead of the label values endpoint (/api/v1/label/<label_name>/values). While the label values endpoint is generally more performant, some users may prefer the series endpoint because it supports the POST method, whereas the label values endpoint only allows GET requests.

Exemplars:

Support for exemplars is available only for the Prometheus data source. For more information on exemplars refer to Introduction to exemplars. An exemplar is a trace that represents a specific measurement taken within a given time interval.

Click the + sign to add exemplars.

Internal link - Toggle on to enable an internal link. This will display the data source selector, where you can choose the backend tracing data store for your exemplar data.
URL - (Visible if you disable Internal link) Defines the external link’s URL trace backend. You can interpolate the value from the field by using the ${__value.raw} macro.
Data source - (Visible whenInternal link is enabled.) Select the data source that the exemplar will link to from the drop-down.
URL label - Adds a custom display label to override the value of the Label name field.
Label name - The name of the field in the labels object used to obtain the traceID property.
Remove exemplar link - Click the X to remove existing links.

You can add multiple exemplars.

Private data source connect - Only for Grafana Cloud users. Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC refer to Private data source connect (PDC) and Configure Grafana private data source connect (PDC) for steps on setting up a PDC connection.

If you use PDC with SIGv4 (AWS Signature Version 4 Authentication), the PDC agent must allow internet egress tosts.<region>.amazonaws.com:443.

Click Manage private data source connect to open your PDC connection page and view your configuration details.

After you have configured your Prometheus data source options, click Save & test at the bottom to test out your data source connection.

You should see a confirmation dialog box that says:

Successfully queried the Prometheus API.

Next, you can start to visualize data by building a dashboard, or by querying data in the Explore view.

You can also remove a connection by clicking Delete.

Provision the Prometheus data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to Provision Grafana.

Note
After you have provisioned a data source you cannot edit it.

Example of a Prometheus data source configuration:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    jsonData:
      httpMethod: POST
      manageAlerts: true
      allowAsRecordingRulesTarget: true
      prometheusType: Prometheus
      prometheusVersion: 3.3.0
      cacheLevel: 'High'
      disableRecordingRules: false
      seriesEndpoint: false
      timeInterval: 10s # Prometheus scrape interval
      incrementalQueryOverlapWindow: 10m
      exemplarTraceIdDestinations:
        # Field with internal link pointing to data source in Grafana.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          name: traceID

        # Field with external link.
        - name: traceID
          url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'

Azure authentication settings

The Prometheus data source works with Azure authentication. To configure Azure authentication refer to Configure Azure Active Directory (AD) authentication.

In Grafana Enterprise, you need to update the .ini configuration file. Refer to Configuration file location to locate your .ini file.

Add the following setting in the [auth] section of the .ini configuration file:

[auth]
azure_auth_enabled = true

Note
If you are using Azure authentication, don’t enable Forward OAuth identity. Both methods use the same HTTP authorization headers, and the OAuth token will override your Azure credentials.

Recording rules (beta)

You can configure the Prometheus data source to disable recording rules in the data source configuration or provisioning file under disableRecordingRules in jsonData.

Troubleshooting configuration issues

Refer to the following troubleshooting information as needed.

Data doesn’t appear in Metrics Drilldown:

If you have successfully tested the connection to a Prometheus data source or are sending metrics to Grafana Cloud and there is no metric data appearing in Explore, make sure you’ve selected the correct data source from the data source drop-down menu. When using remote_write to send metrics to Grafana Cloud, the data source name follows the convention grafanacloud-stackname-prom.

Prometheus query editor

Fri, 03 Apr 2026 19:43:06 +0000

Prometheus query editor

Grafana provides a query editor for the Prometheus data source to create queries in PromQL. For more information about PromQL, see Querying Prometheus. The Prometheus query editor is located on the Explore page. You can also access the PostgreSQL query editor from a dashboard panel. Click the ellipsis in the upper right of the panel and select Edit.

For general documentation on querying data sources in Grafana, refer to Query and transform data. For options and functions common to all query editors, refer to Query editors.

The Prometheus query editor has two modes:

Grafana synchronizes both modes, allowing you to switch between them. Grafana also displays a warning message if it detects an issue with the query while switching modes.

You can configure Prometheus-specific options in the query editor by setting several options regardless of mode.

Builder mode

Builder mode helps you build queries using a visual interface. This option is best for users who have limited experience working with Prometheus and PromQL.

The following video demonstrates how to use the visual Prometheus query builder:

Builder mode contains the following components:

Kick start your query - Click to view a list of predefined operation patterns that help you quickly build queries with multiple operations. These include:
- Rate query starters
- Histogram query starters
- Binary query starters

Click the arrow next to each to see the available options to add to your query.

Explain - Toggle on to display a step-by-step explanation of all query components and operations.

Builder/Code - Click the corresponding Builder or Code tab on the toolbar to select an editor mode.

If you select Builder mode you will see the following options:

Metric - Select a metric from the drop-down. Click the icon to open Metrics explorer, where you can search for metrics by name and filter by type if your instance has a large number of metrics. Refer to Metrics explorer for more detail on using this feature.
Label filters - Select label filters from the drop-down. Select an operator and a value. Select desired labels and their values from the drop-down list. When a metric is selected, the data source requests available labels and their values from the server. Use the + button to add a label, and the x button to remove a label.

Click + Operations to select from a list of operations including Aggregations, Range functions, Functions, Binary operations, Trigonometric and Time functions. You can select multiple operations. Refer to Operations for more detail.

Options:

Legend- Lets you customize the name for the time series. You can use a predefined or custom format.
- Auto - Displays unique labels. Also displays all overlapping labels if a series has multiple labels.
- Verbose - Displays all label names.
- Custom - Lets you customize the legend using label templates. For example, {{hostname}} is replaced with the value of the hostname label. To switch to a different legend mode, clear the input and click outside the field.
Min step - Sets the minimum interval between data points returned by the query. For example, setting this to 1h suggests that data is collected or displayed at hourly intervals. This setting supports the $__interval and $__rate_interval macros. Note that the time range of the query is aligned to this step size, which may adjust the actual start and end times of the returned data.
Format - Determines how the data from your Prometheus query is interpreted and visualized in a panel. Choose from the following format options:
- Time series - The default format. Refer to Time series kind formats for information on time series data frames and how time and value fields are structured.
- Table - Displays data in table format. This format works only in a Table panel.
- Heatmap - Displays Histogram-type metrics in a Heatmap panel by converting cumulative histograms to regular ones and sorting the series by the bucket bound. Converts cumulative histogram data into regular histogram format and sorts the series by bucket boundaries for proper display.
Type - This setting determines the query type. These include:
- Both - The default option. Returns results for both a Range query and an Instant query.
- Range - Returns a range vector - a set of time series a set of time series where each series includes multiple data points over a period of time. You can choose to visualize the data as lines, bars, points, stacked lines, or stacked bars.
- Instant - Returns a single data point per series — the most recent value within the selected time range. The results can be displayed in a table or as raw data. To visualize instant query results in a time series panel, start by adding field override, then add a property to the override called Transform, and set the Transform value to Constant in the drop-down. For more information, refer to the Time Series Transform option documentation.

Note
Grafana adjusts the query time range to align with the dynamically calculated step interval. This alignment ensures consistent metric visualization and supports Prometheus’s result caching requirements. However, this alignment can cause minor visual differences, such as a slight gap at the graph’s right edge or a shifted start time. For example, a 15s step aligns timestamps to Unix times divisible by 15 seconds. A 1w minstep aligns the range to the start of the week, which for Prometheus is Thursday at 00:00 UTC.

Exemplars - Toggle on to run a query that includes exemplars in the graph. Exemplars are unique to Prometheus. For more information see Introduction to exemplars.

Note
There is no option to add exemplars with an Instant query type.

Filter metrics

When you are ready to create a query, you can choose the specific metric name from the drop-down list under Metric. The data source provides the list of available metrics based on the selected time range. You can also enter text into the selector when the drop-down is open to search and filter the list.

Metrics explorer in Builder mode

Click the Open book icon to open the Metrics explorer, where you can search for and filter all the metrics in your instance.

If you would like to explore your metrics in the query builder further, you can open the Metrics explorer by clicking the first option in the metric select component of the query builder.

The Metrics explorer displays all metrics in a paginated table list. The list shows the total number of metrics, as well as the name, type, and description for each metric. You can enter text into the search input to filter results. You can also filter by type.

The following options are included under the Additional Settings drop-down:

Include description in search: Toggle on to search by both name and description.
Include results with no metadata: Toggle on to include metrics that lack type or description metadata.
Disable text wrap: Toggle on to disable text wrapping.
Enable regex search: Toggle on to filter metric names by regex search, which uses an additional call on the Prometheus API.

Note
The Metrics explorer (Builder mode) and Metrics browser (Code mode) are separate elements. The Metrics explorer does not have the ability to browse labels yet, but the Metrics browser can display all labels on a metric name.

Operations

Select the + Operations button to add operations to your query.

The query editor groups operations into the following sections:

Aggregations - for additional information see Aggregation operators.
Range functions - for additional information see Functions.
Functions - for additional information see Functions.
Binary operations - for additional information see Binary operators.
Trigonometric - for additional information see Trigonometric functions.
Time functions - for additional information see Functions.

All operations have function parameters under the operation header. Click the operator to see a full list of supported functions. Some operations allow you to apply specific labels to functions.

Some operations are only valid when used in a specific order. If you add an operation in a way that would create an invalid or illogical query, the query editor automatically places it in the correct position to maintain a valid query structure.

Hints

The query editor can detect which operations are most appropriate for certain selected metrics. When it does, it displays a hint next to the + Operations button.

To add the operation to your query, click the Hint.

When you are satisfied with your query, click Run query.

Code mode

Code mode is for the experienced Prometheus user with prior expertise in PromQL, Prometheus’ query language. The Code mode editor allows you to create queries just as you would in Prometheus. For more information about PromQL see Querying Prometheus.

The user interface (UI) also lets you select metrics, labels, filters, and operations.

You can write complex queries using the text editor with autocompletion features and syntax highlighting. Code mode’s autocomplete feature works automatically while typing. The query editor can autocomplete static functions, aggregations, keywords, and also dynamic items like metrics and labels. The autocompletion drop-down includes documentation for the suggested items where available.

It also contains a Metrics browser to further help you write queries. To open the Metrics browser, click the arrow next to Metrics browser.

Metrics browser in Code mode

The Metrics browser locates metrics and selects relevant labels to help you build basic queries. When you click Metrics browser in Code mode, it displays all available metrics and labels. If supported by your Prometheus instance, each metric also displays its HELP and TYPE as a tooltip.

When you select a metric under Step 1, the browser narrows down the available labels to show only the ones applicable to the metric. You can then select one or more labels shown in Step 2. Select one or more values in Step 3 for each label to tighten your query scope. In Step 4, you can select Use query to run the query, Use as rate query to add the rate operation to your query ($__rate_interval), Validate selector to verify the selector is valid and show the number of series found, or Clear to clear your selections and start over.

Note
If you don’t remember the exact metric name, you can start by selecting a few labels to filter the list. This helps you find relevant label values and narrow down your options.

All lists in the Metrics browser include a search field to quickly filter metrics or labels by keyword. In the Values section, there’s a single search field that filters across all selected labels, making it easier to find matching values. For example, if you have labels like app, job, and job_name, only one of them might contain the value you’re looking for.

When you are satisfied with your query, click Run query.

Incremental dashboard queries (beta)

Starting with Grafana v10, the Prometheus data source supports incremental querying for live dashboards. Instead of re-querying the entire time range on each refresh, Grafana can fetch only new data since the last query.

You can enable or disable this feature in the data source configuration or provisioning file using the incrementalQuerying field in jsonData.

You can also control the overlap between consecutive incremental queries using the incrementalQueryOverlapWindow field in jsonData. By default, this is set to 10m (10 minutes). Increasing the incrementalQueryOverlapWindow value increases the time range covered by each incremental query. This can help in environments where the most recent data may be delayed or incomplete.

Prometheus template variables

Fri, 03 Apr 2026 19:43:06 +0000

Prometheus template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana refers to such variables as template variables. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the displayed data.

For an introduction to templating and template variables, refer to Templating and Add and manage variables.

Use query variables

Grafana supports several types of variables, but Query variables are specifically used to query Prometheus. They can return a list of metrics, labels, label values, query results, or series.

Select a Prometheus data source query type and enter the required inputs:

Query Type	Input(* required)	Description	Used API endpoints
`Label names`	`metric`	Returns a list of all label names matching the specified `metric` regex.	/api/v1/labels
`Label values`	`label`*, `metric`	Returns a list of label values for the `label` in all metrics or the optional metric.	/api/v1/label/`label`/values or /api/v1/series
`Metrics`	`metric`	Returns a list of metrics matching the specified `metric` regex.	/api/v1/label/__name__/values
`Query result`	`query`	Returns a list of Prometheus query result for the `query`.	/api/v1/query
`Series query`	`metric`, `label` or both	Returns a list of time series associated with the entered data.	/api/v1/series
`Classic query`	classic query string	Deprecated, classic version of variable query editor. Enter a string with the query type using a syntax like the following: `label_values(<metric>, <label>)`	all

For details on metric names, label names, and label values, refer to the Prometheus documentation.

Query options

With the query variable type, you can set the following query options:

Option	Description
Data source	Select your data source from the drop-down list.
Select query type	Options are `default`, `value` and `metric name`. Each query type hits a different Prometheus endpoint.
Regex	Optional, if you want to extract part of a series name or metric node segment.
Sort	Default is `disabled`. Options include `alphabetical`, `numerical`, and `alphabetical case-sensitive`.
Refresh	When to update the values for the variable. Options are `On dashboard load` and `On time range change`.

Selection options

The following selection options are available:

Multi-value - Check this option to enable multiple values to be selected at the same time.
Include All option - Check this option to include all variables.

Use interval and range variables

You can use global built-in variables in query variables, including the following:

$__interval
$__interval_ms
$__range
$__range_s
$__range_ms

For details, refer to Global built-in variables. The label_values function doesn’t support queries, so you can use these variables in conjunction with the query_result function to filter variable queries.

Configure the variable’s refresh setting to On Time Range Change to ensure it dynamically queries and displays the correct instances when the dashboard time range is modified.

Example:

Populate a variable with the top 5 busiest request instances ranked by average QPS over the dashboard’s selected time range:

query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
Regex: /"([^"]+)"/

Populate a variable with the instances having a certain state over the time range shown in the dashboard, using $__range_s:

query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
Regex:

Use `$__rate_interval`

Grafana recommends using $__rate_interval with the rate and increase functions instead of $__interval or a fixed interval value. Since $__rate_interval is always at least four times the scrape interval, it helps avoid issues specific to Prometheus, such as gaps or inaccuracies in query results.

For example, instead of using the following:

rate(http_requests_total[5m])

or:

rate(http_requests_total[$__interval])

Use the following:

rate(http_requests_total[$__rate_interval])

The value of $__rate_interval is calculated as:

max($__interval + scrape_interval, 4 * scrape_interval)

Here, scrape_interval refers to the min step setting (also known as query_interval) specified per PromQL query, if set. If not, Grafana falls back to the Prometheus data source’s scrape interval setting.

The min interval setting in the panel is modified by the resolution setting, and therefore doesn’t have any effect on scrape interval.

For details, refer to the Grafana blog $__rate_interval for Prometheus rate queries that just work.

Choose a variable syntax

The Prometheus data source supports two variable syntaxes for use in the Query field:

$<varname>, for example rate(http_requests_total{job=~"$job"}[$_rate_interval]), which is easier to read and write but does not allow you to use a variable in the middle of a word.
[[varname]], for example rate(http_requests_total{job=~"[[job]]"}[$_rate_interval])

If you’ve enabled the Multi-value or Include all value options, Grafana converts the labels from plain text to a regex-compatible string, which requires you to use =~ instead of =.

Use the ad hoc filters variable type

Prometheus supports the special ad hoc filters variable type, which allows you to dynamically apply label/value filters across your dashboards. These filters are automatically added to all Prometheus queries, allowing dynamic filtering without modifying individual queries.

Troubleshoot Prometheus data source issues

Fri, 03 Apr 2026 19:43:06 +0000

Troubleshoot Prometheus data source issues

This document provides troubleshooting information for common errors you may encounter when using the Prometheus data source in Grafana.

Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to Prometheus.

Failed to connect to Prometheus

Error message: “There was an error returned querying the Prometheus API”

Cause: Grafana cannot establish a network connection to the Prometheus server.

Solution:

Verify that the Prometheus server URL is correct in the data source configuration.
Check that Prometheus is running and accessible from the Grafana server.
Ensure the URL includes the protocol (http:// or https://).
Verify the port is correct (the Prometheus default port is 9090).
Ensure there are no firewall rules blocking the connection.
If Grafana and Prometheus are running in separate containers, use the container IP address or hostname instead of localhost.
For Grafana Cloud, ensure you have configured Private data source connect if your Prometheus instance is not publicly accessible.

Request timed out

Error message: “context deadline exceeded” or “request timeout”

Cause: The connection to Prometheus timed out before receiving a response.

Solution:

Check the network latency between Grafana and Prometheus.
Verify that Prometheus is not overloaded or experiencing performance issues.
Increase the Query timeout setting in the data source configuration under Interval behavior.
Check the Grafana server timeout configuration for server-level timeout settings.
Reduce the time range or complexity of your query.
Check if any network devices (load balancers, proxies) are timing out the connection.

Failed to parse data source URL

Error message: “Failed to parse data source URL”

Cause: The URL entered in the data source configuration is not valid.

Solution:

Verify the URL format is correct (for example, http://localhost:9090 or https://prometheus.example.com:9090).
Ensure the URL includes the protocol (http:// or https://).
Remove any trailing slashes or invalid characters from the URL.

Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

Unauthorized (401)

Error message: “401 Unauthorized” or “Authorization failed”

Cause: The authentication credentials are invalid or missing.

Solution:

Verify that the username and password are correct if using basic authentication.
Check that the authentication method selected matches your Prometheus configuration.
If using a reverse proxy with authentication, verify the credentials are correct.
For AWS SigV4 authentication, verify the IAM credentials and permissions. Alternatively, consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.

Forbidden (403)

Error message: “403 Forbidden” or “Access denied”

Cause: The authenticated user does not have permission to access the requested resource.

Solution:

Verify the user has read access to the Prometheus API.
Check Prometheus security settings and access control configuration.
If using a reverse proxy, verify the proxy is not blocking the request.
For AWS Managed Prometheus, verify the IAM policy grants the required permissions. Alternatively, consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.

Query errors

The following errors occur when there are issues with PromQL syntax or query execution.

Query syntax error

Error message: “parse error: unexpected character” or “bad_data: 1:X: parse error”

Cause: The PromQL query contains invalid syntax.

Alternative cause: A proxy between Grafana and Prometheus requires authentication. When proxy authentication fails, the proxy redirects the request to an HTML authentication page. Grafana cannot parse the HTML response, which results in a parse error. This appears to be a query issue but is actually a proxy authentication issue.

Solution:

Check your query syntax for typos or invalid characters.
Verify that metric names and label names are valid identifiers.
Ensure string values in label matchers are enclosed in quotes.
Use the Prometheus expression browser to test your query directly.
Refer to the Prometheus querying documentation for syntax guidance.
If you have a proxy between Grafana and Prometheus, verify that proxy authentication is correctly configured. Check your proxy logs for authentication failures or redirects.

Query returns no data for a metric

Symptom: The query returns no data and the visualization is empty.

Cause: The specified metric does not exist in Prometheus, or there is no data for the selected time range.

Solution:

Verify the metric name is spelled correctly.
Check that the metric is being scraped by Prometheus.
Use the Prometheus API to browse available metrics at /api/v1/label/__name__/values.
Use the target metadata API to verify which metrics a target exposes.
Verify the time range includes data for the metric.

Query timeout limit exceeded

Error message: “query timed out in expression evaluation” or “query processing would load too many samples”

Cause: The query took longer than the configured timeout limit or would return too many samples.

Solution:

Reduce the time range of your query.
Add more specific label filters to limit the data scanned.
Increase the Query timeout setting in the data source configuration.
Use aggregation functions like sum(), avg(), or rate() to reduce the number of time series.
Increase the query.timeout or query.max-samples settings in Prometheus if you have admin access.

Too many time series

Error message: “exceeded maximum resolution of 11,000 points per timeseries” or “maximum number of series limit exceeded”

Cause: The query is returning more time series or data points than the configured limits allow.

Solution:

Reduce the time range of your query.
Add label filters to limit the number of time series returned.
Increase the Min interval or Resolution in the query options to reduce the number of data points.
Use aggregation functions to combine time series.
Adjust the Series limit setting in the data source configuration under Other settings.

Invalid function or aggregation

Error message: “unknown function” or “parse error: unexpected aggregation”

Cause: The query uses an invalid or unsupported PromQL function.

Solution:

Verify the function name is spelled correctly and is a valid PromQL function.
Check that you are using the correct syntax for the function.
Ensure your Prometheus version supports the function you are using.
Refer to the PromQL functions documentation for available functions.

Configuration errors

The following errors occur when the data source is not configured correctly.

Invalid Prometheus type

Error message: Unexpected behavior when querying metrics or labels

Cause: The Prometheus type setting does not match your actual Prometheus-compatible database.

Solution:

Open the data source configuration in Grafana.
Under Performance, select the correct Prometheus type (Prometheus, Cortex, Mimir, or Thanos).
Different database types support different APIs, so setting this incorrectly may cause unexpected behavior.

Scrape interval mismatch

Symptom: Data appears sparse, or rate() queries return no data or incomplete results.

Cause: The Scrape interval setting in Grafana does not match the actual scrape interval in Prometheus. This especially affects rate() queries, which require at least two data points within the specified time window. For example, if your actual scrape interval is 5 minutes but Grafana uses the default (15 seconds for OSS, 1 minute for Grafana Cloud), a query like rate(http_requests_total[1m]) returns no data because there are no data points within that 1-minute window.

Solution:

Check your Prometheus configuration file for the scrape_interval setting.
Update the Scrape interval in the Grafana data source configuration under Interval behavior to match.
Use $__rate_interval instead of hardcoded time windows in rate() queries. This variable automatically adjusts based on your scrape interval.
For more information, refer to $__rate_interval for Prometheus rate queries that just work.

TLS and certificate errors

The following errors occur when there are issues with TLS configuration.

Certificate verification failed

Error message: “x509: certificate signed by unknown authority” or “certificate verify failed”

Cause: Grafana cannot verify the TLS certificate presented by Prometheus.

Solution:

If using a self-signed certificate, enable Add self-signed certificate in the TLS settings and add your CA certificate.
Verify the certificate chain is complete and valid.
Ensure the certificate has not expired.
As a temporary workaround for testing, enable Skip TLS verify (not recommended for production).

TLS handshake error

Error message: “TLS: handshake failure” or “connection reset”

Cause: The TLS handshake between Grafana and Prometheus failed.

Solution:

Verify that Prometheus is configured to use TLS.
Check that the TLS version and cipher suites are compatible.
If using client certificates, ensure they are correctly configured in the TLS client authentication section.
Verify the server name matches the certificate’s Common Name or Subject Alternative Name.

Other common issues

The following issues don’t produce specific error messages but are commonly encountered.

Empty query results

Cause: The query returns no data.

Solution:

Verify the time range includes data in Prometheus.
Check that the metric and label names are correct.
Test the query directly in the Prometheus expression browser.
Ensure label filters are not excluding all data.
For rate or increase functions, ensure the time range is at least twice the scrape interval.

Slow query performance

Cause: Queries take a long time to execute.

Solution:

Reduce the time range of your query.
Add more specific label filters to limit the data scanned.
Increase the Min interval in the query options.
Check Prometheus server performance and resource utilization.
Enable Disable metrics lookup in the data source configuration for large Prometheus instances.
Enable Incremental querying (beta) to cache query results.
Consider using recording rules to pre-aggregate frequently queried data.

Data appears delayed or missing recent points

Cause: The visualization doesn’t show the most recent data.

Solution:

Check the dashboard time range and refresh settings.
Verify the Scrape interval is configured correctly.
Ensure Prometheus has finished scraping the target.
Check for clock synchronization issues between Grafana and Prometheus.
For rate() and similar functions, remember that they need at least two data points to calculate.

Exemplars not showing

Cause: Exemplar data is not appearing in visualizations.

Solution:

Verify that exemplars are enabled in the data source configuration under Exemplars.
Check that your Prometheus version supports exemplars (2.26+).
Ensure your instrumented application is sending exemplar data.
Verify the tracing data source is correctly configured for the exemplar link.
Enable the Exemplars toggle in the query editor.

Alerting rules not visible

Cause: Prometheus alerting rules are not appearing in the Grafana Alerting UI.

Solution:

Verify that Manage alerts via Alerting UI is enabled in the data source configuration.
Check that Prometheus has alerting rules configured.
Ensure Grafana can access the Prometheus rules API endpoint.
Note that for Prometheus (unlike Mimir), the Alerting UI only supports viewing existing rules, not creating new ones.

Get additional help

If you continue to experience issues after following this troubleshooting guide:

Check the Prometheus documentation for API and PromQL guidance.
Review the Grafana community forums for similar issues.
Contact Grafana Support if you are a Cloud Pro, Cloud Contracted, or Enterprise user.
When reporting issues, include:
- Grafana version
- Prometheus version and type (Prometheus, Mimir, Cortex, Thanos)
- Error messages (redact sensitive information)
- Steps to reproduce
- Relevant configuration such as data source settings, query timeout, and TLS settings (redact tokens, passwords, and other credentials)

Prometheus data source on Grafana Labs

Configure the Prometheus data source

Configure the Prometheus data source

Before you begin

Configure the data source using the UI

Configuration options

Provision the Prometheus data source

Azure authentication settings

Recording rules (beta)

Troubleshooting configuration issues

Prometheus query editor

Prometheus query editor

Builder mode

Filter metrics

Metrics explorer in Builder mode

Operations

Hints

Code mode

Metrics browser in Code mode

Incremental dashboard queries (beta)

Prometheus template variables

Prometheus template variables

Use query variables

Query options

Selection options

Use interval and range variables

Use $__rate_interval

Choose a variable syntax

Use the ad hoc filters variable type

Troubleshoot Prometheus data source issues

Troubleshoot Prometheus data source issues

Connection errors

Failed to connect to Prometheus

Request timed out

Failed to parse data source URL

Authentication errors

Unauthorized (401)

Forbidden (403)

Query errors

Query syntax error

Query returns no data for a metric

Query timeout limit exceeded

Too many time series

Invalid function or aggregation

Configuration errors

Invalid Prometheus type

Scrape interval mismatch

TLS and certificate errors

Certificate verification failed

TLS handshake error

Other common issues

Empty query results

Slow query performance

Data appears delayed or missing recent points

Exemplars not showing

Alerting rules not visible

Get additional help

Use `$__rate_interval`