Investigate trends and spikes on Grafana Labs

Choose root or full span data

Fri, 03 Apr 2026 19:43:06 +0000

Choose root or full span data

Tracing data is highly structured and annotated and reflects events that happen in your services. You can choose the type of services you want to observe and think about.

By default, Traces Drilldown displays information about root spans. You can change this by using the selector in the filter bar.

Use Root spans for trace‑level insights and faster performance (one span/trace). Root spans show only traces where the root span has an error value.
Use All spans when you need to drill down into every operation within those traces. All spans show traces containing errors anywhere in the call chain, for example, database errors or downstream service failures.

For most observability use cases, start with root spans to get accurate service-level metrics, then switch to all spans when you need to investigate specific internal operations or errors deeper in the call chain.

Query root spans only

Using Root spans, you get exactly one span per trace (the root span or the first span in a trace) so you see one data point per trace in your results.

When to use:

High-level or service-level investigations, for example, error rate by root operation.
Fast filtering by trace-wide metrics, for example, trace duration, and success versus failure at the entry point.

Benefits:

End-to-end view: Root spans represent the complete, end‑to‑end request or job. Querying only root spans ensures you measure the full request lifecycle, exactly what your RED (Rate, Errors, Duration) metrics are built on. Duration and error‑rate histograms reflect user‑facing operations.
Speed: Only inspects the first span per trace.

Query all spans

With this option you query every matching span in every trace.

When to use:

Deep‑dive troubleshooting where you need every operation in the call graph.
Filtering by child‑span attributes, for example, database calls and background jobs.

Trade‑offs:

Skewed RED metrics: Unless used with an appropriate filter, aggregating duration or error rates across every span inflates counts and misrepresents true end‑to‑end performance. Your RED metrics become a mix of server, client, database, and internal spans. The average latency and error rates no longer align with user‑facing operations.
Performance: Scanning all spans is heavier, especially in wide or deep traces.
Result size: You may hit maximum spans per span-set limits if your traces are large.

Note
Errors can cascade through downstream services, and error counts differ from error rates. When using All spans, verify the error type on spans and consider rate (errors/requests) to avoid overestimating the impact.

Choose a RED metric

Fri, 03 Apr 2026 19:43:06 +0000

Choose a RED metric

Traces Drilldown uses RED metrics generated from your tracing data to guide your investigation. In this context, RED metrics mean:

Rates show the rate of incoming spans per second.
Errors show spans that are failing.
Duration displays the amount of time those spans take; represented as a heat map that shows response time and latency.

When you select a RED metric, the tabs underneath the metric selection change to match the context. Breakdown and Comparison tabs are always present regardless of which metric you choose. For example, selecting Duration displays Root cause latency and Slow traces tabs in addition to Breakdown and Comparison. Choosing Errors shows Root cause errors, Exceptions, and Errored traces tabs in addition to Breakdown and Comparison. Rate shows Service structure and Traces tabs in addition to Breakdown and Comparison. These tabs are used when you analyze tracing data.

To choose a RED metric:

Select a graph to select a Rate, Errors, or Duration metric type. Notice that your selection changes the first drop-down list on the filter bar.
Optional: Select the signal you want to observe. Root spans is the default selection.
Look for spikes or trends in the data to help identify issues.

Tip
If no data or limited data appears, refresh the page. Verify that you have selected the correct data source in the Data source drop-down as well as a valid time range.

Analyze tracing data

Fri, 03 Apr 2026 19:43:06 +0000

Analyze tracing data

To further analyze filtered spans, use tabs that change with the selected metric, such as Breakdown, Comparison, Service structure, Root cause errors, Root cause latency, Exceptions, and Trace list.

When you select a RED metric, the tabs change to match the context.

Each tab provides a brief explanation about the information provided.

Use the Breakdown tab

The Breakdown tab splits the selected metric by the values of a chosen resource or span attribute. When you’re using Duration metrics, Breakdown shows the 90th percentile duration for every value of the selected attribute and orders values by average duration. When you select Rate, Breakdown orders the sequence of attributes by their rate of requests per second, with errors colored red.

You can change the Scope to show Resource or Span.

Using the Attributes sidebar, you can group the selected metric by different attributes. For example, if you have selected Errors as a metric type and then choose the resource.service.name attribute, the displayed results show the number of errors sorted by the resource.service.name with the most matches.

Use the Attributes sidebar to select the attribute for Group by. You can search, scope by Resource or Span, and use Favorites for quick access. Attributes already in your Filters are listed at the top of the Attributes sidebar.

By default, the selected attribute is your first Favorite or resource.service.name. You can reorder Favorites to change this default. The Attributes sidebar displays other commonly used resource level attributes such as cluster, environment, and namespace.

Choose percentiles (Duration only)

When you select the Duration metric, use the Percentiles selector to choose which duration percentiles to plot and aggregate against. The default is 90th percentile (p90). You can select one or more of: p50, p75, p90, p95, p99. If you deselect all values, Traces Drilldown applies p90 by default.

The percentile choice drives what Duration values are summarized and shown in the Breakdown tab. The selector appears only when Duration is selected.

Use the Comparison tab

The Comparison tab helps you surface and rank which span attributes are most correlated with the selected metric so you can immediately spot what’s driving your trace-level issues.

Upon selecting a metric, the tab computes, for each resource or span attribute, how strongly that attribute value differs between the selected subset (selection) and all other spans (baseline). It lists attribute‑value pairs in descending order of that difference, so the top entries are those most uniquely associated with your signal of interest.

If you’re viewing the Rate or Errors metrics, the selection contains all spans with errors, while the baseline contains all spans without errors.
If you’re viewing the Duration metric, by default the selection contains the slowest spans above the 90th percentile, while the baseline contains all other spans. You can manually adjust the selection on the duration heatmap.

The behavior of the comparison also differs depending upon the RED metric you’ve chosen. For example, if you’re viewing Errors metrics, the comparison shows the attribute values that correlate with errors. However, if you’re viewing Duration metrics, the comparison shows the attributes that correlate with high latency.

Use the Attributes sidebar to switch between All (overall differences) and a specific attribute (per‑value differences). Favorites, search, and scope controls work the same as in Breakdown.

Focus on individual attributes with Inspect

Inspect lets you break down and view individual attribute values from a comparison. The overview shows each attribute with its single highest-difference value and an Inspect button, but you can’t see all values for that attribute at once.

When you select Inspect, the app focuses on that one attribute and shows a visualization for every value it contains. For example, selecting Inspect on resource.service.name shows every service with its baseline and selection bars, so you can immediately see which services are most correlated with the selected metric.

Use the Structure tab

The Structure tab lets you extract and view aggregate data from your traces. The tab name differs depending on the metric you’re viewing:

Rate provides Service structure
Errors provides Root cause errors
Duration provides Root cause latency

For Rate, the Service structure tab shows you how your applications talk to each other to fulfill requests. Use this tab to analyze the service structure of the traces that match the current filters.

For Errors, the Root cause errors tab shows the structure of errors beneath your selected filters. Use this tab to immediately see the chain of errors that are causing issues higher up in traces.

When you select Duration metrics, the Root cause latency tab shows the structure of the longest running spans so you can analyze the structure of slow spans.

The pictured spans are an aggregated view compiled using spans from multiple traces.

Use the Exceptions tab (Errors metric only)

Use the Exceptions tab to see which exception messages are occurring within your current filters and time range. Exceptions are grouped by message so you can identify the most frequent failures and access the affected traces.

Examples:

Inspect individual exceptions by clicking a message to open the Trace list pre-filtered for that message, so you can inspect individual traces immediately.
Narrow exceptions by service, environment, namespace, or any span/resource attribute by combining with the Filters bar.

The Exceptions tab has a table with the following columns:

Exception Details: Exception type, message text, emitting service, and last seen time grouped by unique exception
Occurrences: Count of matching exceptions for the selected range
Frequency: Sparkline of occurrences over time

Each row includes Include and Exclude buttons so you can filter the current investigation to focus on or eliminate a specific exception.

The Exceptions tab is available when the Errors metric is selected and exceptions are present. The tab respects the current time range and filters. For example, when Errors are selected, Exceptions works with both Root spans and All spans selections. The results reflect the data in scope.

If you notice a rise in the Errors metric, you can use the Exceptions tab to investigate the issue.

Select Errors as the metric.
Open the Exceptions tab.
The top row shows Payment request failed. Too many requests (error code 429) with rising occurrences and a spiky sparkline.
Click the message to jump to the Trace list pre-filtered by that exception.
Sort by Duration to find the most impacted requests, then open a trace to inspect retries and upstream dependencies.

Use the Trace list tab

Each RED metric has a trace list:

Rate provides a tab that lists Traces.
Errors provides a list of traces with errors.
Duration provides a list of Slow traces.

From this view, you can add additional attributes to new columns using Add extra columns.

Use the Attributes sidebar to add columns. Select multiple attributes to include them as table columns. Use Search attributes and Favorites to find attributes. Attributes already in your Filters are listed at the top of the Attributes sidebar.

Change the selected time range

Use the time picker at the top right to modify the data shown in Traces Drilldown.

You can select a time range of up to 24 hours in duration. This time range can be any 24-hour period in your configured trace data retention period. The default is 30 days.

For more information about the time range picker, refer to Use dashboards.

Open a trace by ID

Use the header’s Trace ID input to open a specific trace:

Paste a trace ID into the Trace ID input in the header.
Press Enter to open the trace in the drawer.

When you open a trace from the results list, the Trace ID input clears to avoid stale values.

If a time range is selected, Traces Drilldown searches within the currently selected time range. Otherwise, it searches across all time. If a trace isn’t found, an error message appears—verify the ID and widen the time range if necessary.

This behavior depends on the Tempo data source configuration. Refer to Tempo data source configuration for more information.

Refine your investigation with filters

Fri, 03 Apr 2026 19:43:06 +0000

Refine your investigation with filters

Use filters to focus your investigation on specific data.

Filters are available on the Breakdown and Comparison views. Refer to Analyze tracing data to learn how to use these views.

Add filters

Each time you add a filter, the condition appears in the list of filters at the top of the page. The list of filters expands as you investigate and explore your tracing data using Traces Drilldown.

Refine your investigation by adding filters.
Optional: Use the tabs underneath the metrics selection to provide insights into breakdowns, comparisons, latency, and other explorations.
Choose filters to focus on problem areas. Each selected filter is added to the Filters bar at the top of the page. You can select filters on the Comparison and Breakdown tabs in the following ways:
- Select Add to filters.
- Use the Filters bar near the top.
- Attributes shown with a filter icon in the Attributes sidebar are already applied in your current Filters. The Attributes sidebar helps you pick and favorite attributes used for grouping, comparison, and Trace list columns. Refer to the UI reference.

Example

Let’s say that you want to investigate a spike in root spans with errors that are longer than 200ms.

Select Root spans.
Select the Errored traces tab.
In Filter by label values, enter span:duration, select greater than (>) from the drop-down list, and then enter 200ms.
After the data updates, sort the table by the Duration column.

Modify a filter

Selecting an option for a filter automatically updates the displayed data. If there are no matches, the app displays a “No data for selected query” message.

To modify an applied filter:

Select the filter to modify in the filter bar.
Select an option from the drop-down list.

You can also click in the Filters bar to add filters using drop-down lists.

Remove filters

To remove a filter, select Remove filter (X) at the end of the filter you want to remove.

Change the time range

Use the time picker at the top right to modify the data shown in Traces Drilldown.

You can select a time range of up to 24 hours in duration. By default, this time range can be any 24-hour period in your configured trace data retention period. The default retention period is 30 days. Your configuration may vary from these values.

For more information about the time range picker, refer to Use dashboards.

View exemplars

Fri, 03 Apr 2026 19:43:06 +0000

View exemplars

Exemplars connect your high-level metric trends to the individual traces that caused them, making it much faster to move from “something’s wrong” to “here’s exactly what happened.”

An exemplar is a specific trace representative of measurement taken in a given time interval. While metrics excel at giving you an aggregated view of your system, traces give you a fine-grained view of a single request; exemplars are a way to link the two.

Use exemplars to help isolate problems within your data distribution by pinpointing query traces exhibiting high latency within a time interval. After you localize the latency problem to a few exemplar traces, you can combine it with additional system based information or location properties to perform a root cause analysis faster, leading to quick resolutions to performance issues.

Think of exemplars as “bookmarks” in your metric data that point to actual traces. When you see a spike in error rates or latency in your metrics, exemplars let you click directly on a specific data point and jump to the actual trace that contributed to that metric.

For more information, refer to Introduction to exemplars.

Exemplars in Traces Drilldown

In Traces Drilldown, exemplar data is represented by a small diamond next to the bar graphs. You can view the exemplar information by hovering the cursor over the small diamond.

As you view metrics in the Breakdown, Service structure, or other investigation tabs, look for small diamond icons next to the bar chart metrics.

When you hover your cursor over a diamond, a tooltip appears showing:

Exemplar details
Trace information
Timestamp and metadata

Tip
Not seeing exemplars? Refer to Exemplars in Grafana Cloud for information about how to enable them.

Select View trace to open a slide-out trace panel that displays:

The full trace that the exemplar represents
All spans within that trace
Timing and performance details
Service relationships

Save and load queries

Fri, 03 Apr 2026 19:43:06 +0000

Save and load queries

Save your current filter configuration as a named query so you can quickly return to it later. This is useful when you have a set of filters you use repeatedly, such as filters for a specific service, error type, or latency threshold.

Saved queries capture your current Filters bar and data source. They don’t capture the time range, selected metric, group-by attribute, active tab, or other exploration state.

Save a query

To save your current filters:

Add at least one filter using the Filters bar.
Select the Save button (save icon) in the header bar, next to the data source selector.
Enter a Title and optional Description for the query.
Select Save.

If a saved query with the same filter expression already exists for the current data source, a warning shows the title of the existing entry.

Note
The Save button only appears when at least one filter is applied.

Load a saved query

To load a previously saved query:

Select the Load button (folder-open icon) in the header bar, next to the data source selector.
Browse the list of saved queries.
Select a query to preview its title, description, and filter expression.
Select Select to apply the saved filters.

Loading a saved query replaces your current filters. If the saved query uses a different data source, the data source switches automatically.

To remove a saved query you no longer need, select the Delete (trash) icon next to it in the load modal.

Storage

Saved queries are stored in one of two ways, depending on your Grafana setup:

Saved Queries (query library): When the queryLibrary feature toggle is enabled in Grafana 12.4 or later, saved queries are stored server-side using Grafana’s query library. Queries saved this way are available across browsers and devices.
Local storage: In Grafana open source, older Grafana versions, or when the query library toggle is not enabled, saved queries are stored in your browser’s local storage. These queries are specific to your browser and device and can’t be shared.