Traces Drilldown on Grafana Labs

Access or install Traces Drilldown

Fri, 03 Apr 2026 19:43:06 +0000

Access or install Traces Drilldown

You can access Grafana Traces Drilldown using any of these:

Grafana Cloud: The easiest method, since no setup or installation is required.
Self-managed Grafana open source or Enterprise: You must install the Traces Drilldown plugin.

Traces Drilldown requires Grafana Tempo 2.6 or later with TraceQL metrics configured.

Set up in Grafana Cloud

To use Traces Drilldown with Grafana Cloud, you need the following:

Grafana Cloud account
Grafana stack in Grafana Cloud receiving tracing data from your stack’s default Hosted Traces data source or a Tempo data source

Set up in self-managed Grafana

To use Traces Drilldown with self-managed Grafana open source or Grafana Enterprise, you need:

Your own Grafana instance running 11.6 or later
Tempo 2.6 or later with TraceQL metrics configured
Configured Tempo data source receiving tracing data

Next, access Traces Drilldown.

Install the Traces Drilldown plugin

Note
Grafana v12 and later includes all Drilldown apps, including Traces Drilldown. No separate installation is required. Go to Access Traces Drilldown.

Traces Drilldown is distributed as a Grafana plugin. You can find it in the official Grafana Plugin Directory.

Install in your Grafana instance

You can install Traces Drilldown in your Grafana instance using grafana cli:

grafana cli plugins install grafana-exploretraces-app

Alternatively, follow these steps to install Traces Drilldown in Grafana:

In Grafana, go to Administration > Plugins and data > Plugins.
Search for “Traces Drilldown”.
Select Traces Drilldown.
Click Install.

The plugin is automatically activated after installation.

Install in a Docker container

To install the app in a Docker container, configure the following environment variable:

GF_INSTALL_PLUGINS=grafana-exploretraces-app

Access Traces Drilldown

To access Traces Drilldown, use the following steps:

Open your Grafana stack in a web browser.
In the main menu, select Drilldown > Traces.

Next steps

To learn how to use Traces Drilldown to explore your tracing data:

Concepts

Fri, 03 Apr 2026 19:43:06 +0000

Concepts

Distributed traces provide a way to monitor applications by tracking requests across services. Traces record the details of a request to help understand why an issue is or was happening.

Tracing is best used for analyzing the performance of your system, identifying bottlenecks, monitoring latency, and providing a complete picture of how requests are processed.

To use the Grafana Traces Drilldown app, you should understand these concepts:

Concepts
- Rate, error, and duration metrics
- Traces and spans

Rate, error, and duration metrics

The Traces Drilldown app lets you explore rate, error, and duration (RED) metrics generated from your traces by Tempo.

Useful for investigating	Metric	Meaning
Unusual spikes in activity	Rate	Number of requests per second
Overall issues in your tracing ecosystem	Errors	Number of those requests that are failing
Response times and latency issues	Duration	Amount of time those requests take, represented as a histogram

For more information about the RED method, refer to The RED Method: how to instrument your services.

Traces and spans

A trace represents the journey of a request or an action as it moves through all the nodes of a distributed system, especially containerized applications or microservices architectures. This makes them the ideal observability signal for discovering bottlenecks and interconnection issues.

Traces are composed of one or more spans. A span is a unit of work within a trace that has a start time relative to the beginning of the trace, a duration, and an operation name for the unit of work. It usually has a reference to a parent span in a trace, unless it’s the first span, also known as the root span. It frequently includes key/value attributes that are relevant to the span itself, for example, the HTTP method used in the request, as well as other metadata such as the service name, sub-span events, or links to other spans.

For more information, refer to Understand your data.

Get started with Traces Drilldown

Fri, 03 Apr 2026 19:43:06 +0000

Get started with Traces Drilldown

You can use traces to identify errors in your apps and services and then to optimize and streamline them.

When working with traces, start with the big picture. Investigate using primary signals, RED metrics, filters, and structural or trace list tabs to explore your data. To learn more, refer to Concepts.

Note
Expand your observability journey and learn about the Drilldown apps suite.

Before you begin

To use Grafana Traces Drilldown with Grafana Cloud, you need:

A Grafana Cloud account
A Grafana stack in Grafana Cloud with a configured Tempo data source

To use Traces Drilldown with self-managed Grafana, you need:

Your own Grafana v11.6 or later instance with a configured Tempo data source
Installed Traces Drilldown plugin

For more details, refer to Access Traces Drilldown.

Explore your tracing data

Most investigations follow these steps:

Select the primary signal.
Choose the metric you want to use: rates, errors, or duration.
Define filters to refine the view of your data.
Use the structural or trace list to drill down into the issue.

Give it a try using Grafana Play

With Grafana Play, you can explore and see how it works, learning from practical examples to accelerate your development. This feature can be seen on the Grafana Play site.

Try it

Prefer a step-by-step walkthrough? Refer to the Investigation walkthrough to follow along on play.grafana.org.

Example: Investigate source of errors

This example demonstrates investigation techniques and patterns you can use when investigating errors. It shows how to use advanced features like the Comparison tab and Inspect to find root causes.

For a hands-on walkthrough you can follow step-by-step, refer to the Investigation walkthrough.

For example, you want to uncover the source of errors in your spans. You need to compare the errors in the traces to locate the problem trace. Here’s how this works.

Choose the level of data and a metric

To identify the trouble spot, you want to use raw tracing data instead of only the root span, which is the first span of every trace. Select All spans in the Filters, then choose the Errors metric.

Handle errors

If you’re seeing errors in your traces, here are three common misunderstandings to avoid.

First, not all red spans are application failures. A span marked ’error’ might indicate a timeout or expected validation failure. Check the error message and type before assuming something’s broken.

Second, errors cascade. When one service fails, downstream spans inherit that error status. Look for the root span with the error to find the actual source, not only the last service in the chain.

Finally, remember that error count isn’t the same as error rate. Ten errors might seem alarming, but if you handled ten thousand requests, that’s only 0.1%. Always consider the context.

Check the span attributes and error details. They’ll tell you what really happened.

Correlate attributes

Use the Comparison tab to correlate attributes values with errors. The results are ordered by the difference in those attributes by the highest ones first. This helps you see what’s causing the errors immediately. The Comparison tab analyzes the difference between two sets of traces:

Green bars (Baseline): Normal/healthy trace behavior
Red bars (Selection): Current selection with status = error filter

The view compares your selection (red) to the baseline (green) and ranks attributes by the largest difference. This indicates a significant spike in HTTP 500 (Internal Server Error) responses during your selected time range. The visualization highlights that:

500 errors aren’t normal for this system, they don’t appear in the baseline comparison
There were 500 traces containing HTTP 500 status codes during the error period
This represents a 100% deviation from normal behavior

Click Add to filters to narrow the investigation to these values, or choose Inspect to explore the full distribution.

Hovering over any of the bars shows a tooltip with information about the value and the percentage of the total.

Inspect the problem

Select Inspect on a card to drill into the distribution for that attribute. In this example, selecting Inspect on span.http.status_code shows the distribution by value. Using this view shows the following:

Normal state: All requests completed successfully (200/201)
Error state: Significant portion return 500 errors
Root cause: something caused the internal server errors during the selected time frame

Use Add to filters on the 500 card to keep only error spans and continue the investigation.

Use Root cause errors

Select Root cause errors for an aggregated view of all of the traces that have errors in them. This screen provides critical insights into where and how the HTTP 500 error occurred in your distributed system.

Using this view, you can see that the Frontend > Recommendations services have problems. Specifically, that the /api/pizza endpoint chain is failing.

To view additional details, click the link icon and select View linked span to open the trace drawer.

Error spans have a red icon next to them. Select the down arrow next to the span with an error to see details.

What you learned

This example demonstrated how to:

Use All spans to find errors deeper in the call chain
Understand common error investigation misunderstandings
Use the Comparison tab to correlate attributes with errors
Use Inspect to drill into attribute distributions
Use Root cause errors to see error chain structures

For a step-by-step walkthrough you can follow along, refer to the Investigation walkthrough.

Determine your use case

Fri, 03 Apr 2026 19:43:06 +0000

Determine your use case

Before you start investigating, identify your use case to choose the right approach and metric type.

Your use case determines which RED metric you start with and how you navigate through your tracing data. You might know exactly what’s wrong, or you might need to explore to find issues.

Why this concept matters

Identifying your use case helps you start your investigation efficiently. It guides you to the right RED metric and workflow, saving time and helping you find root causes faster.

Grafana Traces Drilldown supports three main types of investigations: error investigation, performance analysis, and activity monitoring. Each use case has a different starting point and workflow.

How it works

Each use case maps to a specific RED metric and investigation workflow. Your investigation goal determines which metric you start with and which tabs and views are most useful.

Error investigation uses the Errors metric to find failed requests and their root causes. Performance analysis uses the Duration metric to identify slow operations and latency bottlenecks. Activity monitoring uses the Rate metric to understand service communication patterns and request flows.

Traces Drilldown adapts its interface based on your selected metric. When you choose Errors, you see error-specific tabs like Exceptions and Root cause errors. When you choose Duration, you see latency-focused tabs like Root cause latency and Slow traces. When you choose Rate, you see Service structure to visualize service communication.

Use case 1: Investigate errors

Use this when you know requests are failing or you’ve seen error alerts.

You might have noticed:

Error alerts from your monitoring system
Failed requests in your application logs
User reports of errors or failed operations
Spikes in error rates on dashboards

How to start

Select Errors as your metric type.
Start with Root spans to see service-level error patterns.
Use the Comparison tab to identify which attributes correlate with errors.
Use the Breakdown tab to see which services or operations have the most errors.
Use the Exceptions tab to find common error messages.
Use Root cause errors to see the error chain structure.

When to switch to All spans: If you need to find errors deeper in the call chain, like database errors or downstream service failures that don’t appear at the root level, switch to All spans.

Example scenarios

You know a service is failing but not why:

Select Errors metric and Root spans.
Filter by the service name.
Use Comparison to see which attributes differ between successful and failed requests.
Use Root cause errors to see the error chain structure.

You see error alerts but don’t know the source:

Select Errors metric and Root spans.
Use Breakdown to see which services have the most errors.
Drill into the problematic service using filters.
Use Comparison to identify what’s different about the failing requests.

You need to find internal errors:

Start with Errors metric and Root spans to see service-level patterns.
If errors don’t appear at the root level, switch to All spans.
This reveals database errors, downstream service failures, or internal operation errors.
Use Exceptions to find common error messages.

Use case 2: Analyze performance

Use this when you want to identify slow operations, latency bottlenecks, or optimize response times.

You might be investigating:

Slow response times reported by users
High latency alerts
Performance degradation over time
Need to optimize specific operations

How to start

Select Duration as your metric type.
Start with Root spans for end-to-end request latency.
Use the duration heatmap to identify latency patterns.
Select percentiles (p90, p95, p99) based on your SLA requirements.
Use Root cause latency to see which operations are slowest.
Use Slow traces to examine individual slow requests.
Use Breakdown to see duration by different attributes like service, environment, or region.

When to switch to All spans: If you need to find slow internal operations like database queries or background jobs that don’t appear at the root level, switch to All spans.

Example scenarios

Users report slow responses:

Select Duration metric and Root spans.
Look at the heatmap for latency spikes.
Use Root cause latency to see which service operations are causing delays.
Use Slow traces to examine individual slow requests.

You want to optimize a specific endpoint:

Select Duration metric and Root spans.
Add filters for the endpoint.
Use Breakdown to see duration by different attributes like service, environment, or region.
Select appropriate percentiles (p90, p95, p99) based on your optimization goals.

You need to find slow database queries:

Select Duration metric and All spans (database queries appear as child spans).
Filter by database-related attributes.
Use Breakdown to see which queries are slowest.
Examine the slowest spans in Slow traces to identify problematic queries.

Use case 3: Monitor activity

Use this when you want to understand service communication patterns, request flows, or overall system activity.

You might want to:

Understand how services communicate
Monitor request rates and patterns
Identify unusual activity spikes
Map service dependencies

How to start

Select Rate as your metric type.
Start with Root spans for service-level request rates.
Use Service structure to visualize service-to-service communication.
Use Breakdown to see request rates by different attributes.
Use Comparison to identify unusual patterns compared to baseline.
Use Traces tab to examine individual requests.

When to switch to All spans: If you need to see internal operations or child spans within traces, switch to All spans. Most activity monitoring use cases work well with Root spans.

Example scenarios

You want to understand service dependencies:

Select Rate metric and Root spans.
Use Service structure to see how services call each other.
Identify the communication patterns and dependencies.
Use Traces to examine individual request flows.

You notice unusual activity spikes:

Select Rate metric and Root spans.
Use Breakdown to see which services or operations have increased rates.
Use Comparison to compare against normal baseline behavior.
Switch to Errors or Duration if the spike indicates problems.

You’re doing capacity planning:

Select Rate metric and Root spans.
Use Breakdown by service, environment, or region.
Understand request distribution patterns.
Use Service structure to see communication volumes between services.

Choose your starting point

Your starting point depends on what you already know:

You know what’s wrong:

Errors present → Start with Errors metric and Root spans.
Performance issues → Start with Duration metric and Root spans.
Specific service affected → Add a filter for that service first, then select the appropriate metric.

You need to explore:

Start with Rate metric and Root spans to get an overview.
Look for unusual patterns in the graphs.
Switch to Errors or Duration based on what you find.

You’re doing proactive analysis:

Start with Rate metric and Root spans to understand normal patterns.
Use Comparison to identify deviations from baseline.
Switch to Errors or Duration when you find areas of concern.

RED metrics - Understanding Rate, Errors, and Duration metrics
Traces and spans - How traces and spans work in distributed systems

After you’ve determined your use case:

Choose a RED metric to match your investigation goal.
Choose root or full span data based on the depth you need.
Analyze tracing data using the appropriate tabs for your metric type.
Add filters to refine your investigation as you discover patterns.

Investigate trends and spikes

Fri, 03 Apr 2026 19:43:06 +0000

Investigate trends and spikes

Grafana Traces Drilldown provides powerful tools that help you identify and analyze problems in your applications and services.

Using these steps, you can use the tracing data to investigate issues.

Select Root spans or All spans to look at either the first span in a trace (the root span) or all span data.
Choose the metric you want to use: rates, errors, or duration.
Analyze data using Breakdown, Comparison, Service structure (Rate), Root cause errors and Exceptions (Errors), Root cause latency (Duration), and Traces tabs.
Add filters to refine the view of your data.
Save and load queries to preserve and reuse filter configurations.

You can use these steps in any order and move between them as many times as needed. Depending on what you find, you may start with root spans, delve into error data, and then select All spans to access all of the tracing data.

Give it a try using Grafana Play

With Grafana Play, you can explore and see how it works, learning from practical examples to accelerate your development. This feature can be seen on the Grafana Play site.

Try it

Traces Drilldown UI reference

Fri, 03 Apr 2026 19:43:06 +0000

Traces Drilldown UI reference

Grafana Traces Drilldown helps you focus your tracing data exploration. Some sections change based on the metric you choose. For details on workflows, refer to Analyze tracing data.

Data source selection: At the top left, you select the data source for your traces. In this example, the data source is set to grafanacloud-traces.
Filters: The filter bar helps you refine the data displayed. You can select the type of trace data, either Root spans or All spans. You can also add specific label values to narrow the scope of your investigation.
Select metric type: Choose between Rate, Errors, or Duration metrics. In this example, the Rate metric is selected, showing the number of spans per second.
- The Rate graph (top left) shows the rate of spans over time.
- The Errors graph (top right) displays the error rate over time, with red bars indicating errors.
- The Duration heatmap (bottom right) visualizes the distribution of span durations and can help identify latency patterns.
Investigation-focused tabs: Each metric type has its own set of tabs that help you explore your tracing data. These tabs differ depending on the metric type you’ve selected. For example, when you use Rate, the investigation tabs show Breakdown, Service structure, Comparison, and Traces.
- Exceptions (Errors only): Group exception messages with counts, trend sparkline, emitting service, and last-seen.
- Percentiles (Duration only): Choose p50, p75, p90, p95, p99 for Duration views. Default: p90. If you clear all, p90 applies automatically.
Add to filters: Each attribute group includes an Add to filters option, so you can add your selections into the current investigation.
Time range selector: At the top right, you can adjust the time range for displayed data using the time picker. In this example, the time range is set to the last 24 hours. Refer to Set dashboard time range for more information.

You can also open a specific trace by ID by entering the trace ID into the Trace ID input and pressing Enter. Refer to Open a trace by ID for more information.

Use the Save (save icon) and Load (folder-open icon) buttons in the header to save your current filters as a named query or load a previously saved one. The Save button appears when at least one filter is applied. Refer to Save and load queries for more information.
Attributes sidebar: Use the Attributes sidebar to select and manage attributes across views. Search attributes with regular expressions. Press Escape or click Clear to reset the search.

Click the star icon to add or remove a favorite. Drag and drop favorites to reorder them. Switch between scopes: Favorites, All, Resource, Span. A filter icon marks attributes already applied in the Filters bar.

In Breakdown and Comparison views, selecting an attribute sets the current Group by attribute. In Trace list view, select multiple attributes to add or remove table columns. The app saves favorites in your browser.

Query result streaming

When you first open Traces Drilldown, you may notice a green dot on the upper right corner of any of the metrics graphs.

This green dot indicates that Traces Drilldown is displaying data that’s still being received, or streamed. Streaming lets you view partial query results before the entire query completes.

Open in Explore app

You can open a trace in the Explore app by clicking the Open in Explore button. This opens the trace in the Explore app, where you can use the full power of Explore to analyze the trace.

If you’re using Explore, you can open a trace in Traces Drilldown by clicking the Open in Traces Drilldown button.

Traces Drilldown on Grafana Labs

Access or install Traces Drilldown

Access or install Traces Drilldown

Set up in Grafana Cloud

Set up in self-managed Grafana

Install the Traces Drilldown plugin

Install in your Grafana instance

Install in a Docker container

Access Traces Drilldown

Next steps

Concepts

Concepts

Rate, error, and duration metrics

Traces and spans

Get started with Traces Drilldown

Get started with Traces Drilldown

Before you begin

Explore your tracing data

Example: Investigate source of errors

Choose the level of data and a metric

Handle errors

Correlate attributes

Inspect the problem

Use Root cause errors

What you learned

Determine your use case

Determine your use case

Why this concept matters

How it works

Use case 1: Investigate errors

How to start

Example scenarios

Use case 2: Analyze performance

How to start

Example scenarios

Use case 3: Monitor activity

How to start

Example scenarios

Choose your starting point

Related concepts

Related tasks

Investigate trends and spikes

Investigate trends and spikes

Traces Drilldown UI reference

Traces Drilldown UI reference

Query result streaming

Open in Explore app