Concepts on Grafana Labs

Traces and telemetry

Fri, 03 Apr 2026 19:43:06 +0000

Traces and telemetry

Metrics, logs, traces, and profiles form the pillars of observability. Correlating between the four pillars of observability helps create a holistic view of your application and infrastructure.

Metrics

Metrics provide a high level picture of the state of a system. Metrics are the foundation of alerts because metrics are numeric values and can be compared against known thresholds. Alerts constantly run in the background and trigger when a value is outside of an expected range. This is typically the first sign that something is going on and are where discovery first starts. Metrics indicate that something is happening.

Logs

Logs provide an audit trail of activity from a single process that create informational context. Logs act as atomic events, detailing what’s occurring in the services in your application. Whereas metrics are quantitative (numeric) and structured, logs are qualitative (textual) and unstructured or semi-structured. They offer a higher degree of detail, but also at the expense of creating significantly higher data volumes. Logs let you know what’s happening to your application.

Traces

Traces add further to the observability picture by telling you what happens at each step or action in a data pathway. Traces provide the map–the where–something is going wrong. A trace provides a graphic representation of how long each step in the data flow pathway takes to complete. For example, how long a HTTP request, a database lookup, or a call to a third party service takes. It can show where requests initiate and finish, as well as how your system responds. This data helps you locate problem areas and assess their impact, often in places you never would have anticipated or found without this ability to trace the request flow.

Profiles

Profiles help you understand how your applications utilize compute resources such as CPU time and memory. This helps identify specific lines of code or functions to optimize and improve performance and efficiency.

Why traces?

Metrics in themselves aren’t sufficient to find the root cause and solve complex issues. The same can be said for logs, which can contain a significant amount of information but lack the context of the interactions and dependencies between the different components of your complex environment. Each pillar of observability—metrics, logs, traces, profiles—has its own unique strength when it comes to root causing issues. To get the most value of your observability strategy, you need to be able to correlate them.

Traces have the unique ability to show relationships between services. They help identify which services are upstream from your service, which is helpful when you want to understand which services might be negatively impacted by problems in your service. Traces also help identify which services are downstream from your service. This is valuable since your application relies on their downstream services, and problems with those services may be the cause of elevated errors or latency reported by your service. For example, you can directly see the failing database and all impacted failing edge endpoints.

Using traces and exemplars, you can go from a metric data point and get to an associated trace.

Or from traces to logs:

And vice versa, from logs to traces.

Trace structure

Fri, 03 Apr 2026 19:43:06 +0000

Trace structure

Traces are telemetry data structured as trees. Traces are made of spans (for example, a span tree); there is a root span that can have zero to multiple branches that are called child spans. Each child span can itself be a parent span of one or multiple child spans, and so on so forth.

In the specific context of Tempo and TraceQL query language, a span has the following associated fields:

name: the span name
duration: difference between the end time and start time of the span
status: enum: {ok, error, unset}. For details, refer to OTel span status documentation.
kind: enum: {server, client, producer, consumer, internal, unspecified}. For more details, refer to OTel span kind documentation.
Attributes

The first four properties are intrinsics. Intrinsics are the core, built-in fields that are fundamental to the identity and lifecycle of spans and traces. These fields are defined by the OpenTelemetry specification and are always present. They describe the essential structure of distributed tracing data.

Attributes are custom span metadata in the form of key-value pairs. There are four types of attributes: span attributes, resource attributes, event attributes, and link attributes.

Span attributes are key-value pairs that contain metadata that you can use to annotate a span to carry information about the operation it’s tracking. For example, in an eCommerce application, if a span tracks an operation that adds an item to a user’s shopping cart, the user ID, added item ID and cart ID can be captured and attached to the span as span attributes.

A resource attribute represents information about an entity producing the span. For example, a span created by a process running in a container deployed by Kubernetes could link a resource that specifies the cluster name, namespace, pod, and container name. Resource attributes are resource-related metadata (key-value pairs) that are describing the Resource.

An event attribute represents a unique point in the time during the span’s duration. For more information, refer to All about span events and Span events in the OTEL documentation.

A link attribute lets you query link data in a span. A span link associates one span with one or more other spans that are a causal relationship. For more information on span links, refer to the Span Links documentation in the Open Telemetry project.

The OpenTelemetry specification defines Semantic Attributes for Spans and for Resources. Semantic Span Attributes are a set of naming schemes for attributes shared across languages, frameworks, and runtimes. For more details, refer to Trace Semantic Conventions and Resource Semantic Conventions.

Visualize tracing data in Grafana

Fri, 03 Apr 2026 19:43:06 +0000

Visualize tracing data in Grafana

Grafana has a built-in Tempo data source that can be used to query Tempo and visualize traces. This page describes the high-level features and their availability. Use the latest versions for best compatibility and stability.

Query your data

Using tracing data in Grafana and Grafana Cloud Traces, you can search for traces, generate metrics from spans, and link your tracing data with logs, metrics, and profiles.

Note
Trace results for matching spans are returned on a first-match basis. These results may not be the latest traces stored by Tempo.

Use Traces Drilldown to investigate tracing data

Grafana Traces Drilldown helps you visualize insights from your Tempo traces data. Using the app, you can:

Use Rate, Errors, and Duration (RED) metrics derived from traces to investigate issues
Uncover related issues and monitor changes over time
Browse automatic visualizations of your data based on its characteristics
Do all of this without writing TraceQL queries

Expand your observability journey and learn about the Drilldown apps suite.

Search for traces

Search for traces using common dimensions such as time range, duration, span tags, service names, etc. Use the Explore trace view to quickly diagnose errors and high latency events in your system.

Search is non-deterministic

Most search functions are deterministic. When given the same criteria, a deterministic algorithm returns consistent results. For example, let’s say that you query a search engine for the definition of “traces.” The results list the same top matches for each query for “traces” in that search engine.

However, Tempo search is non-deterministic. If you perform the same search twice, you’ll get different lists, assuming the possible number of results for your search is greater than the number of results you have your search set to return.

When performing a search, Tempo does a massively parallel search over the given time range, and takes the first N results. Even identical searches differ due to things like machine load and network latency. This approach values speed over predictability and is quite simple; enforcing that the search results are consistent would introduce additional complexity (and increase the time the user spends waiting for results). TraceQL follows the same behavior.

By adding most_recent=true to your TraceQL queries, the search results become deterministic. For more information, refer to Retrieve most recent results.

Use trace search results as panels in dashboards

You can embed tracing panels and visualizations in dashboards. You can also save queries as panels. For more information, refer to the Traces Visualization documentation.

For example dashboards, visit play.grafana.org.

Use TraceQL to query data and generate metrics

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces.

Using Grafana Explore, you can search traces. The default traces search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results return faster because the queries limit what is searched.

You can construct queries using the TraceQL query editor or use the Search query type.

For details about constructing queries, refer to the TraceQL documentation.

TraceQL metrics queries

Note
TraceQL metrics is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the feature toggle in Grafana to use this feature. Do not enable this feature in production environments.

TraceQL language provides metrics queries as an experimental feature. Metric queries extend trace queries by applying an aggregate function to trace query results. For example: { span:name = "foo" } | rate() by (span:status) This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs.

Grafana Traces Drilldown is powered by metrics queries.

For more information about available queries, refer to TraceQL metrics queries.

Generate metrics from spans

RED metrics can drive service graphs and other ready-to-go visualizations of your span data. RED metrics represent:

Rate, the number of requests per second
Errors, the number of those requests that are failing
Duration, the amount of time those requests take

For more information about RED method, refer to The RED Method: how to instrument your services.

To enable metrics-generator in Grafana Cloud, refer to Enable metrics-generator.

To enable metrics-generator for Tempo, refer to Configure metrics-generator.

These metrics exist in your Hosted Metrics instance and can also be easily used to generate powerful custom dashboards.

Metrics automatically generate exemplars as well which allows easy metrics to trace linking.

Service graph view

Service graph view displays a table of request rate, error rate, and duration metrics (RED) calculated from your incoming spans. It also includes a node graph view built from your spans.

To use the service graph view, you need to enable service graphs and span metrics. After it’s enabled, this pre-configured view is immediately available in Explore > Service Graphs.

Refer to the service graph view documentation for further information.

Integrate other telemetry signals

Link traces and logs

If you’re already doing request/response logging with trace IDs, they can be easily extracted from logs to jump directly to your traces.

In the other direction, you can configure Grafana and Grafana Cloud to create a link from an individual span to your Loki logs. If you see a long-running span or a span with errors, you can immediately jump to the logs of the process causing the error.

Note
Cloud Traces only supports custom tags added by Grafana Support. Cloud Traces supports these default tags: cluster, hostname, namespace, and pod. Contact Support to add a custom tag.

Link traces and metrics

Grafana can correlate different signals by adding the functionality to link between traces and metrics. Trace to metrics lets you navigate from a trace span to a selected data source. Using trace to metrics, you can quickly see trends or aggregated data related to each span.

For example, you can use span attributes to metric labels by using the $__tags keyword to convert span attributes to metrics labels.

To set up Trace to metrics for your data source, refer to Trace to metric configuration.

Link traces and profiles

Using Trace to profiles, you can correlate different signals by adding the functionality to link between traces and profiles.

Trace to profiles lets you link your Grafana Pyroscope data source to tracing data in Grafana or Grafana Cloud. When configured, this connection lets you run queries from a trace span into the profile data.

For more information, refer to the Traces to profiles documentation and the Grafana Pyroscope data source documentation.

For Cloud Traces, Refer to the Traces to profiles documentation for configuration instructions.

Glossary

Fri, 03 Apr 2026 19:43:06 +0000

Glossary

The following terms are often used when discussing traces.

Active series: A time series that receives new data points or samples.
Cardinality: The total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate.
Data source: A basic storage for data such as a database, a flat file, or even live references or measurements from a device. A file, database, or service that provides data. For example, trace data is imported into Grafana by configuring and enabling a Tempo data source.
Exemplar: Any data that serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary trace IDs, which are typically used to reference a trace.
Log: Chronological events, usually text-based, allowing for the diagnosis of problems. Logs can provide informational context, such as detailed records of all events during user interactions, for example, when events happen, who used the system, status messages, etc.
Metric: A number that helps an operator understand the state of a system, such as the number of active users, error count, average response time, and more.
Span: A unit of work done within a trace.
Trace: A trace represents the whole journey of a request or an action as it moves through all the nodes of a distributed system, especially containerized applications or microservices architectures. Traces are composed of spans. For more information, refer to What is Distributed Tracing?.