Concepts on Grafana Labs

Traces and telemetry

Thu, 31 Jul 2025 14:21:37 +0000

Traces and telemetry

Metrics, logs, traces, and profiles form the pillars of observability. Correlating between the four pillars of observability helps create a holistic view of your application and infrastructure.

Metrics

Metrics provide a high level picture of the state of a system. Metrics are the foundation of alerts because metrics are numeric values and can be compared against known thresholds. Alerts constantly run in the background and trigger when a value is outside of an expected range. This is typically the first sign that something is going on and are where discovery first starts. Metrics indicate that something is happening.

Logs

Logs provide an audit trail of activity from a single process that create informational context. Logs act as atomic events, detailing what’s occurring in the services in your application. Whereas metrics are quantitative (numeric) and structured, logs are qualitative (textual) and unstructured or semi-structured. They offer a higher degree of detail, but also at the expense of creating significantly higher data volumes. Logs let you know what’s happening to your application.

Traces

Traces add further to the observability picture by telling you what happens at each step or action in a data pathway. Traces provide the map–the where–something is going wrong. A trace provides a graphic representation of how long each step in the data flow pathway takes to complete. For example, how long a HTTP request, a database lookup, or a call to a third party service takes. It can show where requests initiate and finish, as well as how your system responds. This data helps you locate problem areas and assess their impact, often in places you never would have anticipated or found without this ability to trace the request flow.

Profiles

Profiles help you understand how your applications utilize compute resources such as CPU time and memory. This helps identify specific lines of code or functions to optimize and improve performance and efficiency.

Why traces?

Metrics in themselves aren’t sufficient to find the root cause and solve complex issues. The same can be said for logs, which can contain a significant amount of information but lack the context of the interactions and dependencies between the different components of your complex environment. Each pillar of observability—metrics, logs, traces, profiles—has its own unique strength when it comes to root causing issues. To get the most value of your observability strategy, you need to be able to correlate them.

Traces have the unique ability to show relationships between services. They help identify which services are upstream from your service, which is helpful when you want to understand which services might be negatively impacted by problems in your service. Traces also help identify which services are downstream from your service. This is valuable since your application relies on their downstream services, and problems with those services may be the cause of elevated errors or latency reported by your service. For example, you can directly see the failing database and all impacted failing edge endpoints.

Using traces and exemplars, you can go from a metric data point and get to an associated trace.

Or from traces to logs:

And vice versa, from logs to traces.

Trace structure

Tue, 08 Jul 2025 20:53:47 +0000

Trace structure

Glossary

Tue, 02 Jun 2026 16:22:08 +0200

Glossary

The following terms are often used when discussing traces.

Active series: A time series that receives new data points or samples.
Cardinality: The total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate.

Child span: A span that is nested within a parent span. Each child span represents a sub-operation or downstream call within the broader operation represented by its parent. A child span can itself be a parent of other child spans.
Data source: A basic storage for data such as a database, a flat file, or even live references or measurements from a device. A file, database, or service that provides data. For example, trace data is imported into Grafana by configuring and enabling a Tempo data source.

Event attribute: A key-value pair attached to a span event, which represents a unique point in time during the span’s duration. For more information, refer to Span events in the OpenTelemetry documentation.
Exemplar: Any data that serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary trace IDs, which are typically used to reference a trace.

Intrinsics: The core, built-in fields that are fundamental to the identity and lifecycle of spans and traces. Intrinsic fields include name, duration, status, and kind. These fields are defined by the OpenTelemetry specification and are always present.

Link attribute: A key-value pair attached to a span link. A span link associates one span with one or more causally related spans. For more information, refer to Span Links in the OpenTelemetry documentation.
Log: Chronological events, usually text-based, allowing for the diagnosis of problems. Logs can provide informational context, such as detailed records of all events during user interactions, for example, when events happen, who used the system, status messages, etc.
Metric: A number that helps an operator understand the state of a system, such as the number of active users, error count, average response time, and more.

Resource attribute: A key-value pair that represents information about the entity producing a span, such as the cluster name, namespace, Pod, or container name.

Root span: The first span in a trace, representing the initial request or operation. A root span has no parent span and serves as the top of the span tree.

Semantic attribute: A standardized naming scheme for attributes shared across languages, frameworks, and runtimes, as defined by the OpenTelemetry specification. For more information, refer to Trace Semantic Conventions in the OpenTelemetry documentation.
Span: A unit of work done within a trace.

Span attribute: A key-value pair that contains metadata to annotate a span with information about the operation it tracks. For example, a span tracking an “add to cart” operation might include the user ID, item ID, and cart ID as span attributes.
Spanset: The group of spans for each individual trace returned from a TraceQL query.
Trace: A trace represents the whole journey of a request or an action as it moves through all the nodes of a distributed system, especially containerized applications or microservices architectures. Traces are composed of spans. For more information, refer to What is Distributed Tracing?.