Metrics-generator on Grafana Labs

Active series

Fri, 03 Apr 2026 19:43:06 +0000

Active series

An active series is a time series that receives new data points or samples. When you stop writing new data points to a time series, shortly afterwards it is no longer considered active.

Metrics generated by Tempo’s metrics generator can provide both RED (Rate/Error/Duration) metrics and interdependency graphs between services in a trace (the Service Graph functionality in Grafana). These capabilities rely on a set of generated span metrics and service metrics.

Any spans that are ingested by Tempo can create many metric series. However, this doesn’t mean that every time a span is ingested that a new active series is created.

The number of active series generated depends on the label pairs generated from span data that are associated with the metrics, similar to other Prometheus-formated data.

For additional information, refer to the Active series and DPM documentation.

Active series calculation

Active series for a metric increase when a new value for a label key is introduced. For example, the span_kind label has a total of five possible values, and the status_code label has a total of three possible values.

At first glance, you might make an assumption that this means that at least 15 (5*3) active series will be generated for each span. But this isn’t the case.

Let’s consider a span that’s emitted from some piece of code in a service:

Here’s a single service with a single span. If the code inside the span never leaves the service, then the span_kind label generated by the metrics generator will be SPAN_KIND_INTERNAL and never deviate. It’ll never be one of the other four possible values.

Similarly, if the code inside the span never errors, it’ll only have the STATUS_CODE_OK state for the span_status label. This means that the metrics generator will only generate a single active series, where the service name will be Service 1 and the span name will be span1. If we looked at the Prometheus data for the traces_spanmetrics_call_total metric, we’d see a single active series:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	1

It doesn’t matter how many times that span occurs in a trace either, for example maybe a span is generated within a loop. In code run once, 10 times, 100 times, 1000 times, only a single active series will be produced, where a counter might be increased 1, 10, 100, or 1000 times:

If you looked at the Prometheus data, you’d see an instant value for traces_spanmetrics_call_total similar to the table. Again, one active series for the metric:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	120

However, let’s now assume that it does loop and there are occasionally errors.

There are now two potential outcomes for a span when the code loops: one where everything successfully completes and one where there is an error. This means that when the span completes status_code is now either STATUS_CODE_OK or STATUS_CODE_ERROR. Because of that, the label values can be one of two values on a metric, and we now have two active series being generated based on the status_code, one for the OK status and one for the error.

Again, we could loop once, 10 times, 100, or more times, but there will only ever be two active series.

If we now looked at Prometheus instant values for traces_spanmetrics_call_total, we’d now see the following table:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	96
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	24

What happens if you call out to another service though? Let’s add an option where, based on some arbitrary data, we sometimes make a downstream call to another service, but otherwise continue to runs loops in our own service:

In this scenario, span1’s span_kind label would now be one of either SPAN_KIND_INTERNAL or SPAN_KIND_CLIENT (as it has acted as a client calling a downstream server). If a call to the downstream service could also potentially fail, then for SPAN_KIND_CLIENT, the status_code could be either STATUS_CODE_ERROR or STATUS_CODE_OK.

At this point, traces_spanmetrics_call_total would have four different variations in labels:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	34
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	6
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_OK	23
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_ERROR	3

Because of the variation in values, we now have four active series for our metric instead of one. But, as far as Service 1 is concerned, there’s still only four active series, because there isn’t any other variation of the values for labels. You can run 1 trace, 10 traces, 100 traces (each with however many loops of spans there are) and only four active series will ever be produced.

We’ve actually only told half the story in our last diagram. Service 1 called a second service, Service 2, which continues the trace by adding a new span, span2. If there was a loop inside Service 2 with a single span that was generated from an upstream call from Service 1, and then a number of spans that were driven internally, which could also error, we’d end up with the possible values in the metric for traces_spanmetrics_call_total below:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	89
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	13
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_OK	44
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_ERROR	9
Service 2	span2	SPAN_KIND_SERVER	STATUS_CODE_OK	30
Service 2	span2	SPAN_KIND_SERVER	STATUS_CODE_ERROR	14
Service 2	span2	SPAN_KIND_INTERNAL	STATUS_CODE_OK	99
Service 2	span2	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	23

At this point, all our traces will be composed of two potential span names, each of which produce two separate types of span_kind and two separate types of status_code. So we have eight active series for a metric.

The variability of values for each potential span condition determines the number of active series being produced by Tempo when ingesting spans for a trace, and not the number of traces of spans that are seen.

Custom span attributes

There’s another consideration for active series: extra label key/value pairs that can be added onto metrics from a span’s attributes. The Tempo metrics generator allows the user to use arbitrary span attributes to be created as label pairs for metrics. When considering the number of active series generated, you also need to determine how many possible values there are for the span attribute being turned into a label.

For example, if you added an http.method span attribute into a metric label pair, there are five possible values (because there are five possible REST methods):

HEAD
GET
POST
PUT
DELETE

If this label pair is added to every span metric, that’s another 5 potential active series generated for each metric (in all likelihood this is a very worst case scenario, very few spans will call all five REST methods). Instead of 8 active series in the last table above, we’d have 40 (8 * 5).

Cardinality

Fri, 03 Apr 2026 19:43:06 +0000

Cardinality

Cardinality refers to the total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate. For more information on cardinality, see the What are cardinality spikes and why do they matter? blog post.

Because writes to a time-series database (TSDB) database are in series, high cardinality does not make a big difference to performance at ingest. However, cardinality can have a major impact on querying where, the higher the cardinality, the more items are required to be iterated over.

Traces collection and metrics

Tempo’s server-side metrics generation adds functionality to the collection of traces by creating Prometheus-based metrics that track a variety of metrics such as:

Total span call counts
Span latency histograms
Total span size count

The metrics-generator creates metrics which define the relationship between services via edges and nodes. Each of these metrics are queryable using a set of Prometheus labels (key/value pairs).

Each new value for a label increases the number of active series associated with a metric. (To learn more about active series, read the Trace active series documentation.)

This is also known as an increase in cardinality, and the number of active series generated for a metric is directly proportional to the number of labels that exist for that metrics alongside the number of values each label has added.

In a non-modified instance of the metrics generator, a small number of labels are added automatically. Because labels like span_kind and status_code only have a few valid values, the largest variable for the number of active series produced for each metric depends on the number of service names and span names associated with trace spans.

The metrics-generator can also be configured to also add extra labels on metrics, using span attribute key/value pairs which are mapped directly to these labels see the custom span attribute documentation.

Be careful when configuring custom attributes: the greater the number of values seen in a specific attribute, the greater the number of active series will be produced. For more information about active series, refer to the active series documentation

Let’s say that you are adding a custom attribute that includes unique customer IDs as a metrics label. If you have 100 customers, this could potentially multiple the number of active series generated by up to 100 (for example, going from 25,000 active series to 2.5M). Always consider which attributes will actually be useful as labels for querying metrics, as well as the cardinality that they will increase metrics by.

Dry-running the metrics-generator

An often most reliable solution is by running the metrics-generator in a dry-run mode. Using the dry-run mode generates metrics but does not collecting them, thus not writing them to a metrics storage. The override metrics_generator_disable_collection is defined for this use-case.

To get an estimate, run the metrics-generator normally and set the override to true. Then, check tempo_metrics_generator_registry_active_series to get an estimation of the active series for that set-up.

Span metrics

Fri, 03 Apr 2026 19:43:06 +0000

Span metrics

The span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics.

Span metrics generate two metrics:

A counter that computes requests
A histogram that tracks the distribution of durations of all requests

Span metrics are of particular interest if your system is not monitored with metrics, but it has distributed tracing implemented. You get out-of-the-box metrics from your tracing pipeline.

Even if you already have metrics, span metrics can provide in-depth monitoring of your system. The generated metrics will show application level insight into your monitoring, as far as tracing gets propagated through your applications.

Last but not least, span metrics lower the entry barrier for using exemplars. An exemplar is a specific trace representative of measurement taken in a given time interval. Since traces and metrics co-exist in the metrics-generator, exemplars can be automatically added, providing additional value to these metrics.

How to run

To enable service graphs in Tempo/GET, enable the metrics generator and add an overrides section which enables the span-metrics generator. See here for configuration details.

How it works

The span metrics processor works by inspecting every received span and computing the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any attribute present in the span.

This processor is designed with the goal to mirror the implementation from the OpenTelemetry Collector of the processor with the same name.

Note
To learn more about cardinality and how to perform a dry run of the metrics generator, see the Cardinality documentation.

Metrics

The following metrics are exported:

Metric	Type	Labels	Description
traces_spanmetrics_latency	Histogram	Dimensions	Duration of the span
traces_spanmetrics_calls_total	Counter	Dimensions	Total count of the span
traces_spanmetrics_size_total	Counter	Dimensions	Total size of spans ingested

Note
In Tempo 1.4 and 1.4.1, the histogram metric was called traces_spanmetrics_duration_seconds. This was changed later to be consistent with the metrics generated by the Grafana Agent and the OpenTelemetry Collector.

By default, the metrics processor adds the following labels to each metric: service, span_name, span_kind, status_code, status_message, job, and instance.

service - The name of the service that generated the span
span_name - The unique name of the span
span_kind - The type of span, this can be one of five values:
- SPAN_KIND_SERVER - The span was generated by a call from another service
- SPAN_KIND_CLIENT - The span made a call to another service
- SPAN_KIND_INTERNAL - The span does not have interaction outside of the service it was generated in
- SPAN_KIND_PUBLISHER - The span created data that was pushed onto a bus or message broker
- SPAN_KIND_CONSUMER - The span consumed data that was on a bus or messaging system
status_code - The result of the span, this can be one of three values:
- STATUS_CODE_UNSET - Result of the span was unset/unknown
- STATUS_CODE_OK - The span operation completed successfully
- STATUS_CODE_ERROR - The span operation completed with an error
status_message (optionally enabled) - The message that details the reason for the status_code label
job - The name of the job, a combination of namespace and service; only added if metrics_generator_processor_span_metrics_enable_target_info: true
instance - The instance ID; only added if metrics_generator_processor_span_metrics_enable_target_info: true

Additional user defined labels can be created using the dimensions configuration option. When a configured dimension collides with one of the default labels (e.g. status_code), the label for the respective dimension is prefixed with double underscore (i.e. __status_code).

Custom labeling of dimensions is also supported using the dimension_mapping configuration option.

An optional metric called traces_target_info using all resource level attributes as dimensions can be enabled in the enable_target_info configuration option.

If you use a ratio-based sampler, you can use the custom sampler below to not lose metric information. However, you also need to set metrics_generator.processor.span_metrics.span_multiplier_key to "X-SampleRatio".

package tracer
import (
	"go.opentelemetry.io/otel/attribute"
	tracesdk "go.opentelemetry.io/otel/sdk/trace"
)

type RatioBasedSampler struct {
	innerSampler        tracesdk.Sampler
	sampleRateAttribute attribute.KeyValue
}

func NewRatioBasedSampler(fraction float64) RatioBasedSampler {
	innerSampler := tracesdk.TraceIDRatioBased(fraction)
	return RatioBasedSampler{
		innerSampler:        innerSampler,
		sampleRateAttribute: attribute.Float64("X-SampleRatio", fraction),
	}
}

func (ds RatioBasedSampler) ShouldSample(parameters tracesdk.SamplingParameters) tracesdk.SamplingResult {
	sampler := ds.innerSampler
	result := sampler.ShouldSample(parameters)
	if result.Decision == tracesdk.RecordAndSample {
		result.Attributes = append(result.Attributes, ds.sampleRateAttribute)
	}
	return result
}

func (ds RatioBasedSampler) Description() string {
	return "Ratio Based Sampler which gives information about sampling ratio"
}

Filtering

In some cases, you may want to reduce the number of metrics produced by the spanmetrics processor. You can configure the processor to use an include filter to match criteria that must be present in the span in order to be included. Following the include filter, you can use an exclude filter to reject portions of what was previously included by the filter policy.

Currently, only filtering by resource and span attributes with the following value types is supported.

bool
double
int
string

Additionally, these intrinsic span attributes may be filtered upon:

name
status (code)
kind

The following intrinsic kinds are available for filtering.

SPAN_KIND_SERVER
SPAN_KIND_INTERNAL
SPAN_KIND_CLIENT
SPAN_KIND_PRODUCER
SPAN_KIND_CONSUMER

Intrinsic keys can be acted on directly when implementing a filter policy. For example:

---
metrics_generator:
  processor:
    span_metrics:
      filter_policies:
        - include:
            match_type: strict
            attributes:
              - key: kind
                value: SPAN_KIND_SERVER

In this example, spans which are of kind “server” are included for metrics export.

When selecting spans based on non-intrinsic attributes, it is required to specify the scope of the attribute, similar to how it is specified in TraceQL. For example, if the resource contains a location attribute which is to be used in a filter policy, then the reference needs to be specified as resource.location. This requires users to know and specify which scope an attribute is to be found and avoids the ambiguity of conflicting values at differing scopes. The following may help illustrate.

---
metrics_generator:
  processor:
    span_metrics:
      filter_policies:
        - include:
            match_type: strict
            attributes:
              - key: resource.location
                value: earth

In the above examples, we are using match_type of strict, which is a direct comparison of values. You can use regex, an additional option for match_type, to build a regular expression to match against.

---
metrics_generator:
  processor:
    span_metrics:
      filter_policies:
        - include:
            match_type: regex
            attributes:
              - key: resource.location
                value: eu-.*
        - exclude:
            match_type: regex
            attributes:
              - key: resource.tier
                value: dev-.*

In the above, we first include all spans which have a resource.location that begins with eu- with the include statement, and then exclude those with begin with dev-. In this way, a flexible approach to filtering can be achieved to ensure that only metrics which are important are generated.

Example

Service graphs

Fri, 03 Apr 2026 19:43:06 +0000

Service graphs

A service graph is a visual representation of the interrelationships between various services. Service graphs help you to understand the structure of a distributed system, and the connections and dependencies between its components:

Infer the topology of a distributed system. As distributed systems grow, they become more complex. Service graphs help you to understand the structure of the system.
Provide a high-level overview of the health of your system. Service graphs display error rates, latencies, as well as other relevant data.
Provide an historic view of a system’s topology. Distributed systems change very frequently, and service graphs offer a way of seeing how these systems have evolved over time.

How they work

The metrics-generator processes traces and generates service graphs in the form of Prometheus metrics.

Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request. The processor uses the OpenTelemetry semantic conventions to detect a myriad of requests. It currently supports the following requests:

A direct request between two services where the outgoing and the incoming span must have span.kind, client, and server, respectively.
A request across a messaging system where the outgoing and the incoming span must have span.kind, producer, and consumer respectively.
A database request; in this case the processor looks for spans containing attributes span.kind=client as well as db.name.

Every span that can be paired up to form a request is kept in an in-memory store, until its corresponding pair span is received or the maximum waiting time has passed. When either of these conditions are reached, the request is recorded and removed from the local store.

Each emitted metrics series have the client and server label corresponding with the service doing the request and the service receiving the request.

  tempo_service_graph_request_total{client="app", server="db", connection_type="database"} 20

Virtual nodes

Virtual nodes are nodes that form part of the lifecycle of a trace, but spans for them are not being collected because they’re outside the user’s reach (for example, an external service for payment processing) or are not instrumented (for example, a frontend application).

Virtual nodes can be detected in two different ways:

The root span has span.kind set to server. This indicates that the request has initiated by an external system that’s not instrumented, like a frontend application or an engineer via curl.
A client span does not have its matching server span, but has a peer attribute present. In this case, we make the assumption that a call was made to an external service, for which Tempo won’t receive spans.
- The default peer attributes are peer.service, db.name and db.system.
- The order of the attributes is important, as the first one that is present will be used as the virtual node name.

Metrics

The following metrics are exported:

Metric	Type	Labels	Description
traces_service_graph_request_total	Counter	client, server, connection_type	Total count of requests between two nodes
traces_service_graph_request_failed_total	Counter	client, server, connection_type	Total count of failed requests between two nodes
traces_service_graph_request_server_seconds	Histogram	client, server, connection_type	Time for a request between two nodes as seen from the server
traces_service_graph_request_client_seconds	Histogram	client, server, connection_type	Time for a request between two nodes as seen from the client
traces_service_graph_unpaired_spans_total	Counter	client, server, connection_type	Total count of unpaired spans
traces_service_graph_dropped_spans_total	Counter	client, server, connection_type	Total count of dropped spans

Duration is measured both from the client and the server sides.

Possible values for connection_type: unset, messaging_system, or database.

Additional labels can be included using the dimensions configuration option.

Since the service graph processor has to process both sides of an edge, it needs to process all spans of a trace to function properly. If spans of a trace are spread out over multiple instances, spans are not paired up reliably.

Service graph view

Fri, 03 Apr 2026 19:43:06 +0000

Service graph view

Grafana’s service graph view utilizes metrics generated by the metrics-generator (or Grafana Agent) to display span request rates, error rates, and durations, as well as service graphs. Once the requirements are set up, this pre-configured view is immediately available.

Using the service graph view, you can:

Discover spans which are consistently erroring and the rates at which they occur
Get an overview of the overall rate of span calls throughout your services
Determine how long the slowest queries in your service take to complete
Examine all traces that contain spans of particular interest based on rate, error and duration values (RED signals)

Requirements

You have to enable span metrics and service graph generation on the Grafana backend so metrics that are generated as traces are ingested.

To use the service graph view, you need:

Tempo or Grafana Cloud Traces with either 1) the metrics generator enabled and configured or 2) the Grafana Agent enabled and configured to send data to a Prometheus-compatible metrics store
Services graphs, which are enabled by default in Grafana
Span metrics enabled in your Tempo data source configuration

The service graph view can be derived from metrics generated by either Tempo’s metrics-generator or by the Grafana Agent.

For information on how to configure these features, refer to the Grafana Tempo data sources documentation.

What does the service graph view show?

Using this view, you can see the top five spans with a type of server (listed in the Name column). You can refine any of this data using the filters. Selecting any of the data points lets you see more specific data.

The service graph view provides a span metrics visualization (table, screen section 2) and service graph (screen section 3). In addition, you can use the filters (screen section 1) to customize the data displayed.

Any information in the table that has an underline can be selected to show more detailed information. You can also select any node in the service graph to display additional information. In the dashboard shown below, the Ingester.QueryStream span has a request rate of 144220.22 requests per second. The /cortex.Ingester/Query span has the highest request rate.

Error rate example

Let’s say we want to learn more about why cortex.Ingester has the highest error rates. Selecting the second row of the Error rate column displays details about the span metrics in a new window on the right side.

The metrics query used to generate the data appears in the Metrics browser field.

Span metrics table

The span metrics, shown in the table, are generated by the metrics-generator or the Grafana Agent. These metrics are created from ingested tracing data, including RED metrics.

Span metrics generate two metrics:

A counter that computes requests
A histogram that tracks the distribution of durations of all requests

For information about span metrics and how they are calculated, refer to the Span metrics documentation.

Table contents

The span metrics table contains seven columns with five column headings. Selecting a heading sorts the data by ascending or descending values.

Column	Explanation	PromQL query for span
Name	Use the span name. OTel semantic conventions generally expect the span name to be some kind of low cardinality indicator of the http route or database function being performed.	N/A
Rate	LCD gauge (horizontal bar graph). Instances per second of the span. Clicking this field can jump to the appropriate metrics.	`sum(rate( traces_spanmetrics_calls_total{ span_name="", <filters> }[$__range]))`
Error Rate	Number and LCD gauge (horizontal bar graph). Clicking this field shows more detailed metrics.	`sum(rate( traces_spanmetrics_calls_total{ span_name="", span_status="STATUS_CODE_ERROR", <filters> }[$__range]))`
Duration	p90 duration: 90% of all occurrences of this span complete within this time. Clicking this field shows the appropriate metrics.	`histogram_quantile(.9, sum(rate( traces_spanmetrics_duration_seconds_bucket{ span_name="", span_status="STATUS_CODE_ERROR", <filters> }[$__range]) by (le))`
Links	Provide links to example traces given the span name and other applied filters. Link to a search for all spans with the same name from the same Tempo data source.	N/A

Service graphs

A service graph (node graph) is a visual representation of the interrelationships between various services. Service graphs help to understand the structure of a distributed system, and the connections and dependencies between its components.

Service graphs infer the topology of a distributed system, provide a high level overview of the health of your system, and a historic view of a system’s topology. Service graphs show error rates and latencies, among other relevant data. The service graph layout can be the default or grid.

The grid layout changes the service graph to a series of rows and columns.

If you are using the metrics-generator, then it processes traces and generates service graphs in the form of time series metrics like:

tempo_service_graph_request_total{client="app", server="db"} 20

For information about service graphs and how they are calculated, refer to the Service Graphs documentation.

Use filters to reveal details

The service graph view uses service graphs and span metrics to provide a gateway to your tracing information. This dashboard is derived from a fixed set of metrics queries. These underlying queries can not be changed. However, you can choose which traces are included in the metrics query by filtering.

You can explore data by clicking on selectable items or by using filters.

Selecting items or nodes for more detail

Clicking on selectable items, such as underlined text in the table or nodes on the service graph, lets you reveal specific details based upon your selection.

In the table, you can select items in the Rate, Error Rate, Duration (p90), and Links columns. Choosing one of these items provides details about the span metrics.

You can view request rate, request histogram, failed request rate, and traces for any node in the service graph. To view more information, select the node in the service graph and then choose an option from the popup. For details on navigating the service graph, refer to the Node graph panel documentation.

Filter with metric queries

Using the filters at the top of the screen, you can narrow the data set based upon span attributes (key-value pairs or labels). The filters build a query to refine what is shown in the service graph and span metrics. You can add one or more label filters.

To use the filters:

At the top of the Service Graph, select the text box after Filter to display a list of available labels. In this case, server is selected.
Select or search for a value for the label. In this case, the value of server is equal to tempo-ingester. The default operator is equals (=).
Optional: Change the operator by selecting = and choosing a new option from the drop-down.
Optional: Add additional key-value pairs to refine the data set. Any subsequent label filters use AND, which requires both key-value pairs to be presents for matches.
Select Run query.

Filters can be removed by selecting the filter drop-down and choosing – remove filter –.

In the example below, each field or label represents a key-value pair. Number 1 selects a service as the label whose value is Go-http-client (2). The second key-value pair has a client as a label whose value is 02e807.

If your metrics queries are too specific, they may not return any results.

Updating the filter to be less specific returns a result. In this case, the results show only span metrics data associated with the span_name label with a value of /base.Ruler/Rules. No service graph data was available.