Release notes on Grafana Labs

Version 2.2 release notes

Fri, 03 Apr 2026 12:35:46 -0500

Version 2.2 release notes

The Tempo team is pleased to announce the release of Tempo 2.2.

This release gives you:

Major additions to the TraceQL language: structural operators (descendant, child, sibling), results grouped by attribute (by()), a select operator, and 3 new intrinsic attributes.
Faster TraceQL results, thanks to a streaming endpoint that returns partial results as the query executes as well as a multitude of performance-related improvements.
An experimental metrics-summary API that returns RED metrics for recently received spans grouped by your attribute of choice.

Tempo 2.2 makes vParquet2, a Parquet version designed to be more compatible with other Parquet implementations, the default block format. This block format is required for using structural operators and improves query performance relative to previous formats.

Read the Tempo 2.2 blog post for more examples and details about these improvements.

Note
For a complete list of changes, enhancements, and bug fixes refer to the Tempo 2.2 changelog.

Features and enhancements

Some of the most important features and enhancements in Tempo 2.2 are highlighted below.

Expanding the TraceQL language

With this release, we’ve added to the TraceQL language. TraceQL now offers:

Structural operators: descendant (»), child (>), and sibling (~) (documentation). Find relevant traces based on their structure and relationships among spans. [PR #2625 #2660]
A select() operation that allows you to specify arbitrary span attributes that you want included in the TraceQL response (documentation) [PR 2494]
A by() operation that groups span sets within a trace by an attribute of your choosing. This operation is not supported in the Grafana UI yet; you can only use by() when querying Tempo’s search API directly. (documentation [PR 2490]
New intrinsic attributes for use in TraceQL queries: traceDuration, rootName, and rootServiceName (documentation) [PR #2503]

Read the Tempo 2.2 blog post for examples of how to use these new language additions.

To learn more about the TraceQL syntax, see the TraceQL documentation. For information on planned future extensions to the TraceQL language, see future work.

Get TraceQL results faster

We’re always trying to reduce the time you spend waiting to get results to your TraceQL queries, and we’ve made some nice progress on this front with this release.

We’ve added a GRPC streaming endpoint to Tempo’s query frontend that allows a client to stream search results from Tempo. The Tempo CLI has been updated to use this new streaming endpoint [PR #2366] . As of version 10.1, Grafana supports it as well, though you must first enable the traceQLStreaming feature toggle [PR #72288].

By streaming results to the client, you can start to look at traces matching your query before the entire query completes. This is particularly helpful for long-running queries; while the total time to complete the query is the same, you can start looking at your first matches before the full set of matched traces is returned.

In addition to streaming partial results, we’ve merged a number of improvements to speed up TraceQL queries. Here are just a few:

Add support for query batching between frontend and queriers to improve throughput [PR 2677]
Improve performance of TraceQL regex [PR 2484]
Fully skip over Parquet row groups with no matches in the column dictionaries [PR 2676]
New synchronous read mode for vParquet and vParquet2 [PRs 2165, 2535]
Improved TraceQL throughput by asynchronously creating jobs. [PR 2530]

Metrics summary API (experimental)

Tempo has added an experimental API that returns RED metrics (span count, erroring span count, and latency information) for spans of kind=server sent in the last hour, grouped by an attribute of your choice. For example, you could use this API to compare error rates of spans with different values of the namespace attribute. From here, you might see that spans from namespace=A have a significantly higher error rate than those from namespace=B. As another example, you could use this API to compare latencies of your spans broken down by the region attribute. From here, you might notice that spans from region=North-America have higher latencies than those from region=Asia-Pacific.

This API is meant to enable ad-hoc analysis of your incoming spans; by segmenting your spans by attribute and looking for differences in RED metrics, you can more quickly isolate where problems like elevated error rates or higher latencies are coming from.

Unlike RED metrics computed by Tempo’s [metrics-generator]((../../metrics-generator/), the values returned by this API are not persisted as time series. This has the advantage that you do not need to provide your own time series databases for storing and querying these metrics. It also allows you to compute RED metrics broken down by high cardinality attributes that would be too expensive to store in a time series database. Use the metrics generator if you want to store and visualize RED metrics over multi-hour or multi-day time ranges, or you want to alert on these metrics.

To learn more about this API, refer to the metrics summary API documentation.

This work is represented in multiple PRs: 2368, 2418, 2424, 2442, 2480, 2481, 2501, 2579, and 2582.

Other enhancements

Tempo’s tag values and tag names APIs now support filtering [PR 2253]. This lets you retrieve all valid attribute values and names given certain criteria. For example, you can get a list of values for the attribute namespace seen on spans with attribute resource=A. This feature is off by default; to enable, configure autocomplete_filtering_enabled. (documentation). Grafana’s autocomplete can make use of this filtering capability to provide better suggestions starting in v10.2 PR [67845].
Tempo’s metrics-generator now supports span filtering. Setting up filters allows you to compute metrics over the specific spans you care about, excluding others. It also can reduce the cardinality of generated metrics, and therefore the cost of storing those metrics in a Prometheus-compatible TSDB. (documentation) [PR 2274]
Tempo’s metrics-generator can now detect virtual nodes (documentation) [PR 2365]. As a result, you’ll now see these virtual nodes represented in your service graph. For more information, refer to the virtual nodes documentation.

Upgrade considerations

When upgrading to Tempo 2.2, be aware of these breaking changes:

JSonnet users only: We’ve converted the metrics-generator component from a k8s deployment to a k8s statefulset. Refer to the PR for seamless migration instructions. [PRs #2533, #2467]
Removed or renamed configuration parameters (see section below)

While not a breaking change, upgrading to Tempo 2.2 will by default change Tempo’s block format to vParquet2. To stay on a previous block format, read the Parquet configuration documentation. We strongly encourage upgrading to vParquet2 as soon as possible as this is required for using structural operators in your TraceQL queries and provides query performance improvements, in particular on queries using the duration intrinsic.

Removed or renamed configuration parameters

The following fields were removed or renamed.

Parameter Comments

```

query_frontend:

tolerate_failed_blocks: <int>

```

Remove support for `tolerant_failed_blocks` [PR 2416]

```

storage:

trace:

s3:

insecure_skip_verify: true // renamed to tls_insecure_skip_verify

```

Renamed `insecure_skip_verify` to `tls_insecure_skip_verify` [PR 2407]

Bug fixes

For a complete list, refer to the Tempo changelog.

2.2.4

Updated Alpine image version to 3.18 to patch CVE-2022-48174 PR 3046
Bumped Jaeger query docker image to 1.50.0 PR 2998

2.2.3

Fixed S3 credentials providers configuration PR 2889

2.2.2

Fixed node role auth IDMSv1 PR 2760

2.2.1

Fixed incorrect metrics for index failures PR 2781
Fixed a panic in the metrics-generator when using multiple tenants with default overrides PR 2786
Restored tenant_header_key removed in PR 2414 PR 2786
Disabled streaming over HTTP by default PR 2803

2.2

Fixed an issue in the metrics-generator that prevented scaling up parallelism when remote writing of metrics was lagging behind [PR 2463]
Fixed an issue where metrics-generator was setting wrong labels for traces_target_info [PR 2546]
Fixed an issue where matches and other spanset level attributes were not persisted to the TraceQL results. [PR 2490]
Fixed an issue where ingester search could occasionally fail with file does not exist error [PR 2534]

Version 2.1 release notes

Fri, 03 Apr 2026 12:35:46 -0500

Version 2.1 release notes

The Tempo team is pleased to announce the release of Tempo 2.1.

With this release, Tempo enhances TraceQL so you can select traces using features such as span kind, aribtrary arithmetic, and new aggregates min/max/sum. In addition, Tempo 2.1 introduces vParquet2, a new experimental Parquet version designed to be more compatible with other Parquet implementations.

Note: For a complete list of changes and enhancements, refer to the Tempo 2.1 changelog.

Features and enhancements

Some of the most important features and enhancements in Tempo 2.1 are highlighted below.

Enhancements to TraceQL

With this release, we’ve expanded the TraceQL language. In Tempo 2.0, TraceQL allowed users to select traces based upon resource attributes, time, duration, and basic aggregates. In Tempo 2.1, users can also select traces based on the following:

Enable querying by kind, an intrinsic attribute (PR 2217)
- { kind = server }, { kind = client }, { kind = consumer }
- { kind = server && duration > 1s }
Support arithmetic expressions (PR 2146)
- { span.bytesProcessed > 10 * 1024 * 1024 }
- { span.bytesProcessed / 1024 / 1024 > 10 }
- { span.jobs / span.bytes > 3 }
Enable min, max, and sum aggregate filters
- {} | min(duration) > 1s
- {} | max(span.bytesProcessed) < 1024 * 1024
- {} | sum(duration) > 1h
Support for more numeric types: Compare floating point numbers and integers, allow floating point numbers for durations, and support negative numbers when doing aggregate filters. (PR 2139, 2304, 2289)
- { 1 = 1.0 }
- { duration > 1.5s }, {duration > 1s500ms }
- {} | count() > -1

This update also includes major performance improvements to TraceQL query performance. [PR 2119]

To learn more about the TraceQL syntax, see the TraceQL documentation. For information on planned future extensions to the TraceQL language, see future work.

TraceQL query editor improvements (experimental)

Starting in Grafana 9.5, users have access to a Tempo Search tab powered by TraceQL. To enable this feature, use Grafana’s traceqlSearch feature flag. The editor can autocomplete queries and has a streamlined interface.

Grafana will soon include a new Search screen that can build a TraceQL query based upon options selected in drop-downs. Contact Support to enable this feature in Grafana Cloud.

For more information, refer to the TraceQL query editor documentation.

vParquet2 (experimental)

Tempo 2.1 offers an experimental update to the Parquet backend: vParquet2. Due to the dedicated span duration column, there is a performance gain for TraceQL queries containing duration. vParquet2 will also be a prerequisite for TraceQL’s structural operators, which will be introduced in a future release. [PR 2244]

You can select the block format Tempo uses by changing the block format version option. To use the new format, set the block format to vParquet2. For more information, read the Parquet configuration documentation.

The new encoding implements Parquet schema changes that are not compatible with vParquet:

Renamed columns to make the schema more consistent and closer to names and terminologies used by OTEL
Add a dedicated span column for duration
Additional columns that will enable new TraceQL features, including hierarchical queries

Other enhancements and features

For a complete list, refer to the Tempo 2.1 changelog.

PR 2105 Log when a trace is too large to compact. Traces are dropped when they exceed a certain limit set in the ingesters. Any dropped traces are now logged with the number of dropped spans and the trace ID.
PR 2195 Add support for Azure Workload Identity authentication
PR 2130 Add command to tempo-cli to migrate a tenant
PR 2093 Add support for s3 session token in static configuration

Upgrade considerations

When upgrading to Tempo 2.1, be aware of these breaking changes.

Breaking changes to metric names exposed by Tempo

All Prometheus metrics exposed by Tempo on its /metrics endpoint that were previously prefixed with cortex_ have now been renamed to be prefixed with tempo_ instead. (PR 2204)

Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR 2008)

The query_frontend_result_metrics_inspected_bytes metric was removed in favor of query_frontend_bytes_processed_per_second.

Remove support for Search on v2 blocks

Users can no longer search blocks in v2 format. Only vParquet and vParquet2 formats support search. The following search configuration options were removed from the overrides section:

overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:

The following metrics configuration was also removed:

tempo_ingester_trace_search_bytes_discarded_total

This change means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks. One way to avoid this is to upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks. You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming traces. Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1. All of your blocks will be searchable.

Finally, parquet files are no longer cached when carrying out searches (PR 2164).

Bug fixes

For a complete list, refer to the Tempo 2.1 changelog.

2.1.1

PR 2400 Fixed issue where Tempo sometimes flips booleans from false->true at storage time.

2.1

PR 2220 Correctly connect context during compaction
PR 2081 Apply rate() to the bytes/s panel in the tenant’s dashboard.
PR 2111 Retry copy operations during compaction in GCS backend.
PR 2164 Improve locking and search head block in SearchTagValuesV2
PR 2152 Fix not closing WAL block file before attempting to delete the folder.
PR 2219 Stop searching for virtual tags if there are any hits. This prevents invalid values from showing up for intrinsics like status.
PR 2254 Correctly return unique spans when &&ing and ||ing spansets.

Version 2.0 release notes

Fri, 03 Apr 2026 12:35:46 -0500

Version 2.0 release notes

The Tempo team is pleased to announce the newest major release of Tempo: version 2.0!

Tempo 2.0’s headline feature is the introduction of TraceQL. TraceQL is a query language modeled on PromQL and LogQL that allows you to interactively extract insights from your trace data. It provides a powerful, flexible way to pinpoint the traces you need to answer questions about your systems and analyze traces based on their structure.

Continuing in the tradition of Tempo’s close integration with Grafana as a visualization layer, we’ve also added a TraceQL query editor to Grafana’s Tempo data source. The editor is available in Grafana Cloud and for open source users in Grafana 9.3.2. We encourage open source users to upgrade to Grafana 9.4 as soon as it is released to take advantage of some additional enhancements and bug fixes.

In addition to TraceQL, Tempo 2.0 also marks the promotion of the Apache Parquet block format to the new default storage format. With this change, the Tempo maintainers are signaling that this new format has received enough testing to be ready for broad adoption.

These two changes actually go hand-in-hand: without the increased search speed enabled by the Apache Parquet block format, it would be impossible to pull off the sophisticated queries made possible by TraceQL.

Note: For a complete list of changes and enhancements, refer to the Tempo 2.0 changelog.

Features and enhancements

Some of the most important features and enhancements in Tempo 2.0 are highlighted below.

TraceQL, the trace query language

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. TraceQL makes it possible to find traces that otherwise would have been difficult or even impossible to identify with Tempo’s existing search. It also makes it possible for you to understand the broad context for an event sequence. To understand more about why we built TraceQL and the use cases we think it unlocks, see our blog post.

With this release, we’ve implemented a subset of the full TraceQL language we envision. Users can select traces based on the following:

Span and resource attributes, timing, and duration
Basic aggregates: count() and avg()

To learn more about the TraceQL syntax, see the TraceQL documentation. For information on how we plan to extend the TraceQL language, see future work.

You can run a TraceQL query either by issuing it to Tempo’s q parameter of the search API endpoint, or, for those using Tempo in conjunction with Grafana, by using Grafana’s TraceQL query editor, described below.

TraceQL requires Tempo’s Parquet columnar format, which is the default block format in Tempo 2.0. For more information, refer to the Tempo Apache Parquet block format documentation.

TraceQL query editor in Grafana

To make it easier to run and visualize the results of TraceQL queries, we’ve added a TraceQL query editor to Grafana’s Tempo data source. The editor is available in Grafana’s Explore interface starting in Grafana version 9.3.2.

The query editor helps you learn TraceQL by offering autocomplete.

For more information, refer to the TraceQL query editor documentation.

Apache Parquet block format is the new default

The experimental Apache Parquet columnar block format introduced in Tempo v1.5 has been promoted to stable and is now the default trace storage for Tempo 2.0. We’re excited to switch to it because of the increased speed and efficiency it provides for search.

With our previous block format, we’d seen our ability to search trace data cap out at a rate of ~40-50 GB per second. Since switching to Parquet, we’re now hitting search speeds of 300 GB/s on common queries and doing so with less compute. If we define the bytes processed as the total trace bytes that are scanned, then this example query resulted in ~1.2 TB/s:

{ resource.cluster ="foo" }

and this query resulted in ~550 GB/s:

{ span.foo ="bar" }

Another benefit of the Parquet block format relative to our previous format is that users can now leverage the large ecosystem of existing Parquet tools and libraries to process and transform their trace data. We look forward to see what downstream applications people create with this ability.

The Parquet block format can be used as a drop-in replacement for Tempo’s existing block format. No data conversion or upgrade process is necessary. As soon as the Parquet format is enabled, Tempo starts writing data in that format, leaving existing data as-is.

Tempo search will return results from both the legacy and Parquet block formats. However, search on legacy blocks has been marked as deprecated and will be removed in Tempo 2.1.

TraceQL queries will only search trace data stored in Parquet blocks. For more information, refer to the Parquet documentation.

Note: There is a potential issue loading Tempo 1.5’s experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues please report them so we can isolate and fix this issue.

Search and metrics-generator enabled by default

Search and metrics-generator are now enabled by default in Tempo 2.0, and their corresponding configuration parameters (search_enabled and metrics_generator_enabled) have been removed.

We’ve been excited by the feedback we’ve gotten on both features and are satisfied with their stability so we want all newcomers to Tempo to have access to them from day 1.

Although the metrics-generator component is now enabled by default, no actual metrics will be generated until certain tenant-specific configuration parameters have been set. For more information, refer to the metrics-generator configuration documentation.

NOTE: Grafana Cloud customers need to contact Grafana Support to enable metrics generation.

As part of promoting the metrics-generator to default on, we’ve made the following improvements to it:

The metrics-generator now handles collisions between user-defined and default dimensions. User-defined dimensions colliding with default dimensions will be prefixed with __. PR 1794
The metrics-generator now makes intrinsic dimensions configurable and disables status_message by default PR 1960.
The metrics-generator now computes a metric that tells you the size (in bytes) of spans received broken down by team, service, or other relevant labels. Since span volume roughly correlates with the cost to operate Tempo, this metric is helpful for cost attribution and chargeback purposes. PR 1662.
Older spans are now filtered out before metrics are aggregated. This is helpful since including these older span in metric computation can lead to misleading or incorrect results. PR 1612.
Label names and values that exceed a configurable length are truncated. In this way, the Tempo metrics-generator sanitizes metrics before sending them to a receiving time-series-database (which usually has its own limits around maximum label length). PR 1897.

Zone-aware replication for ingesters

Zone awareness is a feature that ensures data is replicated across failure domains (which we refer to as “zones”) to provide greater reliability. A failure domain is whatever you define it to be, but commonly may be an availability zone, data center, or server rack.

When zone awareness is enabled for ingesters, incoming trace data is guaranteed to be replicated to ingesters in different zones. This allows the system to withstand the loss of one or more zones (depending on the replication factor).

Thank you to community member manohar-koukuntla for making this contribution!

# use the following fields in _config field of jsonnet config, to enable zone aware ingesters.
    multi_zone_ingester_enabled: false,
    multi_zone_ingester_migration_enabled: false,
    multi_zone_ingester_replicas: 0,
    multi_zone_ingester_max_unavailable: 25,

Upgrade considerations

When upgrading to Tempo 2.0, be aware of these breaking changes.

Defaults updated

The following configuration parameters have been updated with new default values. These defaults were changed to work better with the new Parquet block format. PR 1978

If you are sticking with the v2 format you may prefer the old values.

query_frontend:
    max_oustanding_per_tenant: 2000
    search:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        max_duration: 168h
        query_ingesters_until: 30m
    trace_by_id:
        query_shards: 50
querier:
    max_concurrent_queries: 20
    search:
        prefer_self: 10
ingester:
    concurrent_flushes: 4
    max_block_duration: 30m
    max_block_bytes: 524288000
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        search:
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576

Removed and renamed configuration parameters

The following tables describe the parameters that have been removed or renamed. (PR#1978)

Removed

Parameter	Comments
query_frontend: query_shards:	Replaced by `trace_by_id.query_shards`
querier: query_timeout:	Replaced by two different settings: `search.query_timeout` and `trace_by_id.query_timeout`
ingester: use_flatbuffer_search:	Removed and automatically determined based on block format
`search_enabled`	Removed. Now defaults to true.
`metrics_generator_enabled`	Removed. Now defaults to true.
storage: wal: version:	Removed and pinned to `block.version`

Renamed

Parameter	Comments
query_frontend: query_shards:	Renamed `query_frontend.trace_by_id.query_shards`
querier: query_timeout:	Renamed `querier.trace_by_id.query_timeout`

The following compactor configuration parameters were renamed.

Parameter	Comments
compaction: chunk_size_bytes:	Renamed to `v2_in_buffer_bytes`
compaction: flush_size_bytes:	Renamed to `v2_out_buffer_bytes`
compaction: iterator_buffer_size:	Renamed to `v2_prefetch_traces_count`

The following storage configuration parameters were renamed.

Parameter	Comments
wal: encoding:	Renamed to `v2_encoding`
block: index_downsample_bytes:	Renamed to `v2_index_downsample_bytes`
block: index_page_size_bytes:	Renamed to `v2_index_page_size_bytes`
block: encoding:	Renamed to `v2_encoding`
block: row_group_size_bytes:	Renamed to `parquet_row_group_size_bytes`

Other upgrade considerations

PR 1879. The Azure Storage configuration now uses snake case with underscores (_) instead of dashes (-)Use snake case on Azure Storage config. Here is an example of using snake case on Azure Storage config:

# config.yaml
storage:
  trace:
    azure:
      storage_account_name:
      storage_account_key:
      container_name:

PR 1678. Parquet is the new default block version. To continue using the v2 block format, see the Parquet configuration documentation.

PR 1810. We’ve deleted the TempoRequestErrors alert from mixin. Any Jsonnet users relying on this alert should copy this into their own environment.

PR 1754. We’ve added TLS support to the vulture. Internal types are updated to use scope instead of instrumentation_library. This is a breaking change in “trace by ID” type queries if JSON is requested.

Bug fixes

2.0.1

Enhancements

PR 2059 No longer return status.code from /api/search/tags unless it is an attribute present in the data,
PR 2062 Pad leading zeroes in span id to always be 16 characters.

Bug fixes

PR 2058 Suppress logspam in single binary mode when metrics generator is disabled.
PR 2055 Error more gracefully while reading some blocks written by an interim commit between 1.5 and 2.0.
PR 2095 Correctly coalesce trace level data when combining Parquet traces.
PR 2114 Unescape query parameters in AWS Lambda to allow TraceQL queries to work.

2.0 bug fixes

Version 2.0 includes the following fixes:

PR 1887 Stop distributors on OTel receiver fatal error.
PR 1700 New wal file separator + for the NTFS filesystem and backward compatibility with the old separator :.
PR 1697 Honor caching and buffering settings when finding traces by ID.
PR 1723 Correctly propagate errors from the iterator layer up through the queriers.
PR 1781 Make multitenancy work with HTTP.
PR 1799 Fix parquet search bug fix on http.status_code that may cause incorrect results to be returned.
PR 1813 Fix failing SearchTagValues endpoint after startup.
PR 1913 tempo-mixin: tweak dashboards to support metrics without cluster label present.
PR 1920 Fix docker-compose examples not running on Apple M1 hardware.
PR 1939 Fix TraceQL parsing of most binary operations to not require spacing.
PR 1947 Don’t persist tenants without blocks in the ingester.
PR 1948 TraceQL: span scope not working with ranges.
PR 1997 TraceQL: skip live traces search.
PR 2003 Return more consistent search results by combining partial traces.

Version 1.5 release notes

Fri, 03 Apr 2026 12:35:46 -0500

Version 1.5 release notes

The Tempo team is pleased to announce the release of Tempo 1.5.

With this release, Tempo adds experimental support for a new trace storage format based on Apache Parquet, anonymous usage reporting to help the Tempo team understand how Tempo is used, and enhancements to the metrics-generator.

We encourage users to upgrade to Grafana 9.1 to take advantage of new Tempo-powered capabilities, such as the service graph view which visualizes span metrics (traces data for rates, error rates, and durations (RED)) and service graphs.

Note: For a complete list of changes and enhancements, refer to the Tempo 1.5 changelog.

Features and enhancements

Version 1.5 contains the following features and enhancements.

Apache Parquet block format support

Tempo now supports a columnar block format for trace storage based on Apache Parquet. We’re excited about this new format because it will enable increased speed and efficiency for Tempo search. It also allows users to leverage the large ecosystem of existing Parquet tools and libraries to process and transform their trace data.

We’re releasing the Parquet block format as experimental in Tempo 1.5, with plans to move it to stable in a future release once we’ve had more opportunity to test it.

The new Parquet block format can be used as a drop-in replacement for Tempo’s existing block format. No data conversion or upgrade process is necessary. As soon as the Parquet format is enabled, Tempo starts writing data in that format, leaving existing data as-is. For more information, refer to the Parquet documentation. Please note, however, that enabling the Parquet block format means Tempo will require more CPU and memory resources than it previously did.

To use Parquet, set the block format option to vParquet in the configuration file. Refer to the Parquet documentation and the Configuration documentation.

# block format version. options: v2, vParquet
[version: <string> | default = v2]

For more information about the Parquet implementation, refer to the Parquet design document and Issue 1480.

Metrics-generator enhancements

The metrics-generator now includes databases and requests across messaging systems in service graph metrics. The OpenTelemetry semantic conventions are used to detect when a span is involved in one of these systems. PR 1576

In addition, metrics-generator supports per-tenant processor configuration. PR 1434

Anonymous usage reporting

By default, Tempo will report anonymous usage data about itself to Grafana Labs. This data is used to understand which features are commonly enabled, as well as which deployment modes, replication factors, and compression levels are most popular, etc.

By providing information on how people use Tempo, usage reporting helps the Tempo team decide where to focus their development and documentation efforts. No private information is collected, and all reports are completely anonymous.

Reporting is controlled by the usage_report configuration option and can be disabled. For instructions, refer to the Configuration documentation. PR 1465

Upgrade considerations

When upgrading to Tempo v1.5, be aware of these breaking changes:

PR 1478 To build advanced visualization features into Grafana, we have decided to change our spanmetric names to match OpenTelemetry (OTel) conventions. Any functionality added to Grafana will work whether you use Tempo, Grafana Agent, or the OTel Collector to generate metrics. The old histogram metric name in Tempo 1.4.x:
```
traces_spanmetrics_duration_seconds_{sum,count,bucket}
```
has been replaced with this new name in Tempo 1.5:
```
traces_spanmetrics_latency_{sum,count,bucket}
```
Additionally, the default label span_status has been renamed to status_code.

PR 1556 Jsonnet users will need to specify ephemeral storage requests and limits for the metrics generator.

 _config+:: {
   metrics_generator+: {
     ephemeral_storage_request_size: '10Gi',
     ephemeral_storage_limit_size: '11Gi',
   },
 }

PR 1481 As a result of the anonymous usage reporting feature, distributors and metrics-generators now require the same level of access to object storage as compactors and ingesters. Users who disable anonymous usage reporting are exempt from this requirement.
PR 1558 Deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds have been removed in favor of tempodb_backend_request_duration_seconds. These metrics have been deprecated since v1.1.

Bug fixes

1.5.0 bug fixes

Version 1.5.0 includes the following fixes:

PR 1441 Fixed nil pointer panic when the trace by id path errors.
PR 1472 Updated Tempo microservices Helm values example which missed the enabled key for thriftHttp.
PR 1468 Fixed race condition in forwarder overrides loop.
PR 1538 Fixed v2 backend check on span name to be substring.
PR 1548 Fixed wal check on span name to be substring.
Issue 1258 Prevented the ingester panic “cannot grow buffer”.
PR 1554 metrics-generator: do not remove x-scope-orgid header in single tenant modus.
PR 1589 Fixed an issue where backend does not support root.name and root.service.name.
Issue 1603 Fixed the ingester to continue starting up after block replay error.

Version 1.4 release notes

Fri, 03 Apr 2026 12:35:46 -0500

Version 1.4 release notes

The Tempo team is excited to announce the release of Tempo 1.4.

Features and enhancements

Server-side metrics As the headline feature for Tempo 1.4, Server-side metrics generate metrics from your traces, for advanced insight into your application’s performance over time.
Start and end time start and end parameters were added to the /api/traces/{traceID} endpoint. This allows more efficient searches by only querying a subset of blocks on the backend. PR 1388
Protect instances from very large traces Additional protection against large traces has been added. These features all key off the existing max_bytes_per_trace configuration parameter. PR 1317 and PR 1318
Hedge everything Improve backend search throughput Tempo using our favorite solution: hedging.
```
querier:
  search:
    external_hedge_requests_at: 5s
    external_hedge_requests_up_to: 3
```

Upgrade considerations

When upgrading to Tempo v1.4, be aware of these breaking changes:

PR 1227 A new ingestion endpoint was added to support faster search and proper start/end times on blocks. Roll out all ingesters before rolling the distributors to prevent downtime. Also, during this period, the ingesters will use considerably more resources and should be scaled up, or incoming traffic should be heavily throttled. Once all distributors and ingesters have rolled, performance will return to normal. Internally, we have observed ~1.5x CPU load on the ingesters during the rollout.

PR 1350 A few querier options were moved while adding hedging to external endpoints.

querier:
search_query_timeout: 30s
search_external_endpoints: []
search_prefer_self: 2

was relocated to

querier:
search:
    query_timeout: 30s
    prefer_self: 2
    external_endpoints: []

PR 1297 Updated vulture testing of full backend search, and as a result, dropped the CLI parameter tempo-search-retention-duration.

Bug fixes

1.4.0 bug fixes

PR 1390 Correct the handling of Azure “Blob Not Found” errors.
PR 1314 Correct block start and end times to be based on actual trace start/end times, instead of on trace ingestion times.
PR 1387 Eliminate a data race and ingester crash during search by trace ID.
PR 1389 Eliminate spurious “failed to mark block compacted during retention” errors.
PR 1379 Eliminate an error with the message “Writer is closed” by resetting the compression writer correctly on the error path.

1.4.1 bug fixes

PR 1417 Metrics-generator: don’t inject X-Scope-OrgID header for single-tenant setups.
PR 1420 Compactor: populate compaction_objects_combined_total and tempo_discarded_spans_total{reason="trace_too_large_to_compact"} metrics again.
PR 1422 Distributor: prevent panics when concurrently calling shutdown to forwarder’s queueManager.

Version 1.3 release notes

Fri, 03 Apr 2026 12:35:46 -0500

Version 1.3 release notes

The Tempo team is excited to announce the release of Tempo 1.3.

Features and enhancements

Search In addition to the existing search of recent traces, support has been added to search traces in the backend store. Refer to Tempo search for details.
Improve ring functionality The compactor ring automatically forgets about unhealthy ring members.
Protect instances from very large traces The ingester keeps track of traces that exceed the configured maximum size, extending protection from very large traces to after the traces are flushed to disk.
Expose additional metrics These new metrics are exposed:
- tempo_limits_overrides
- tempo_limits_defaults
- tempo_ingester_live_traces
- tempodb_compaction_outstanding_blocks
Azure credentials may be set in the environment Instead of setting values in the configuration, environment variables may specify AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY.

Upgrade considerations

When upgrading to Tempo v1.3, be aware of these breaking changes:

PR 1142: The OpenTelemetry Collector’s GRPC receiver’s default port changed from 55680 to 4317.
PR 1174: Configuration option querier.search_max_result_limit is renamed to query_frontend.search.max_result_limit. Configuration option querier.search_default_result_limit is renamed to query_frontend.search.default_result_limit.
PR 1173: Removed the deprecated Push method from tempopb.Pusher.

Known issues

Grafana does not yet support the tempoBackendSearch feature toggle to enable searches in the backend datastore.

Bug fixes

1.3.1 bug fixes

PR 1260 mapno: Fixed a panic that occurred when using etcd as the ring’s backend storage.

1.3.2 bug fixes

PR 1295 joe-elliott: Fixed an issue which corrupted the start/end time ranges of Tempo searches that targeted ingesters.

Version 1.2 release notes

Fri, 03 Apr 2026 12:35:46 -0500

Version 1.2 release notes

The Tempo team is excited to announce the release of Tempo 1.2. This release introduces Tempo search.

Features and enhancements

Recent traces search The headline feature of Tempo 1.2 is the ability to search for data that is still in the ingester component. This experimental feature must be enabled using feature flags in both Grafana and Tempo. Refer to Tempo search for details.
Scalable monolithic deployment mode (formerly scalable single binary deployment mode) The new scalable monolithic deployment mode is operationally simpler, although less robust than a fully distributed deployment. We consider it a balanced operational mode that some may find attractive.
Massive performance improvements were achieved by more efficiently batching trace data being ingested. In terms of CPU needed, Tempo v1.2 is twice as efficient as Tempo v1.1. Refer to PR 1075 for details.
API improvements Some informational endpoints were consolidated within the status endpoint. New endpoints were also added.

Upgrade considerations

When upgrading to Tempo v1.2, be aware of these changes:

Support for v0 and v1 blocks were dropped as announced with Tempo v1.1. Refer to the Tempo v1.1 changelog for details.
PR 1007 altered the Querier API. When upgrading components within a deployment, there will be a read outage until all queriers and query frontends have rolled to the upgraded version.
API improvements consolidated informational endpoints within the status endpoint.
Metric ingester_bytes_metric_total is renamed ingester_bytes_received_total by PR 979.
Metric cortex_runtime_config_last_reload_successful is renamed tempo_runtime_config_last_reload_successful by PR 945.
The tempo-cli flag --storage.trace.maintenance-cycle is renamed -storage.trace.blocklist_poll by PR 897.

Bug fixes

1.2.1 bug fixes

Honor the default values of configuration parameters max_bytes_per_trace and max_search_bytes_per_trace. PR #1109 (@BitProcessor)
Reclassify the writing of empty traces to be a warning instead of an error. PR #1113 (@mdisibio)

1.2.0 bug fixes

Fixed errors with magic numbers and other block mishandling when an ingester forcefully shuts down. Issue #937 (@mdisibio)
Fixed a memory leak in the compactor component. PR #806 (@mdisibio)
Set a span’s tag span.kind to client in query-frontend to allow query frontend spans to be paired with querier server spans. PR #975 (@mapno)
Metric tempodb_backend_hedged_roundtrips_total now correctly counts hedged roundtrips. PR #1079 (@joe-elliott)
Updated the go-kit logger package to remove spurious debug logs. PR #1094 (@bboreham)