Manage Tempo on Grafana Labs

Tempo architecture

Fri, 03 Apr 2026 12:35:46 -0500

Tempo architecture

This topic provides an overview of the major components of Tempo. Refer to the example setups or deployment options for help deploying.

Tempo comprises of the following top-level components.

Distributor

The distributor accepts spans in multiple formats including Jaeger, OpenTelemetry, Zipkin. It routes spans to ingesters by hashing the traceID and using a distributed consistent hash ring. The distributor uses the receiver layer from the OpenTelemetry Collector. For best performance it is recommended to ingest OTel Proto. For this reason the Grafana Agent uses the otlp exporter/receiver to send spans to Tempo.

Ingester

The Ingester batches trace into blocks, creates bloom filters and indexes, and then flushes it all to the backend. Blocks in the backend are generated in the following layout.

<bucketname> / <tenantID> / <blockID> / <meta.json>
                                      / <index>
                                      / <data>
                                      / <bloom_0>
                                      / <bloom_1>
                                        ...
                                      / <bloom_n>

Query Frontend

The Query Frontend is responsible for sharding the search space for an incoming query.

Traces are exposed via a simple HTTP endpoint: GET /api/traces/<traceID>

Internally, the Query Frontend splits the blockID space into a configurable number of shards and queues these requests. Queriers connect to the Query Frontend via a streaming gRPC connection to process these sharded queries.

Querier

The querier is responsible for finding the requested trace id in either the ingesters or the backend storage. Depending on parameters it will query both the ingesters and pull bloom/indexes from the backend to search blocks in object storage.

The querier exposes an HTTP endpoint at: GET /querier/api/traces/<traceID>, but its not expected to be used directly.

Queries should be sent to the Query Frontend.

Compactor

The Compactors stream blocks to and from the backend storage to reduce the total number of blocks.

Metrics generator

This is an optional component that derives metrics from ingested traces and writes them to a metrics storage. Refer to the metrics-generator documentation to learn more.

Server-side metrics architecture

Fri, 03 Apr 2026 12:35:46 -0500

Server-side metrics architecture

Server-side metrics is a feature that derive metrics from ingested traces.

To generate metrics, it uses an additional component: the metrics-generator. If present, the distributor will write received spans to both the ingester and the metrics-generator. The metrics-generator processes spans and writes metrics to a Prometheus datasource using the Prometheus remote write protocol.

Architecture

Generating and writing metrics introduces a whole new domain to Tempo unlike any other functionality thus far. For this reason, a component, the metrics-generator, is dedicated to working with metrics. This results in a clean division of responsibility and limits the blast radius from a metrics processors or the Prometheus remote write exporter blowing up.

                                                                      │
                                                                      │
                                                                   Ingress
                                                                      │
                                                                      ▼
                                                          ┌──────────────────────┐
                                                          │                      │
                                                          │     Distributor      │
                                                          │                      │
                                                          └──────────────────────┘
                                                                    2│ │1
                                                                     │ │
                                                  ┌──────────────────┘ └────────┐
                                                  │                             │
                                                  ▼                             ▼
┌ ─ ─ ─ ─ ─ ─ ─ ─                     ┏━━━━━━━━━━━━━━━━━━━━━━┓      ┌──────────────────────┐
                 │                    ┃                      ┃      │                      │
│   Prometheus    ◀────Prometheus ────┃  Metrics-generator   ┃      │       Ingester       │◀───Queries────
                 │    Remote Write    ┃                      ┃      │                      │
└ ─ ─ ─ ─ ─ ─ ─ ─                     ┗━━━━━━━━━━━━━━━━━━━━━━┛      └──────────────────────┘
                                                                                │
                                                                                │
                                                                                │
                                                                                ▼
                                                                       ┌─────────────────┐
                                                                       │                 │
                                                                       │     Backend     │
                                                                       │                 │
                                                                       └─────────────────┘

Configuration

For a detailed view of all the config options for the metrics generator, visit its config page.

Best practices for traces

Fri, 03 Apr 2026 12:35:46 -0500

Best practices for traces

This page provides some general best practices for tracing.

Span and resource attributes

Traces are built from spans, which denote units of work such as a call to, or from, an upstream service. Spans are constructed primarily of span and resource attributes. Spans also have a hierarchy, where parent spans can have children or siblings.

In the screenshot below, the left side of the screen (1) shows the list of results for the query. The right side (2) lists each span that makes up the selected trace.

A span attribute is a key/value pair that provides context for its span. For example, if the span deals with calling another service via HTTP, an attribute could include the HTTP URL (maybe as the span attribute key http.url) and the HTTP status code returned (as the span attribute http.status_code). Span attributes can consist of varying, non-null types.

Unlike a span attribute, a resource attribute is a key/value pair that describes the context of how the span was collected. Generally, these attributes describe the process that created the span. For example, this could be a set of resource attributes concerning a Kubernetes cluster, in which case you may see resource attributes, for example: k8s.namespace, k8s.container_name, and k8s.cluster. These can also include information on the libraries that were used to instrument the spans for a trace, or any other infrastructure information.

For more information, read the Attribute and Resource sections in the OpenTelemetry specification.

Naming conventions for span and resource attributes

When naming attributes, use consistent, nested namespaces to ensures that attribute keys are obvious to anyone observing the spans of a trace and that common attributes can be shared by spans. Using our example from above, the http prefix of the attribute is the namespace, and url and status_code are keys within that namespace. Attributes can also be nested, for example http.url.protocol might be HTTP or HTTPS, whereas http.url.path might be /api/v1/query.

For more details around semantic naming conventions, refer to the Recommendations for OpenTelemetry Authors documentation.

Some third-party libraries provide auto-instrumentation that generate span and span attributes when included in a source base.

For more information about instrumenting your app for tracing, refer to the Instrument for distributed tracing documentation.

Determining where to add spans

When instrumenting, determine the smallest piece of work that you need to observe in a trace to be of value to ensure that you don’t over (or under) instrument.

Creating a new span for any work that has a relatively significant duration allows the observation of a trace to immediately show where significant amounts of time are spent during the processing of a request into your application or system.

For example, adding a span for a call to another services (either instrumented or not) may take an unknown amount of time to complete, and therefore being able to separate this work shows when services are taking longer than expected.

Adding a span for a piece of work that might call many other functions in a loop is a good signal of how long that loop is taking (you might add a span attribute that counts how many time the loop runs to determine if the duration is acceptable). However, adding a span for each method or function call in that loop might not, as it might produce hundreds or thousands of worthless spans.

Monitor Tempo

Fri, 03 Apr 2026 12:35:46 -0500

Monitor Tempo

Tempo is instrumented to expose metrics, logs, and traces. Furthermore, the Tempo repository has a mixin that includes a set of dashboards, rules, and alerts. Together, these can be used to monitor Tempo in production.

Instrumentation

Tempo is already instrumented with metrics, logs, and traces. These can be collected to observe Tempo.

Metrics

Tempo is instrumented with Prometheus metrics. It emits RED metrics for most services and backends. (RED metrics are a standardized format for monitoring microservices, where R stands for requests, E stands for errors, and D stands for duration)

The Tempo mixin provides several dashboards using these metrics.

Logs

Tempo emits logs in the key=value (logfmt) format.

Traces

Tempo uses the Jaeger Golang SDK for tracing instrumentation. As of this writing, the complete read path and some parts of the write of Tempo are instrumented for tracing.

The tracer can be configured using environment variables. To enable tracing, set one of the following: JAEGER_AGENT_HOST and JAEGER_AGENT_PORT, or JAEGER_ENDPOINT.

The Jaeger client uses remote sampling by default, if the management server is not available no traces will be sent. To always send traces (no sampling), set the following environment variables:

JAEGER_SAMPLER_TYPE=const
JAEGER_SAMPLER_PARAM=1

Dashboards

The Tempo mixin has four Grafana dashboards in the yamls folder that you can download and import into your Grafana UI. Currently, these work well when Tempo is run in a Kubernetes (k8s) environment and metrics scraped have the cluster and namespace labels.

Tempo Reads dashboard

This is available as tempo-reads.json.

The Reads dashboard gives information information on Requests, Errors and Duration (R.E.D) on the Query Path of Tempo. Each query touches the Gateway, Tempo-Query, Query-Frontend, Queriers, Ingesters, Cache (if present) and the backend.

Use this dashboard to monitor the performance of each of the above mentioned components and to decide the number of replicas in each deployment.

Tempo Writes dashboard

This is available as tempo-writes.json.

The Writes dashboard gives information information on Requests, Errors and Duration (R.E.D) on the write/ingest Path of Tempo. A write query touches the Gateway, Distributors, Ingesters and eventually the backend. This dashboard also gives information on the number of operations performed by the Compactor to the backend.

Use this dashboard to monitor the performance of each of the above mentioned components and to decide the number of replicas in each deployment.

Tempo Resources dashboard

This is available as tempo-resources.json.

The Resources dashboard provides information on CPU, Container Memory and Go Heap Inuse, and is useful for resource provisioning for the different Tempo components.

Use this dashboard to see if any components are running close to their assigned limits!

Tempo Operational dashboard

This is available as tempo-operational.json.

The Tempo Operational dashboard deserves special mention b/c it probably a stack of dashboard anti-patterns. It’s big and complex, doesn’t use jsonnet and displays far too many metrics in one place. And I love it. For just getting started the Reads, Write and Resources dashboards are great places to learn how to monitor Tempo in an opaque way.

This dashboard is included in this repo for two reasons:

It provides a stack of metrics for other operators to consider monitoring while running Tempo.
We want it in our internal infrastructure and we vendor the tempo-mixin to do this.

Rules and alerts

The Rules and Alerts are available as yaml files in the compiled mixin on the repository.

To set up alerting, download the provided json files and configure them for use on your Prometheus monitoring server.

Check the runbook to understand the various steps that can be taken to fix firing alerts!

Use polling to monitor Tempo's backend status

Fri, 03 Apr 2026 12:35:46 -0500

Use polling to monitor Tempo’s backend status

Tempo maintains knowledge of the state of the backend by polling it on regular intervals. There are currently only two components that need this knowledge and, consequently, only two that poll the backend: compactors and queriers.

To reduce calls to the backend, only a small subset of compactors actually list all blocks and build what’s called a tenant index. The tenant index is a gzip’ed JSON file located at /<tenant>/index.json.gz containing an entry for every block and compacted block for that tenant. This is done once every blocklist_poll duration.

All other compactors and all queriers then rely on downloading this file, unzipping it and using the contained list. Again, this is done once every blocklist_poll duration.

Due to this behavior, a given compactor or querier will often have an out-of-date blocklist. During normal operation, it will stale by at most twice the configured blocklist_poll.

Note: For details about configuring polling, see polling configuration.

Monitor polling with dashboards and alerts

See our Jsonnet for example alerts and runbook entries related to polling.

If you are building your own dashboards or alerts, here are a few relevant metrics:

tempodb_blocklist_poll_errors_total A holistic metric that increments for any error with polling the blocklist. Any increase in this metric should be reviewed.
tempodb_blocklist_poll_duration_seconds Histogram recording the length of time in seconds to poll the entire blocklist.
tempodb_blocklist_length Total blocks as seen by this component.
tempodb_blocklist_tenant_index_errors_total A holistic metrics that indcrements for any error building the tenant index. Any increase in this metric should be reviewed.
tempodb_blocklist_tenant_index_builder A gauge that has the value 1 if this compactor is attempting to build the tenant index and 0 if it is not. At least one compactor must have this value set to 1 for the system to be working.
tempodb_blocklist_tenant_index_age_seconds The age of the last loaded tenant index. now() minus this value indicates how stale this components view of the blocklist is.

Tune the consistent hash rings

Fri, 03 Apr 2026 12:35:46 -0500

Tune the consistent hash rings

Tempo uses the Consistent Hash Ring implementation from Cortex. By default, the ring is gossiped between all Tempo components. However, it can be configured to use Consul or Etcd, if desired.

There are four consistent hash rings: distributor, ingester, metrics-generator, and compactor. Each hash ring exists for a distinct reason.

Distributor

Participants: Distributors

Used by: Distributors

Unless you are running with limits, this ring does not impact Tempo operation.

This ring is only used when global rate limits are used. The distributors use it to count the other active distributors. Incoming traffic is assumed to be evenly spread across all distributors and (global_rate_limit / # of distributors) is used to rate limit locally.

Ingester

Participants: Ingesters

Used by: Distributors, Queriers

This ring is used by the distributors to load balance traffic into the ingesters. When spans are received the trace id is hashed and they are sent to the appropriate ingesters based on token ownership in the ring. Queriers also use this ring to find the ingesters for querying recent traces.

Metrics-generator

Participants: Metrics-generators

Used by: Distributors

This ring is used by distributors to load balance traffic to the metrics-generators. When spans are received, the trace ID is hashed, and the traces are sent to the appropriate metrics-generators based on token ownership in the ring.

Compactor

Participants: Compactors

Used by: Compactors

This ring is used by the compactors to shard compaction jobs. Jobs are hashed into the ring and the owning compactor is the only one allowed to compact a specific set of blocks to prevent race conditions on compaction.

Interacting with the rings

Web pages are available at the following endpoints. They show every ring member, their tokens and includes the ability to “Forget” a ring member. “Forgetting” is useful when a ring member leaves the ring without properly shutting down (and therefore leaves its tokens in the ring).

Distributor

Available on: Distributors

Path: /distributor/ring

Unhealthy distributors have little impact but should be forgotten to reduce cost of maintaining the ring .

Ingester

Available on: Distributors

Path: /ingester/ring

Unhealthy ingesters will cause writes to fail. If the ingester is really gone, forget it immediately.

Metrics-generators

Available on: Distributors

Path: /metrics-generator/ring

Unhealthy metrics-generators will cause writes to fail. If the metrics-generator is really gone, forget it immediately.

Compactor

Available on: Compactors

Path: /compactor/ring

Unhealthy compactors will allow the blocklist to grow significantly. If the compactor is really gone, forget it immediately.

Configuring the rings

Ring/Lifecycler configuration control how a component interacts with the ring. Refer to the configuration topic for more details.

Resize ingester persistent volume operations

Fri, 03 Apr 2026 12:35:46 -0500

Resize ingester persistent volume operations

Tempo ingesters make heavy use of local disks to store write-ahead logs and blocks before being flushed to the backend (GCS, S3, etc.). It is important to monitor the free volume space as full disks can lead to data loss and other errors. The amount of disk space available affects how much volume a Tempo ingester can process and the length of time an outage to the backend can be tolerated.

Therefore it may be necessary to increase the disk space for ingesters as usage increases.

We recommend using SSDs for local storage.

Increase VCC size

When deployed as a StatefulSet with Persistent Volume Claims (PVC), some manual steps are required. The following configuration has worked successfully on GKE with GCS:

Edit the persistent volume claim (pvc) for each ingester to the new size.
Bash
```
kubectl patch pvc -n <namespace> -p '{"spec": {"resources": {"requests": {"storage": "'15Gi'"}}}}' <pod-name>
```
Check all disks have been upgraded by running:

kubectl get pvc -n <namespace>

A restart is not necessary as the pods will automatically detect the increased disk space.
Delete the StatefulSet but leave the pods running:

kubectl delete sts --cascade=false -n <namespace> ingester
Edit and recreate the Statefulset with the new size. This covers new pods. There are many ways to deploy Tempo to Kubernetes, these are examples for the popular ones:
- Raw YAML: kubectl apply -f <something>.yaml
- Helm: helm upgrade ... tempo ...
- Tanka: tk apply ...

Enable multi-tenancy

Fri, 03 Apr 2026 12:35:46 -0500

Enable multi-tenancy

Tempo is a multi-tenant distributed tracing backend. It supports multi-tenancy through the use of a header: X-Scope-OrgID.

If you’re interested in setting up multi-tenancy, consult the multi-tenant example in the repo. This example uses the following settings to achieve multi-tenancy in Tempo.

Note Multi-tenancy on ingestion is currently only working with GPRC and this may never change. It is strongly recommended to use the OpenTelemetry Collector to support multi-tenancy.

Configure multi-tenancy

Configure the OTEL Collector to attach the X-Scope-OrgID header on push:
```
exporters:
  otlp:
    headers:
      x-scope-orgid: foo-bar-baz
```

Configure the Tempo data source in Grafana to pass the tenant with the same header:

- name: Tempo-Multitenant
  jsonData:
    httpHeaderName1: 'X-Scope-OrgID'
  secureJsonData:
    httpHeaderValue1: 'foo-bar-baz'

Enable multi-tenancy on Tempo backend, set the following configuration value on all Tempo components:
```
multitenancy_enabled: true
```
or from the command line:
```
--multitenancy.enabled=true
```
This option will force all Tempo components to require the X-Scope-OrgID header.

Improve performance with caching

Fri, 03 Apr 2026 12:35:46 -0500

Improve performance with caching

Caching is mainly used to improve query performance by storing bloom filters of all backend blocks which are accessed on every query.

Tempo uses an external cache to improve query performance. The supported implementations are Memcached and Redis.

Memcached

Memcached is one of the cache implementations supported by Tempo. It is used by default in the Tanka and Helm examples, see Deploying Tempo.

Connection limit

As a cluster grows in size, the number of instances of Tempo connecting to the cache servers also increases. By default, Memcached has a connection limit of 1024. If this limit is surpassed new connections are refused. This is resolved by increasing the connection limit of Memcached.

These errors can be observed using the tempo_memcache_request_duration_seconds_count metric. For example, by using the following query:

sum by (status_code) (
  rate(tempo_memcache_request_duration_seconds_count{}[$__rate_interval])
)

This metric is also shown in the monitoring dashboards (the left panel):

Note that the already open connections continue to function, just new connections are refused.

Additionally, Memcached will log the following errors when it can’t accept any new requests:

accept4(): No file descriptors available
Too many open connections
accept4(): No file descriptors available
Too many open connections

When using the memcached_exporter, the number of open connections can be observed at memcached_current_connections.

Cache size control

Tempo querier accesses bloom filters of all blocks while searching for a trace. This essentially mandates the size of cache to be at-least the total size of the bloom filters (the working set) . However, in larger deployments, the working set might be larger than the desired size of cache. When that happens, eviction rates on the cache grow high, and hit rate drop. Not nice!

Tempo provides two config parameters in order to filter down on the items stored in cache.

        # Min compaction level of block to qualify for caching bloom filter
        # Example: "cache_min_compaction_level: 2"
        [cache_min_compaction_level: <int>]

        # Max block age to qualify for caching bloom filter
        # Example: "cache_max_block_age: 48h"
        [cache_max_block_age: <duration>]

Using a combination of these config options, we can narrow down on which bloom filters are cached, thereby reducing our cache eviction rate, and increasing our cache hit rate. Nice!

In order to decide the values of these config parameters, you can use a cache summary command in the tempo-cli that prints a summary of bloom filter shards per day and per compaction level. The result looks something like this:

The above image shows the bloom filter shards over 14 days and 6 compaction levels. This can be used to decide the above configuration parameters.

Tempo CLI

Fri, 03 Apr 2026 12:35:46 -0500

Tempo CLI

Tempo CLI is a separate executable that contains utility functions related to the Tempo software. Although it is not required for a working installation, Tempo CLI can be helpful for deeper analysis or for troubleshooting.

Tempo CLI command syntax

The general syntax for commands in Tempo CLI is:

tempo-cli command [subcommand] [options] [arguments...]

--help or -h displays the help for a command or subcommand.

Example:

tempo-cli -h
tempo-cli command [subcommand] -h

Run Tempo CLI

Tempo CLI is currently available as source code. A working Go installation is required to build it. It can be compiled to a native binary and executed normally, or it can be executed using the go run command. It can be packaged as a Docker container using make docker-tempo-cli.

Example:

./tempo-cli [arguments...]
go run ./cmd/tempo-cli [arguments...]

make docker-tempo-cli
docker run docker.io/grafana/tempo-cli [arguments...]

Backend options

Tempo CLI connects directly to the storage backend for some commands, meaning that it requires the ability to read from S3, GCS, Azure or file-system storage. The backend can be configured in a few ways:

Load an existing tempo configuration file using the --config-file (-c) option. This is the recommended option for frequent usage. Refer to Configuration documentation for more information.
Specify individual settings:
- --backend <value> The storage backend type, one of s3, gcs, azure, and local.
- --bucket <value> The bucket name. The meaning of this value is backend-specific. Refer to Configuration documentation for more information.
- --s3-endpoint <value> The S3 API endpoint (i.e. s3.dualstack.us-east-2.amazonaws.com).
- --s3-user <value>, --s3-password <value> The S3 user name and password (or access key and secret key). Optional, as Tempo CLI supports the same authentication mechanisms as Tempo. See S3 permissions documentation for more information.

Each option applies only to the command in which it is used. For example, --backend <value> does not permanently change where Tempo stores data. It only changes it for command in which you apply the option.

Query API command

Call the tempo API and retrieve a trace by ID.

tempo-cli query api <api-endpoint> <trace-id>

Arguments:

api-endpoint URL for tempo API.
trace-id Trace ID as a hexadecimal string.

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).

Example:

tempo-cli query api http://tempo:3200 f1cfe82a8eef933b

Query blocks command

Iterate over all backend blocks and dump all data found for a given trace id.

tempo-cli query blocks <trace-id> <tenant-id>

Note: can be intense as it downloads every bloom filter and some percentage of indexes/trace data.

Arguments:

trace-id Trace ID as a hexadecimal string.
tenant-id Tenant to search.

Options: See backend options above.

Example:

tempo-cli query blocks f1cfe82a8eef933b single-tenant

List blocks

Lists information about all blocks for the given tenant, and optionally perform integrity checks on indexes for duplicate records.

tempo-cli list blocks <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Options:

--include-compacted Include blocks that have been compacted. Default behavior is to display only active blocks.

Output: Explanation of output:

ID Block ID.
Lvl Compaction level of the block.
Objects Number of objects stored in the block.
Size Data size of the block after any compression.
Encoding Block encoding (compression algorithm).
Vers Block version.
Window The window of time that was considered for compaction purposes.
Start The earliest timestamp stored in the block.
End The latest timestamp stored in the block.
DurationDuration between the start and end time.
Age The age of the block.
Cmp Whether the block has been compacted (present when –include-compacted is specified).

Example:

tempo-cli list blocks -c ./tempo.yaml single-tenant

List block

Lists information about a single block, and optionally, scan its contents.

tempo-cli list block <tenant-id> <block-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Options:

--scan Also load the block data, perform integrity check for duplicates, and collect statistics. Note: can be intense.

Example:

tempo-cli list block -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

List compaction summary

Summarizes information about all blocks for the given tenant based on compaction level. This command is useful to analyze or troubleshoot compactor behavior.

tempo-cli list compaction-summary <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

tempo-cli list compaction-summary -c ./tempo.yaml single-tenant

List cache summary

Prints information about the number of bloom filter shards per day per compaction level. This command is useful to estimate and fine-tune cache storage. Read the caching topic for more information.

tempo-cli list cache-summary <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

tempo-cli list cache-summary -c ./tempo.yaml single-tenant

List index

Lists basic index info for the given block.

tempo-cli list index <tenant-id> <block-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Example:

tempo-cli list index -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

View index

View the index contents for the given block.

tempo-cli view index <tenant-id> <block-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Example:

tempo-cli view index -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

Generate bloom filter

To generate the bloom filter for a block if the files were deleted/corrupted.

Note: ensure that the block is in a local backend in the expected directory hierarchy, i.e. path / tenant / blocks.

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.
bloom-fp The false positive to be used for the bloom filter.
bloom-shard-size The shard size to be used for the bloom filter.

Example:

tempo-cli gen bloom --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0 0.05 100000

The bloom filter will be generated at the required location under the block folder.

Generate index

To generate the index/bloom for a block if the files were deleted/corrupted.

Note: ensure that the block is in a local backend in the expected directory hierarchy, i.e. path / tenant / blocks.

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Example:

tempo-cli gen index --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

The index will be generated at the required location under the block folder.

Search blocks command

Search blocks in a given time range for a specific key/value pair.

tempo-cli search blocks <name> <value> <start> <end> <tenant-id>

Note: can be intense as it downloads all relevant blocks and iterates through them.

Arguments:

name Name of the attribute to search for e.g. http.post
value Value of the attribute to search for e.g. GET
start Start of the time range to search: (YYYY-MM-DDThh:mm:ss)
end End of the time range to search: (YYYY-MM-DDThh:mm:ss)
tenant-id Tenant to search.

Options: See backend options above.

Example:

tempo-cli search blocks http.post GET 2021-09-21T00:00:00 2021-09-21T00:05:00 single-tenant --backend=gcs --bucket=tempo-trace-data

Parquet convert command

Converts a parquet file from its existing schema to the one currently in the repository. This utility command is useful when attempting to determine the impact of changing compression or encoding of columns.

tempo-cli parquet convert <in file> <out file>

Arguments:

in file Filename of an existing parquet file containing Tempo trace data
out file File to write to. (Existing file is overwritten.)

Example:

tempo-cli parquet convert data.parquet out.parquet

Migrate tenant command

Copy blocks from one backend and tenant to another. Blocks can be copied within the same backend or between two different backends. Data format will not be converted but tenant ID in meta.json will be rewritten.

tempo-cli migrate tenant <source tenant> <dest tenant>

Arguments:

source tenant Tenant to copy blocks from
dest tenant Tenant to copy blocks into

Options:

--source-config-file <value> Configuration file for the source backend
--config-file <value> Configuration file for the destination backend

Example:

tempo-cli migrate tenant --source-config source.yaml --config-file dest.yaml my-tenant my-other-tenant

Apache Parquet schema

Fri, 03 Apr 2026 12:35:46 -0500

Apache Parquet schema

Starting with Tempo 2.0, Apache Parquet is used as the default column-formatted block format. Refer to the Parquet configuration options for more information.

This document describes the schema used with the Parquet block format.

Fully nested versus span-oriented schema

There are two overall approaches to a columnar schema: fully nested or span-oriented. Span-oriented means a flattened schema where traces are destructured into rows of spans. A fully nested schema means the current trace structures such as Resource/Scope/Spans/Events are preserved (nested data is natively supported in Parquet). In both cases, individual leaf values such as span name and duration are individual columns.

We chose the nested schema for several reasons:

The block size is much smaller for the nested schema. This is due to the high data duplication incurred when flattening resource-level attributes such as service.name to each individual span.
A flat schema is not truly “flat” because each span still contains nested data such as attributes and events.
Nested schema is much faster to search for resource-level attributes because the resource-level columns are very small (1 row for each batch).
Translation to and from the OpenTelemetry Protocol Specification (OTLP) is straightforward.
Easily add computed columns (for example, trace duration) at multiple levels such as per-trace, per-batch, etc.

Static vs dynamic columns

Dynamic vs static columns add another layer to the schema. A dynamic schema stores each attribute such as service.name and http.status_code as its own column and the columns in each parquet file can be different. A static schema is unresponsive to the shape of the data, and all attributes are stored in generic key/value containers.

The dynamic schema is the ultimate dream for a columnar format but it is too complex for a first release. However, the benefits of that approach are also too good to pass up, so we propose a hybrid approach. It is primarily a static schema but with some dynamic columns extracted from trace data based on some heuristics of frequently queried attributes. We plan to continue investing in this direction to implement a fully dynamic schema where trace attributes are blown out into independent Parquet columns at runtime.

For more information, refer to the Parquet design document.

Schema details

The adopted Parquet schema is mostly a direct translation of OTLP but with some key differences.

The table below uses these abbreviations:

rs - resource spans
ss - scope spans


Name	Type	Description
TraceID	byte array	The trace ID in 16-byte binary form.
TraceIDText	string	The trace ID in hexadecimal text form.
StartTimeUnixNano	int64	Start time of the first span in the trace, in nanoseconds since unix epoch.
EndTimeUnixNano	int64	End time of the last span in the trace, in nanoseconds since unix epoch.
DurationNano	int64	Total trace duration in nanoseconds, computed as difference between EndTimeUnixNano and StartTimeUnixNano.
RootServiceName	string	The resource-level `service.name` attribute (rs.Resource.ServiceName) from the root span of the trace if one exists, else null.
RootSpanName	string	The name (rs.ss.Spans.Name) of the root span if one exists, else null.
rs		Short-hand for ResourceSpans
rs.Resource.ServiceName	string	A dedicated column for the resource-level `service.name` attribute if present. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service
rs.Resource.Cluster	string	A dedicated column for the resource-level `cluster` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Namespace	string	A dedicated column for the resource-level `namespace` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Pod	string	A dedicated column for the resource-level `pod` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Container	string	A dedicated column for the resource-level `container` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.K8sClusterName		A dedicated column for the resource-level `k8s.cluster.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#cluster
rs.Resource.K8sNamespaceName	string	A dedicated column for the resource-level `k8s.namespace.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#namespace
rs.Resource.K8sPodName	string	A dedicated column for the resource-level `k8s.pod.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#pod
rs.Resource.K8sContainerName	string	A dedicated column for the resource-level `k8s.container.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#container
rs.Resource.Attrs.Key	string	All resource attributes that do not have a dedicated column are stored as a key value pair in these columns. The Key column stores the name, and then one of the Value columns is populated according to the attribute’s data type. The other value columns will contain null.
rs.Resource.Attrs.Value	string	The attribute value if string type, else null.
rs.Resource.Attrs.ValueInt	int	The attribute value if integer type, else null.
rs.Resource.Attrs.ValueDouble	float	The attribute value if float type, else null.
rs.Resource.Attrs.ValueBool	bool	The attribute value if boolean type, else null.
rs.Resource.Attrs.ValueArray	byte array	The attribute value if nested array type, else null. Protocol buffer encoded binary data.
rs.Resource.Attrs.ValueKVList	byte array	The attribute value if nested key/value map type, else null. Protocol buffer encoded binary data.
rs.ss		Shorthand for ResourceSpans.ScopeSpans
rs.ss.Scope		Shorthand for ResourceSpans.ScopeSpans.Scope
rs.ss.Scope.Name	string	Scope name if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope
rs.ss.Scope.Version	string	The Scope version if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope
rs.ss.Spans.SpanID	byte array	Span unique ID.
rs.ss.Spans.ParentSpanID	byte array	The unique ID of the span’s parent. For root spans without a parent this is null.
rs.ss.Spans.ParentID	int32	Trace local numeric parent ID.
rs.ss.Spans.NestedSetLeft	int32	Left bound of the nested set model. Also used as a trace local numeric span ID.
rs.ss.Spans.NestedSetRight	int32	Right bound of the nested set model.
rs.ss.Spans.Name	string	Span name.
rs.ss.Spans.StartTimeUnixNano	int64	Start time the span in nanoseconds since unix epoch.
rs.ss.Spans.DurationNano	int64	Span duration in nanoseconds.
rs.ss.Spans.Kind	int	The span’s kind. Defined values: 0. Unset; 1. Internal; 2. Server; 3. Client; 4. Producer; 5. Consumer; https://opentelemetry.io/docs/reference/specification/trace/api/#spankind
rs.ss.Spans.StatusCode	int	The span status. Defined values: 0: Unset; 1: OK; 2: Error. https://opentelemetry.io/docs/reference/specification/trace/api/#set-status
rs.ss.Spans.StatusMessage	string	Optional message to accompany Error status.
rs.ss.Spans.HttpMethod	string	A dedicated column for the span-level `http.method` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes
rs.ss.Spans.HttpStatusCode	int	A dedicated column for the span-level `http.status_code` attribute if present and of integer type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes
rs.ss.Spans.HttpUrl	string	A dedicated column for the span-level `http.url` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#http-client
rs.ss.Spans.DroppedAttributesCount	int	Number of attributes that were dropped
rs.ss.Spans.Attrs.Key	string	All span attributes that do not have a dedicated column are stored as a key value pair in these columns. The Key column stores the name, and then one of the Value columns is populated according to the attribute’s data type. The other value columns will contain null.
rs.ss.Spans.Attrs.Value	string	The attribute value if string type, else null.
rs.ss.Spans.Attrs.ValueInt	int	The attribute value if integer type, else null.
rs.ss.Spans.Attrs.ValueDouble	float	The attribute value if float type, else null.
rs.ss.Spans.Attrs.ValueBool	bool	The attribute value if boolean type, else null.
rs.ss.Spans.Attrs.ValueArray	byte array	The attribute value if nested array type, else null. Protocol buffer encoded binary data.
rs.ss.Spans.Attrs.ValueKVList	byte array	The attribute value if nested key/value map type, else null. Protocol buffer encoded binary data.
rs.ss.Spans.DroppedEventsCount	int	The number of events that were dropped
rs.ss.Spans.Events.TimeUnixNano	int64	The timestamp of the event, as nanoseconds since unix epoch.
rs.ss.Spans.Events.Name	string	The event name or message.
rs.ss.Spans.Events.DroppedAttributesCount	int	The number of event attributes that were dropped.
rs.ss.Spans.Events.Attrs.Key	string	All event attributes are stored as a key value pair in these columns. The Key column stores the name.
rs.ss.Spans.Events.Attrs.Value	byte array	The attribute value, Protocol buffer encoded binary data.
rs.ss.Spans.DroppedLinksCount	int	The number of links that were dropped.
rs.ss.Spans.Links	byte array	Protocol-buffer encoded span links if present, else null.
rs.ss.Spans.TraceState	string	The span’s TraceState value if present, else empty string.https://opentelemetry.io/docs/reference/specification/trace/api/#tracestate

To increase the readability table omits the groups list.element that are added for nested list types in Parquet.

Block Schema display in Parquet Message format

message Trace {
  required binary TraceID;
  required group rs (LIST) {
    repeated group list {
      required group element {
        required group Resource {
          required group Attrs (LIST) {
            repeated group list {
              required group element {
                required binary Key (STRING);
                optional binary Value (STRING);
                optional int64 ValueInt (INTEGER(64,true));
                optional double ValueDouble;
                optional boolean ValueBool;
                optional binary ValueKVList (STRING);
                optional binary ValueArray (STRING);
              }
            }
          }
          required binary ServiceName (STRING);
          optional binary Cluster (STRING);
          optional binary Namespace (STRING);
          optional binary Pod (STRING);
          optional binary Container (STRING);
          optional binary K8sClusterName (STRING);
          optional binary K8sNamespaceName (STRING);
          optional binary K8sPodName (STRING);
          optional binary K8sContainerName (STRING);
          optional binary Test (STRING);
        }
        required group ss (LIST) {
          repeated group list {
            required group element {
              required group Scope {
                required binary Name (STRING);
                required binary Version (STRING);
              }
              required group Spans (LIST) {
                repeated group list {
                  required group element {
                    required binary SpanID;
                    required binary ParentSpanID;
                    required int32 ParentID (INTEGER(32,true));
                    required int32 NestedSetLeft (INTEGER(32,true));
                    required int32 NestedSetRight (INTEGER(32,true));
                    required binary Name (STRING);
                    required int64 Kind (INTEGER(64,true));
                    required binary TraceState (STRING);
                    required int64 StartTimeUnixNano (INTEGER(64,false));
                    required int64 DurationNano (INTEGER(64,false));
                    required int64 StatusCode (INTEGER(64,true));
                    required binary StatusMessage (STRING);
                    required group Attrs (LIST) {
                      repeated group list {
                        required group element {
                          required binary Key (STRING);
                          optional binary Value (STRING);
                          optional int64 ValueInt (INTEGER(64,true));
                          optional double ValueDouble;
                          optional boolean ValueBool;
                          optional binary ValueKVList (STRING);
                          optional binary ValueArray (STRING);
                        }
                      }
                    }
                    required int32 DroppedAttributesCount (INTEGER(32,true));
                    required group Events (LIST) {
                      repeated group list {
                        required group element {
                          required int64 TimeUnixNano (INTEGER(64,false));
                          required binary Name (STRING);
                          required group Attrs (LIST) {
                            repeated group list {
                              required group element {
                                required binary Key (STRING);
                                required binary Value;
                              }
                            }
                          }
                          required int32 DroppedAttributesCount (INTEGER(32,true));
                          optional binary Test (STRING);
                        }
                      }
                    }
                    required int32 DroppedEventsCount (INTEGER(32,true));
                    required binary Links;
                    required int32 DroppedLinksCount (INTEGER(32,true));
                    optional binary HttpMethod (STRING);
                    optional binary HttpUrl (STRING);
                    optional int64 HttpStatusCode (INTEGER(64,true));
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  required binary TraceIDText (STRING);
  required int64 StartTimeUnixNano (INTEGER(64,false));
  required int64 EndTimeUnixNano (INTEGER(64,false));
  required int64 DurationNano (INTEGER(64,false));
  required binary RootServiceName (STRING);
  required binary RootSpanName (STRING);
}

Trace-level attributes

For speed and ease-of-use, we are projecting several values to columns at the trace-level:

Trace ID - Don’t store on each span.
Root service/span names/StartTimeUnixNano - These are selected properties of the root span in each trace (if there is one). These are used for displaying results in the Grafana UI. These properties are computed at ingest time and stored once for efficiency, so we don’t have to find the root span.
DurationNanos - The total trace duration, computed at ingest time. This powers the min/max duration filtering in the current Tempo search and is more efficient than scanning the spans duration column. However, it may go away with TraceQL or we could decide to change it to span-level duration filtering too.

Dedicated columns

Projecting attributes to their own columns has benefits for speed and size. Therefore we are taking an opinionated approach and projecting some common attributes to their own columns. All other attributes are stored in the generic key/value maps and are still searchable, but not as quickly. We chose these attributes based on what we commonly use ourselves (scratching our own itch), but we think they will be useful to most workloads.

Resource-level attributes include the following:

service.name
cluster and k8s.cluster.name
namespace and k8s.namespace.name
pod and k8s.pod.name
container and k8s.container.name

Span-level attributes include the following:

http.method
http.url
http.status_code (int)

“Any”-type Attributes

OTLP attributes have variable data types, which is easy to accomplish in formats like protocol-buffers, but does not translate directly to Parquet. Each column must have a concrete type. There are several possibilities here but we chose to have optional values for each concrete type. Array and KeyValueList types are stored as protocol-buffer-encoded byte arrays.

repeated group Attrs {
    required binary  Key (STRING);

    # Only one of these will be set
    optional binary  Value (STRING);
    optional boolean ValueBool;
    optional double  ValueDouble;
    optional int64   ValueInt (INT(64,true));
    optional binary  ValueArray (STRING);
    optional binary  ValueKVList (STRING);
}

Event attributes

Event attributes are stored as protocol-buffer encoded.

repeated group Attrs {
    required binary Key (STRING);
    required binary Value (STRING);
}

Compression and encoding

Parquet has robust support for many compression algorithms and data encodings. We have found excellent combinations of storage size and performance with the following:

Snappy Compression - Enable on all columns
Dictionary encoding - Enable on all string columns (including byte array ParentSpanID). Most strings are very repetitive so this works well to optimize storage size. However we can greatly speed up search by inspecting the dictionary first and eliminating pages with no matches.
Time and duration unix nanos - Delta encoding
Rarely used columns such as DroppedAttributesCount - These columns are usually all zeroes, RLE works well.

Bloom filters

Parquet has native support for bloom filters. However, Tempo does not use them at this time. Tempo already has sophisticated support for sharding and caching bloom filters.

Tune search performance

Fri, 03 Apr 2026 12:35:46 -0500

Tune search performance

Regardless of whether or not you are using TraceQL or the original search API, Tempo will search all of the blocks in the specified time range. Depending on your volume, this may result in slow queries. This document contains suggestions for tuning your backend to improve performance.

General advice is to scale your compactors and queriers. Additional queriers can more effectively run jobs in parallel while additional compactors will more aggressively reduce the length of your blocklist and copies of data (if using RF=3).

Note: All forms of search (TraceQL and tags based) are only supported on the vParquet and forward blocks. v2 blocks can only be used for trace by id lookup.

General guidelines

Tuning the search pipeline can be difficult as it requires balancing a number of different configuration parameters. The below tips can you get your head around the general problem, but the specifics require experimentation.

Review the query-frontend logs for lines like the following to get a feeling for how many jobs your queries are creating:

level=info ts=2023-07-19T19:38:01.354220385Z caller=searchsharding.go:236 msg="sharded search query request stats and SearchMetrics" ...

For a single TraceQL query the maximum number of parallel jobs is constrained by:
- query_frontend.search.concurrent_jobs: This is the maximum number of jobs the frontend will dispatch for one TraceQL query.
- # queriers * querier.max_concurrent_queries * query_frontend.max_batch_size: This is the maximum job capacity of your Tempo cluster. If a given TraceQL query produces less jobs then these two values it should be executed entirely in parallel on the queriers.
Increasing querier.max_concurrent_queries is a great way to get more out of your queriers. However, if queriers are OOMing or saturating other resources then this should be lowered. Lowering query_frontend.max_batch_size will also reduce the total work attempted by one querier.

Configuration

Queriers and query frontends have additional configuration related to search of the backend datastore.

Querier

querier:
  # Control the amount of work each querier will attempt. The total number of
  # jobs a querier will attempt this is this value * query_frontend.max_batch_size
  max_concurrent_queries: 20

With serverless technologies:

Note: Serverless can be a nice way to reduce cost by using it as spare query capacity. However, serverless tends to have higher variance then simply allowing the queriers to perform the searches themselves.

querier:

  search:
    # A list of endpoints to query. Load will be spread evenly across
    # these multiple serverless functions.
    external_endpoints:
    - https://<serverless endpoint>

    # If set to a non-zero value a second request will be issued at the provided duration. Recommended to
    # be set to p99 of search requests to reduce long tail latency.
    external_hedge_requests_at: 8s

    # The maximum number of requests to execute when hedging. Requires hedge_requests_at to be set.
    external_hedge_requests_up_to: 2

Query frontend

Query frontend lists all configuration options.

These suggestions will help deal with scaling issues.

server:
  # At larger scales, searching starts to feel more like a batch job.
  # Increase the server timeout intervals.
  http_server_read_timeout: 2m
  http_server_write_timeout: 2m

query_frontend:
  # When increasing concurrent_jobs, also increase the queue size per tenant,
  # or search requests will be cause 429 errors. This is the total number of jobs
  # per tenant allowed in the queue.
  max_outstanding_per_tenant: 2000

  # The number of jobs the query-frontend will batch together when passing jobs to the queriers. This value
  # This value * querier.max_concurrent_queries is your the max number of jobs a given querier will try at once.
  max_batch_size: 3

  search:
    # At larger scales, increase the number of jobs attempted simultaneously,
    # per search query.
    concurrent_jobs: 2000

    # The query frontend will attempt to divide jobs up by an estimate of job size. The smallest possible
    # job size is a single parquet row group. Increasing this value will create fewer, larger jobs. Decreasing
    # it will create more, smaller jobs.
    target_bytes_per_job: 50_000_000

Serverless environment

Serverless is not required, but with larger loads, serverless can be used to reduce costs. Tempo has support for Google Cloud Run and AWS Lambda. In both cases, you will use the following settings to configure Tempo to use a serverless environment:

querier:
  search:
    # A list of external endpoints that the querier will use to offload backend search requests. They must
    # take and return the same value as /api/search endpoint on the querier. This is intended to be
    # used with serverless technologies for massive parallelization of the search path.
    # The default value of "" disables this feature.
    [external_endpoints: <list of strings> | default = <empty list>]

    # If external_endpoints is set then the querier will primarily act as a proxy for whatever serverless backend
    # you have configured. This setting allows the operator to have the querier prefer itself for a configurable
    # number of subqueries. In the default case of 2 the querier will process up to 2 search requests subqueries before starting
    # to reach out to external_endpoints.
    # Setting this to 0 will disable this feature and the querier will proxy all search subqueries to external_endpoints.
    [prefer_self: <int> | default = 2 ]

    # If set to a non-zero value a second request will be issued at the provided duration. Recommended to
    # be set to p99 of external search requests to reduce long tail latency.
    # (default: 4s)
    [external_hedge_requests_at: <duration>]

    # The maximum number of requests to execute when hedging. Requires hedge_requests_at to be set.
    # (default: 3)
    [external_hedge_requests_up_to: <int>]

See here for cloud-specific details:

Caching

If you have set up an external cache (redis or memcached) in in your storage block you can also use it to cache parquet footers using the following configuration:

storage:
  trace:
    search:
      cache_control:
        footer: true

Search with Google Cloud Run

Fri, 03 Apr 2026 12:35:46 -0500

Search with Google Cloud Run

This document walks you through setting up a Google Cloud Run for serverless backend search. For more guidance on configuration options for full backend search check here.

Build the docker image:

cd ./cmd/tempo-serverless && make build-docker-gcr

This will create the Docker container image to be deployed to Google Cloud Run. The docker image will be named: tempo-serverless:latest and tempo-serverless:<branch>-<commit hash>. Here is an example of that name:

$ docker images | grep tempo-serverless
tempo-serverless                                                           cloud-run-3be4efa               146c9d9fa63c   58 seconds ago   47.9MB
tempo-serverless                                                           latest                          146c9d9fa63c   58 seconds ago   47.9MB

Push the image to a Google Container Registry repo.

Provision the Google Cloud Run service. This example uses Terraform. Configuration values should be adjusted to meet the needs of your installation.

locals {
  // this can be increased if you would like to use multiple functions
  count = 1
}

resource "google_cloud_run_service" "run" {
  count = local.count

  name     = "<service name>"
  location = "<appropriate region>"

  metadata {
    annotations = {
        "run.googleapis.com/ingress"      = "internal",     # this annotation can be used to limit connectivity to the service
    }
  }

  template {
    metadata {
      annotations = {
          "autoscaling.knative.dev/minScale"                   = "1",
          "autoscaling.knative.dev/maxScale"                   = "1000",
          "autoscaling.knative.dev/panic-threshold-percentage" = "110.0",  # default 200.0. how aggressively to go into panic mode and start scaling heavily
          "autoscaling.knative.dev/window"                     = "10s",    # default 60s. window over which to average metrics to make scaling decisions
      }
    }
    spec {
      container_concurrency = 4
      containers {
        image = "<container image created above>"
        resources {
          limits = {
              cpu = "2"
              memory = "1Gi"
          }
        }
        env {
          name = "TEMPO_GCS_BUCKET_NAME"
          value = "<gcs bucket where tempo data is stored>"
        }
        env {
          name = "TEMPO_BACKEND"
          value = "gcs"
        }
        env {
          name = "TEMPO_GCS_HEDGE_REQUESTS_AT"
          value = "400ms"
        }
        env {
          name = "TEMPO_GCS_HEDGE_REQUESTS_UP_TO"
          value = "2"
        }
        env {
          name = "GOGC"
          value = "400"
        }
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }
}

Add the newly-created cloud run service as external endpoints in your querier configuration. The endpoint can be retrieved from the Details tab in Google Cloud Run:
```
querier:
  search:
    external_endpoints:
    - <trigger url from console>
```

Search with AWS Lambda

Fri, 03 Apr 2026 12:35:46 -0500

Search with AWS Lambda

This document explains how to set up AWS Lambda for serverless backend search. Read improve search performance for more guidance on configuration options for backend search.

Build the code package:
Bash
```
cd ./cmd/tempo-serverless && make build-lambda-zip
```
This will create a ZIP file containing the binary required for the function. The file name will be of the form: ./lambda/tempo-<branch name>-<commit hash>.zip. Here is an example of that name:
Bash
```
ls lambda/*.zip
lambda/tempo-serverless-backend-search-297172a.zip
```
Provision an S3 bucket.

Copy the ZIP file into your bucket.

aws s3 cp lambda/tempo-serverless-backend-search-297172a.zip gs://<newly provisioned gcs bucket>

Provision the Lambda. For a Lambda function to be invoked via HTTP, we also need to create an ALB and some other resources. This example uses Terraform and only includes the function definition. Additionally, you will need a VPC, security groups, IAM roles, an ALB, target groups, etc., but that is beyond the scope of this guide.

locals {
  // this can be increased if you would like to use multiple functions
  count = 1
}

resource "aws_lambda_function" "lambda" {
    count = local.count

    function_name    = "${local.name}-${count.index}"
    description      = "${local.name}-${count.index}"
    role             = <arn of appropriate role>
    handler          = "main"
    runtime          = "go1.x"
    timeout          = "60"
    s3_key           = "tempo-serverless-backend-search-297172a.zip"
    s3_bucket        = <S3 bucket created above>
    memory_size      = 1769 # 1 vcpu

    vpc_config {
        subnet_ids         = <appropriate subnets>
        security_group_ids = [<appropriate security groups>]
    }

    environment {
        variables = {
        "TEMPO_S3_BUCKET"               = "<S3 bucket name backing your Tempo instance>"
        "TEMPO_BACKEND"                 = "s3"
        "TEMPO_S3_HEDGE_REQUESTS_AT"    = "400ms"
        "TEMPO_S3_HEDGE_REQUESTS_UP_TO" = "2"
        "TEMPO_S3_ENDPOINT"             = "s3.dualstack.us-east-2.amazonaws.com"
        }
    }
}

Add the hostname of the newly-created ALB to the querier config:

querier:
  search:
    external_endpoints:
    - http://<alb dns hostname>

Generic forwarding

Fri, 03 Apr 2026 12:35:46 -0500

Generic forwarding

Generic forwarding allows asynchronous replication of ingested traces. The distributor writes received spans to both the ingester and defined endpoints, if enabled. This feature works in a “best-effort” manner, meaning that no retries happen if an error occurs during replication.

Warning: Generic forwarding does not work retroactively. Once enabled, the distributor only replicates freshly ingested spans.

Configure generic forwarding

Enabling generic forwarding requires the configuration of the distributor and overrides.

First, define a list of forwarders in the distributor section. Each forwarder must specify a unique name, supported backend, and backend-specific configuration.
Second, reference these forwarders in the overrides section. This allows for fine-grained control over forwarding and makes it possible to enable this feature globally or on a per-tenant basis.

For a detailed view of all the config options for the generic forwarding feature, please refer to distributor and overrides configuration pages.