Set up Tempo on Grafana Labs

Plan your Tempo deployment

Fri, 03 Apr 2026 12:35:46 -0500

Plan your Tempo deployment

Tempo can be deployed in monolithic or microservices modes.

The deployment mode is determined by the runtime configuration target, or by using the -target flag on the command line. The default target is all, which is the monolithic deployment mode.

Note
Monolithic mode was previously called single binary mode. Similarly scalable monolithic mode was previously called scalable single binary mode. While the documentation has been updated to reflect this change, some URL names and deployment tooling (for example, Helm charts) do not yet reflect this change.

Monolithic mode

Monolithic mode deployment runs all top-level components in a single process, forming an instance of Tempo. The monolithic mode is the simplest to deploy, but can not horizontally scale out by increasing the quantity of components. Refer to Architecture for descriptions of the components.

To enable this mode, -target=all is used, which is the default.

Find docker-compose deployment examples at:

Scaling monolithic mode

Monolithic mode can be horizontally scaled out. This scalable monolithic mode is similar to the monolithic mode in that all components are run within one process. Horizontal scale out is achieved by instantiating more than one process, with each having -target set to scalable-single-binary.

This mode offers some flexibility of scaling without the configuration complexity of the full microservices deployment.

Each of the queriers perform a DNS lookup for the frontend_address and connect to the addresses found within the DNS record.

Find a docker-compose deployment example at:

https://github.com/grafana/tempo/tree/main/example/docker-compose/scalable-single-binary

Microservices mode

In microservices mode, components are deployed in distinct processes. Scaling is per component, which allows for greater flexibility in scaling and more granular failure domains. This is the preferred method for a production deployment, but it is also the most complex.

The configuration associated with each component’s deployment specifies a target. For example, to deploy a querier, the configuration would contain target: querier. A command-line deployment may specify the -target=querier flag. Each of the components referenced in Architecture must be deployed in order to get a working Tempo instance.

Find a docker-compose deployment example at:

https://github.com/grafana/tempo/tree/main/example/docker-compose/distributed

Tools used to deploy Tempo

Tempo can be easily deployed through a number of tools, including Helm, Tanka, Kubernetes, and Docker.

Note
The Tanka and Helm examples are equivalent. They are both provided for people who prefer different configuration mechanisms.

Helm

Helm charts are available in the grafana/helm-charts repository:

In addition, several Helm chart examples are available in the Tempo repository.

Kubernetes Tempo Operator

The operator is available in grafana/tempo-operator repository. The operator reconciles TempoStack resource to deploy and manage Tempo microservices installation.

Refer to the operator documentation for more details.

Tanka/Jsonnet

The Jsonnet files that you need to deploy Tempo with Tanka are available here:

Here are a few examples that use official Jsonnet files. They display the full range of configurations available to Tempo.

Kubernetes manifests

You can find a collection of Kubernetes manifests to deploy Tempo in the operations/jsonnet-compiled folder. These are generated using the Tanka/Jsonnet.

Upgrade your Tempo installation

Fri, 03 Apr 2026 12:35:46 -0500

Upgrade your Tempo installation

You can upgrade an existing Tempo installation to the next version. However, any new release has the potential to have breaking changes that should be tested in a non-production environment prior to rolling these changes to production.

The upgrade process changes for each version, depending upon the changes made for the subsequent release.

This upgrade guide applies to on-premise installations and not for Grafana Cloud.

TIP: You can check your configuration options using the status API endpoint in your Tempo installation.

Upgrade to Tempo 2.2

Tempo 2.2 has several considerations for any upgrade:

vParquet2 is now the default block format
Several configuration parameters have been renamed or removed.

For a complete list of changes, enhancements, and bug fixes, refer to the Tempo 2.2 changelog.

Default block format changed to vParquet2

While not a breaking change, upgrading to Tempo 2.2 will by default change Tempo’s block format to vParquet2.

To stay on a previous block format, read theParquet configuration documentation. We strongly encourage upgrading to vParquet2 as soon as possible as this is required for using structural operators in your TraceQL queries and provides query performance improvements, in particular on queries using the duration intrinsic.

Updated JSonnet supports `statefulset` for the metrics-generator

Tempo 2.2 updates the microservices JSonnet to support a statefulset for the metrics_generator component.

Note
This update is important if you use the experimental local-blocks processor.

To support a new processor, the metrics-generator has been converted from a deployment into a statefulset with a PVC. This requires manual intervention to migrate successfully and avoid downtime. Note that currently both a deployment and a statefulset will be managed by the JSonnet for a period of time, after which we will delete the deployment from this repo and you will need to delete user-side references to the tempo_metrics_generator_deployment, as well as delete the deployment itself.

Refer to the PR for seamless migration instructions. [PRs 2533, 2467]

Removed or renamed configuration parameters

The following fields were removed or renamed.

Parameter	Comments
query_frontend: tolerate_failed_blocks:	Remove support for `tolerant_failed_blocks` [PR 2416]
storage: trace: s3: insecure_skip_verify: true // renamed to tls_insecure_skip_verify	Renamed `insecure_skip_verify` to `tls_insecure_skip_verify` [PR 2407]

Upgrade to Tempo 2.1

Tempo 2.1 has two major considerations for any upgrade:

Support for search on v2 block is removed
Breaking changes to metric names

For more information on other enhancements, read the Tempo 2.1 release notes.

Remove support for Search on v2 blocks

Users can no longer search blocks in v2 format. Only vParquet and vParquet2 formats support search. The following search configuration options were removed from the overrides section:

overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:

The following metrics configuration was also removed:

tempo_ingester_trace_search_bytes_discarded_total

Upgrade path to maintain search from Tempo 1.x to 2.1

Removing support for search on v2 blocks means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks. To avoid this, upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks. You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming traces. Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1. All of your blocks will be searchable.

Parquet files are no longer cached when carrying out searches.

Breaking changes to metric names exposed by Tempo

All Prometheus metrics exposed by Tempo on its /metrics endpoint that were previously prefixed with cortex_ have now been renamed to be prefixed with tempo_ instead. (PR 2204)

Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR 2008)

The query_frontend_result_metrics_inspected_bytes metric was removed in favor of ``query_frontend_bytes_processed_per_second.

Upgrade from Tempo 1.5 to 2.0

Tempo 2.0 marks a major milestone in Tempo’s development. When planning your upgrade, consider these factors:

Breaking changes:
- Renamed, removed, and moved configurations are described in section below.
- The TempoRequestErrors alert was removed from mixin. Any Jsonnet users relying on this alert should copy this into their own environment.
Advisory:
- Changed defaults – Are these updates relevant for your installation?
- TraceQL editor needs to be enabled in Grafana to use the query editor.
- Resource requirements have changed for Tempo 2.0 with the default configuration.

Once you upgrade to Tempo 2.0, there is no path to downgrade.

Note: There is a potential issue loading Tempo 1.5’s experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues please report them so we can isolate and fix this issue.

Check Tempo installation resource allocation

Parquet provides faster search and is required to enable TraceQL. However, the Tempo installation will require additional CPU and memory resources to use Parquet efficiently. Parquet is more costly due to the extra work of building the columnar blocks, and operators should expect at least 1.5x increase in required resources to run a Tempo 2.0 cluster. Most users will find these extra resources are negligible compared to the benefits that come from the additional features of TraceQL and from storing traces in an open format.

You can can continue using the previous v2 block format using the instructions provided in the Parquet configuration documentation. Tempo will continue to support trace by id lookup on the v2 format for the foreseeable future.

Enable TraceQL in Grafana

TraceQL is enabled by default in Tempo 2.0. The TraceQL query editor requires Grafana 9.3.2 and later.

The TraceQL query editor is in beta in Grafana 9.3.2 and needs to be enabled with the traceqlEditor feature flag.

Check configuration options for removed and renamed options

The following tables describe the parameters that have been removed or renamed.

Removed and replaced

Parameter	Comments
query_frontend: query_shards:	Replaced by `trace_by_id.query_shards`.
querier: query_timeout:	Replaced by two different settings: `search.query_timeout` and `trace_by_id.query_timeout`.
ingester: use_flatbuffer_search:	Removed and automatically determined based on block format.
`search_enabled`	Removed. Now defaults to true.
`metrics_generator_enabled`	Removed. Now defaults to true.

Renamed

The following compactor configuration parameters were renamed.

Parameter	Comments
compaction: chunk_size_bytes:	Renamed to `v2_in_buffer_bytes`
compaction: flush_size_bytes:	Renamed to `v2_out_buffer_bytes`
compaction: iterator_buffer_size:	Renamed to `v2_prefetch_traces_count`

The following storage configuration parameters were renamed.

Parameter	Comments
wal: encoding:	Renamed to `v2_encoding`
block: index_downsample_bytes:	Renamed to `v2_index_downsample_bytes`
block: index_page_size_bytes:	Renamed to `v2_index_page_size_bytes`
block: encoding:	Renamed to `v2_encoding`
block: row_group_size_bytes:	Renamed to `parquet_row_group_size_bytes`

The Azure Storage configuration section now uses snake case with underscores (_) instead of dashes (-). Example of using snake case on Azure Storage config:

# config.yaml
storage:
  trace:
    azure:
      storage_account_name:
      storage_account_key:
      container_name:

Deploy Tempo with Helm

Fri, 03 Apr 2026 12:35:46 -0500

Deploy Tempo with Helm

The Helm charts for Grafana Tempo and Grafana Enterprise Traces allows you to configure, install, and upgrade Grafana Tempo or Grafana Enterprise Traces within a Kubernetes cluster.

Example Helm chart

The Tempo repository has an example Helm chart that shows a complete microservice-based deployment.

Helm charts for deployment

Tempo has two primary charts used for deployment:

tempo-distributed Helm chart deploys Tempo in microservices mode
tempo Helm chart deploys Tempo in monolithic (single binary) mode

To deploy Tempo using the tempo-distributed Helm chart, read the Get started with Grafana Tempo using Helm.

Deploy Tempo with Tempo Operator

Fri, 03 Apr 2026 12:35:46 -0500

Deploy Tempo with Tempo Operator

The Tempo Operator allows you to configure, install, upgrade, and operate Grafana Tempo on Kubernetes and OpenShift clusters.

Some of the operator features are:

Resource Limits - Specify overall resource requests and limits in the TempoStack CR; the operator assigns fractions of it to each component
AuthN and AuthZ - Supports OpenID Control (OIDC) and role-based access control (RBAC)
Multitenancy - Multiple tenants can send traces to the same Tempo cluster
mTLS - Communication between the Tempo components can be secured via mTLS
Jaeger UI - Traces can be visualized in Jaeger UI and exposed via Ingress or OpenShift Route
Observability - The operator and TempoStack operands expose telemetry (metrics, traces) and integrate with Prometheus ServiceMonitors

The source of the Tempo Operator can be found at grafana/tempo-operator.

Installation

The operator can be installed from:

Kubernetes manifest file on a Kubernetes cluster
operatorhub.io on a Kubernetes cluster
OperatorHub on an OpenShift cluster

Compatibility

Tempo

The supported Tempo version by the operator can be found in the changelog or on the release page.

Kubernetes

The Tempo Operator is supported on Kubernetes 1.21 and 1.27.

cert-manager

The operator Kubernetes manifest installation files use cert-manger v1 custom resources to provision certificates for admission webhooks.

Community

Reach out to us on #tempo-operator Grafana Slack channel.
Participate on Tempo community call.

Deploy on Linux

Fri, 03 Apr 2026 12:35:46 -0500

Deploy on Linux

This guide provides a step-by-step process for installing Tempo on Linux. It assumes you have access to a Linux system and the permissions required to deploy a service with network and file system access. At the end of this guide, you will have deployed a single Tempo instance on a single node.

These instructions focus on a monolithic installation. You can also run Tempo in distributed mode by deploying multiple binaries and using a distributed configuration.

Before you begin

To follow this guide, you need:

A running Grafana instance (see installation instructions)
An Amazon S3 compatible object store
Git, Docker, and docker-compose plugin installed to test Tempo

System requirements

This configuration is an example you can use as a starting point. You may need to have more resources for your system than the minimum specifications listed below. Additional adjustments will be necessary for a production environment.

You must have the permissions required to deploy a service with a network and file system access.

Your Linux system should have at least:

4 CPUs
16 GB of memory

Set up an object storage bucket

Tempo uses object storage as the backend for its trace storage. It also uses object storage for storing various data related to the state of the system.

Tempo supports using the local filesystem as the backend for trace storage as well. This is not recommended for production deployments. This guide focuses on setup with an object storage.

This example uses Amazon S3 on the AWS us-east-1 region as your object store. If you plan on using a different region or object storage service, update the storage fields in the configuration file below. Currently, the supported object storage backends are AWS S3, other S3-compliant object stores, and Google Cloud’s GCS.

After you have provisioned an object storage backend, create the bucket grafana-traces-data. The buckets will be referenced in the configuration file of this guide. You may need to alter the bucket name to be globally unique.

Consider adding a prefix for your organization to the bucket, for example, myorg-grafana-traces-data, and then replacing the names in the rest of these instructions with those bucket names.

Install Tempo

For a linux-amd64 installation, run the following commands via the command line interface on your Linux machine. You need administrator privileges to do this by running as the root user or via sudo as a user with permissions to do so.

Download the Tempo binary, verify checksums (listed in SHA256SUMS), and add network capabilities to the binary. Be sure to download the correct package installation for your OS and architecture:

curl -Lo tempo_2.2.0_linux_amd64.deb https://github.com/grafana/tempo/releases/download/v2.2.0/tempo_2.2.0_linux_amd64.deb
echo e81cb4ae47e1d8069efaad400df15547e809b849cbb18932e23ac3082995535b \
  tempo_2.2.0_linux_amd64.deb | sha256sum -c
dpkg -i tempo_2.2.0_linux_amd64.deb

Create a Tempo configuration file

Copy the following YAML configuration to a file called tempo.yaml.

Paste in your S3 credentials for admin_client and the storage backend. If you wish to give your cluster a unique name, add a cluster property with the appropriate name.

Refer to the Tempo configuration documentation for explanations of the available options.

In the following configuration, Tempo options are altered to only listen to the OTLP gRPC and HTTP protocols. By default, Tempo listens for all compatible protocols.

server:
  http_listen_port: 3200

distributor:
  receivers:
      otlp:
        protocols:
          http:
          grpc:

compactor:
  compaction:
    block_retention: 48h                # configure total trace retention here

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: linux-microservices
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
    - url: http://localhost:9090/api/v1/write
      send_exemplars: true

storage:
  trace:
    backend: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      bucket: grafana-traces-data
      forcepathstyle: true
      # set to false if endpoint is https
      insecure: true
      access_key: # TODO - Add S3 access key
      secret_key: # TODO - Add S3 secret key
    wal:
      path: /tmp/tempo/wal         # where to store the the wal locally
    local:
      path: /tmp/tempo/blocks
overrides:
  metrics_generator_processors: [service-graphs, span-metrics]

Note: In the above configuration, metrics generator is enabled to generate Prometheus metrics data from incoming trace spans. This is sent to a Prometheus remote write compatible metrics store at http://prometheus:9090/api/v1/write (in the metrics_generator configuration block). Ensure you change the relevant url parameter to your own Prometheus compatible storage instance, or disable the metrics generator by removing the metrics_generators_processors if you do not wish to generate span metrics.

Move the configuration file to the proper directory

Copy the tempo.yaml to /etc/tempo/config.yml:

cp tempo.yaml /etc/tempo/config.yml

Restart the tempo service

Use systemctl to restart the service (depending on how you installed Tempo, this may be different):

systemctl start tempo.service

You can replace restart with stop to stop the service, and start to start the service again after it’s stopped, if required.

Verify your cluster is working

To verify that Tempo is working, run the following command:

systemctl is-active tempo

You should see the status active returned. If you do not, check that the configuration file is correct, and then restart the service. You can also use journalctl -u tempo to view the logs for Tempo to determine if there are any obvious reasons for failure to start.

Verify that your storage bucket has received data by signing in to your storage provider and determining that a file has been written to storage. It should be called tempo_cluster_seed.json.

Test your installation

Once Tempo is running, you can use the K6 with Traces Docker example to verify that trace data is sent to Tempo. This procedure sets up a sample data source in Grafana to read from Tempo.

Backend storage configuration

The Tempo examples running with docker-compose all include a version of Tempo and a storage backend like S3 and GCS. Because Tempo is installed with a backend storage configured, you need to change the docker-compose.yaml file to remove Tempo and instead point trace storage to the installed version. These steps are included in this section.

Network configuration

Docker compose uses an internal networking bridge to connect all of the defined services. Because the Tempo instance is running as a service on the local machine host, you need the resolvable IP address of the local machine so the docker containers can use the Tempo service. You can find the host IP address of your Linux machine using a command such as ip addr show.

Steps

Clone the Tempo repository:

git clone https://github.com/grafana/tempo.git

Go into the examples directory:
```
cd tempo/example/docker-compose/local
```
Edit the file docker-compose.yaml, and remove the tempo service and all its properties, so that the first service defined is k6-tracing. The start of your docker-compose.yaml should look like this:
```
version: "3"
services:

k6-tracing:
```
Edit the k6-tracing service, and change the value of ENDPOINT to the local IP address of the machine running Tempo and docker compose, eg. 10.128.0.104:4317. This is the OTLP gRPC port:
```
environment:
  - ENDPOINT=10.128.0.104:4317
```
This ensures that the traces sent from the example application go to the locally running Tempo service on the Linux machine.
Edit the k6-tracing service and remove the dependency on Tempo by deleting the following lines:
```
depends_on:
tempo
```
Save the docker-compose.yaml file and exit your editor.
Edit the default Grafana data source for Tempo that is included in the examples. Edit the file located at tempo/example/shared/grafana-datasources.yaml, and change the url field of the Tempo data source to point to the local IP address of the machine running the Tempo service instead (eg. url: http://10.128.0.104:3200). The Tempo data source section should resemble this:
```
- name: Tempo
  type: tempo
  access: proxy
  orgId: 1
  url: http://10.128.0.104:3200
```
Save the file and exit your editor.
Edit the Prometheus configuration file so it uses the Tempo service as a scrape target. Change the target to the local Linux host IP address. Edit the tempo/example/shared/prometheus.yaml file, and alter the tempo job to replace tempo:3200 with the Linux machine host IP address.
```
  - job_name: 'tempo'
	static_configs:
  	- targets: [ '10.128.0.104:3200' ]
```
Save the file and exit your editor.**
Start the three services that are defined in the docker-compose file:
```
docker compose up -d
```

Verify that the services are running using docker compose ps. You should see something like:

NAME             	IMAGE                                   	COMMAND              	SERVICE         	CREATED         	STATUS          	PORTS
local-grafana-1  	grafana/grafana:9.3.2                   	"/run.sh"            	grafana         	2 minutes ago   	Up 3 seconds    	0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
local-k6-tracing-1   ghcr.io/grafana/xk6-client-tracing:v0.0.2   "/k6-tracing run /ex…"   k6-tracing      	2 minutes ago   	Up 2 seconds
local-prometheus-1   prom/prometheus:latest                  	"/bin/prometheus --c…"   prometheus      	2 minutes ago   	Up 2 seconds    	0.0.0.0:9090->9090/tcp, :::9090->9090/tcp

Grafana is running on port 3000, Prometheus is running on port 9090. Both should be bound to the host machine.

As part of the docker compose manifest, Grafana is now running on your Linux machine, reachable on port 3000. Point your web browser to the Linux machine on port 3000. You might need to port forward the local port if you’re doing this remotely, for example, via SSH forwarding.
Once logged in, navigate to the Explore page, select the Tempo data source and select the Search tab. Select Run query to list the recent traces stored in Tempo. Select one to view the trace diagram:
Alter the Tempo configuration to point to the instance of Prometheus running in docker compose. To do so, edit the configuration at /etc/tempo/config.yaml and change the storage block under the metrics_generator section so that the remote write url is http://localhost:9090. The configuration section should look like this:
```
 storage:
     path: /tmp/tempo/generator/wal
     remote_write:
        - url: http://localhost:9090/api/v1/write
        send_exemplars: true
```
Save the file and exit the editor.
Finally, restart the Tempo service by running:
```
sudo systemctl restart tempo
```
A couple of minutes after Tempo has successfully restarted, select the Service graph tab for the Tempo data source in the Explore page. Select Run query to view a service graph, generated by Tempo’s metrics-generator.

Deploy on Kubernetes with Tanka

Fri, 03 Apr 2026 12:35:46 -0500

Deploy on Kubernetes with Tanka

Using this deployment guide, you can deploy Tempo to Kubernetes using a Jsonnet library and Grafana Tanka to create a development cluster or sand-boxed environment. This procedure uses MinIO to provide object storage regardless of the cloud platform or on-premise storage you use. In a production environment, you can use your cloud provider’s object storage service to avoid the operational overhead of running object storage in production.

To set up Tempo using Kubernetes with Tanka, you need to:

Configure Kubernetes and install Tanka
Set up the Tanka environment
Install libraries
Deploy MinIO object storage
Optional: Enable metrics-generator
Deploy Tempo with the Tanka command

Note
This configuration is not suitable for a production environment but can provide a useful way to learn about Tempo.

Before you begin

To deploy Tempo to Kubernetes with Tanka, you need:

A Kubernetes cluster with at least 40 CPUs and 46GB of memory for the default configuration. Small ingest or query volumes could use a far smaller configuration.
kubectl (version depends upon the API version of Kubernetes in your cluster)

Configure Kubernetes and install Tanka

Follow these steps to configure Kubernetes and install Tanka.

Create a new directory for the installation, and make it your current working directory:
Bash
```
mkdir tempo
cd tempo
```
Create a Kubernetes namespace. You can replace the namespacetempo in this example with a name of your choice.
Bash
```
kubectl create namespace tempo
```
Install Grafana Tanka; refer to Installing Tanka.
Install jsonnet-bundler; refer to the jsonnet-bundler README.

Set up the Tanka environment

Tanka requires the current context for your Kubernetes environment.

Check the current context for your Kubernetes cluster and ensure it’s correct:
Bash
```
kubectl config current-context
```

Initialize Tanka. This will use the current Kubernetes context:

tk init --k8s=false
tk env add environments/tempo
tk env set environments/tempo \
 --namespace=tempo \
 --server-from-context=$(kubectl config current-context)

Install libraries

Install the k.libsonnet, Jsonnet, and Memcachd libraries.

Install k.libsonnet for your version of Kubernetes:

mkdir -p lib
export K8S_VERSION=1.25
jb install github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}@main
cat <<EOF > lib/k.libsonnet
import 'github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}/main.libsonnet'
EOF

Install the Tempo Jsonnet library and its dependencies.

jb install github.com/grafana/tempo/operations/jsonnet/microservices@main

Install the Memcached library and its dependencies.

jb install github.com/grafana/jsonnet-libs/memcached@master

Deploy MinIO object storage

MinIO is an open source Amazon S3-compatible object storage service that is freely available and easy to run on Kubernetes.

Create a file named minio.yaml and copy the following YAML configuration into it. You may need to remove/modify the storageClassName depending on your Kubernetes platform. GKE, for example, may not support local-path name but may support another option such as standard.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            - "sh"
            - "-c"
            - "mkdir -p /storage/tempo-data"
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
            - --console-address
            - ":9001"
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
            - containerPort: 9001
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: api
    - port: 9001
      targetPort: 9001
      protocol: TCP
      name: console
  selector:
    app: minio

Run the following command to apply the minio.yaml file:
Bash
```
kubectl apply --namespace tempo -f minio.yaml
```
To check that MinIO is correctly configured, sign in to MinIO and verify that a bucket has been created. Without these buckets, no data will be stored.
1. Port-forward MinIO to port 9001:
  Bash
```
 kubectl port-forward --namespace tempo service/minio 9001:9001
```
2. Navigate to the MinIO admin bash using your browser: http://localhost:9001. The sign-in credentials are username minio and password minio123.
3. Verify that the Buckets page lists tempo-data.

Configure the Tempo cluster using the MinIO object storage by updating the contents of the environments/tempo/main.jsonnet file by running the following command:

cat <<EOF > environments/tempo/main.jsonnet
// The jsonnet file used to generate the Kubernetes manifests.
local tempo = import 'microservices/tempo.libsonnet';
local k = import 'ksonnet-util/kausal.libsonnet';
local container = k.core.v1.container;
local containerPort = k.core.v1.containerPort;

tempo {
    _images+:: {
        tempo: 'grafana/tempo:latest',
        tempo_query: 'grafana/tempo-query:latest',
    },

    tempo_distributor_container+:: container.withPorts([
            containerPort.new('jaeger-grpc', 14250),
            containerPort.new('otlp-grpc', 4317),
        ]),

    _config+:: {
        namespace: 'tempo',

        compactor+: {
            replicas: 1,
        },
        query_frontend+: {
            replicas: 2,
        },
        querier+: {
            replicas: 3,
        },
        ingester+: {
            replicas: 3,
            pvc_size: '10Gi',
            pvc_storage_class: 'standard',
        },
        distributor+: {
            replicas: 3,
            receivers: {
                jaeger: {
                    protocols: {
                        grpc: {
                            endpoint: '0.0.0.0:14250',
                        },
                    },
                },
                otlp: {
                    protocols: {
                        grpc: {
                            endpoint: '0.0.0.0:4317',
                        },
                    },
                },
            },
        },

        metrics_generator+: {
            replicas: 1,
            ephemeral_storage_request_size: '10Gi',
            ephemeral_storage_limit_size: '11Gi',
            pvc_size: '10Gi',
            pvc_storage_class: 'standard',
        },
        memcached+: {
            replicas: 3,
        },

        bucket: 'tempo-data',
        backend: 's3',
    },

    tempo_config+:: {
        storage+: {
            trace+: {
                s3: {
                    bucket: $._config.bucket,
                    access_key: 'minio',
                    secret_key: 'minio123',
                    endpoint: 'minio:9000',
                    insecure: true,
                },
            },
        },
        metrics_generator+: {
            processor: {
                span_metrics: {},
                service_graphs: {},
            },

            registry+: {
                external_labels: {
                    source: 'tempo',
                },
            },
        },
        overrides+: {
            metrics_generator_processors: ['service-graphs', 'span-metrics'],
        },
    },

    tempo_ingester_container+:: {
      securityContext+: {
        runAsUser: 0,
      },
    },

    local statefulSet = $.apps.v1.statefulSet,
    tempo_ingester_statefulset+:
        statefulSet.mixin.spec.withPodManagementPolicy('Parallel'),
}
EOF

Optional: Enable metrics-generator

In the preceding configuration, metrics generation is enabled. However, you still need to specify where to send the generated metrics data. If you’d like to remote write these metrics onto a Prometheus compatible instance (such as Grafana Cloud metrics or a Mimir instance), you’ll need to include the configuration block below in the metrics_generator section of the tempo_config block above (this assumes basic auth is required, if not then remove the basic_auth section). You can find the details for your Grafana Cloud metrics instance for your Grafana Cloud account by using the Cloud Portal.

storage+: {
    remote_write: [
        {
            url: 'https://<urlForPrometheusCompatibleStore>/api/v1/write',
            send_exemplars: true,
            basic_auth: {
                username: '<username>',
                password: '<password>',
            },
        }
    ],
},

Note: Enabling metrics generation and remote writing them to Grafana Cloud Metrics will produce extra active series that could potentially impact your billing. For more information on billing, refer to Billing and usage. For more information on metrics generation, refer Metrics-generator in the Tempo documentation.

Optional: Reduce component system requirements

Smaller ingestion and query volumes could allow the use of smaller resources. If you wish to lower the resources allocated to components, then you can do this via a container configuration. For example, to change the CPU and memory resource allocation for the ingesters.

To change the resources requirements, follow these steps:

Open the environments/tempo/main.jsonnet file.

Add a new configuration block for the appropriate component (in this case, the ingesters):

tempo_ingester_container+:: {
    resources+: {
        limits+: {
            cpu: '3',
            memory: '5Gi',
        },
        requests+: {
            cpu: '200m',
            memory: '2Gi',
        },
    },
},

Save the changes to the file.

Note
Lowering these requirements can impact overall performance.

Deploy Tempo using Tanka

Deploy Tempo using the Tanka command:
Bash
```
tk apply environments/tempo/main.jsonnet
```

Note
If the ingesters don’t start after deploying Tempo with the Tanka command, this may be related to the storage class selected for the Write Ahead Logs. If this is the case, add an appropriate storage class to the ingester configuration. For example, to add a standard instead of fast storage class, add the following to the config (not tempo_config) section in the previous step:
Bash
  ingester+: {
    pvc_storage_class: 'standard',
  },

Next steps

The Tempo instance will now accept the two configured trace protocols (OTLP gRPC and Jaeger gRPC) via the distributor service at distributor.tempo.svc.cluster.local on the relevant ports:

OTLP gRPC: 4317
Jaeger gRPC: 14250

You can query Tempo using the query-frontend.tempo.svc.cluster.local service on port 3200 for Tempo queries or port 16686 or 16687 for Jaeger type queries.

Now that you’ve configured a Tempo cluster, you’ll need to get data into it. Read the Set up a test app for instructions.

Set up a test application for a Tempo cluster

Fri, 03 Apr 2026 12:35:46 -0500

Set up a test application for a Tempo cluster

Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana. This procedure uses Tempo in microservices mode. For example, if you set up Tempo using the Kubernetes with Tanka procedure, then you can use this procedure to test your set up.

Before you begin

You’ll need:

Grafana 9.0.0 or higher
Microservice deployments require the Tempo querier URL, for example: http://query-frontend.tempo.svc.cluster.local:3200
OpenTelemetry telemetrygen for generating tracing data

Refer to Deploy Grafana on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Set up `remote_write` to your Tempo cluster

To enable writes to your cluster:

Add a remote_write configuration snippet to the configuration file of an existing Grafana Agent.

If you do not have an existing traces collector, refer to Set up with Grafana Agent. For Kubernetes, refer to the Grafana Agent Traces Kubernetes quick start guide.

The example agent Kubernetes ConfigMap configuration below opens many trace receivers (note that the remote write is onto the Tempo cluster using OTLP gRPC):

kind: ConfigMap
metadata:
  name: grafana-agent-traces
apiVersion: v1
data:
  agent.yaml: |
    traces:
        configs:
          - batch:
                send_batch_size: 1000
                timeout: 5s
            name: default
            receivers:
                jaeger:
                    protocols:
                        grpc: null
                        thrift_binary: null
                        thrift_compact: null
                        thrift_http: null
                opencensus: null
                otlp:
                    protocols:
                        grpc: null
                        http: null
                zipkin: null
            remote_write:
              - endpoint: <tempoDistributorServiceEndpoint>
                insecure: true  # only add this if TLS is not used
            scrape_configs:
              - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                job_name: kubernetes-pods
                kubernetes_sd_configs:
                  - role: pod
                relabel_configs:
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                tls_config:
                    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                    insecure_skip_verify: false

If you have followed the Tanka Tempo installation example, then the endpoint value would be:

distributor.tempo.svc.cluster.local:4317

Apply the ConfigMap with:

kubectl apply --namespace default -f agent.yaml

Deploy Grafana Agent using the procedures from the relevant instructions above.

Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

Navigate to Configuration ≫ Data Sources.
Click on Add data source.
Select Tempo.
Set the URL to http://<TEMPO-HOST>:<HTTP-LISTEN-PORT>/, filling in the path to your gateway and the configured HTTP API prefix. If you have followed the Tanka Tempo installation example, this will be: http://query-frontend.tempo.svc.cluster.local:3200/
Click Save & Test.

You should see a message that says Data source is working.

If you see an error that says Data source is not working: failed to get trace with id: 0, check your Grafana version.

To fix the error, upgrade your Grafana to 9.0 or later.

Visualize your data

Once you have created a data source, you can visualize your traces in the Grafana Explore page. For more information, refer to Tempo in Grafana.

Use OpenTelemetry `telemetrygen` to generate tracing data

Next, you can use OpenTelemetry telemetrygen to generate tracing data to test your Tempo installation.

In the following instructions we assume the endpoints for both the Grafana Agent and the Tempo distributor are those described above, for example:

grafana-agent-traces.default.svc.cluster.local for Grafana Agent
distributor.tempo.svc.cluster.local for the Tempo distributor Replace these appropriately if you have altered the endpoint targets.

Install telemetrygen using the installation procedure. NOTE: You do not need to configure an OpenTelemetry Collector as we are using the Grafana Agent.

Generate traces using telemtrygen:

telemetrygen traces --otlp-insecure --rate 20 --duration 5s grafana-agent-traces.default.svc.cluster.local:4317

This configuration sends traces to Grafana Agent for 5 seconds, at a rate of 20 traces per second.

Optionally, you can also send the trace directly to the Tempo database without using Grafana Agent as a collector by using the following:

telemetrygen traces --otlp-insecure --rate 20 --duration 5s distributor.tempo.svc.cluster.local:4317

View tracing data in Grafana

To view the tracing data:

Go to Grafana and select the Explore menu item.
Select the Tempo data source from the list of data sources.
Copy the trace ID into the Trace ID edit field.
Select Run query.
Confirm that traces are displayed in the traces Explore panel. You should see 5 seconds worth of traces, 100 traces in total per run of telemetrygen.

Set up Tempo on Grafana Labs

Plan your Tempo deployment

Plan your Tempo deployment

Monolithic mode

Scaling monolithic mode

Microservices mode

Tools used to deploy Tempo

Helm

Kubernetes Tempo Operator

Tanka/Jsonnet

Kubernetes manifests

Upgrade your Tempo installation

Upgrade your Tempo installation

Upgrade to Tempo 2.2

Default block format changed to vParquet2

Updated JSonnet supports statefulset for the metrics-generator

Removed or renamed configuration parameters

Upgrade to Tempo 2.1

Remove support for Search on v2 blocks

Upgrade path to maintain search from Tempo 1.x to 2.1

Breaking changes to metric names exposed by Tempo

Upgrade from Tempo 1.5 to 2.0

Check Tempo installation resource allocation

Enable TraceQL in Grafana

Check configuration options for removed and renamed options

Removed and replaced

Renamed

Deploy Tempo with Helm

Deploy Tempo with Helm

Example Helm chart

Helm charts for deployment

Deploy Tempo with Tempo Operator

Deploy Tempo with Tempo Operator

Installation

Compatibility

Tempo

Kubernetes

cert-manager

Community

Deploy on Linux

Deploy on Linux

Before you begin

System requirements

Set up an object storage bucket

Install Tempo

Create a Tempo configuration file

Move the configuration file to the proper directory

Restart the tempo service

Verify your cluster is working

Test your installation

Backend storage configuration

Network configuration

Steps

Deploy on Kubernetes with Tanka

Deploy on Kubernetes with Tanka

Before you begin

Configure Kubernetes and install Tanka

Set up the Tanka environment

Install libraries

Deploy MinIO object storage

Optional: Enable metrics-generator

Optional: Reduce component system requirements

Deploy Tempo using Tanka

Next steps

Set up a test application for a Tempo cluster

Set up a test application for a Tempo cluster

Before you begin

Set up remote_write to your Tempo cluster

Create a Grafana Tempo data source

Visualize your data

Use OpenTelemetry telemetrygen to generate tracing data

View tracing data in Grafana

Updated JSonnet supports `statefulset` for the metrics-generator

Set up `remote_write` to your Tempo cluster

Use OpenTelemetry `telemetrygen` to generate tracing data