Configure Tempo on Grafana Labs

Google Cloud Storage permissions

Fri, 03 Apr 2026 12:35:46 -0500

Google Cloud Storage permissions

For configuration options, check the storage section on the configuration page.

Permissions

The following authentication methods are supported:

Google Cloud Platform environment variable GOOGLE_APPLICATION_CREDENTIALS
Google Cloud Platform Workload Identity

The (service-)account that will communicate towards GCS should be assigned to the bucket which will receive the traces and should have the following IAM policies within the bucket:

storage.objects.create
storage.objects.delete
storage.objects.get
storage.buckets.get
storage.objects.list

Amazon S3 permissions

Fri, 03 Apr 2026 12:35:46 -0500

Amazon S3 permissions

For configuration options, refer to the storage section on the configuration page.

The following authentication methods are supported:

AWS environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Static access key and secret credentials specified in access_key and secret_key
MinIO environment variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY
AWS shared credentials configuration file
MinIO client credentials configuration file
AWS IAM (IRSA via WebIdentity,
AWS EC2 instance role)

The following IAM policy shows minimal permissions required by Tempo, where the bucket has already been created.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TempoPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetObjectTagging",
                "s3:PutObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<bucketname>/*",
                "arn:aws:s3:::<bucketname>"
            ]
        }
    ]
}

Lifecycle policy

A lifecycle policy is recommended that deletes incomplete multipart uploads after one day.

Azure blob storage permissions and management

Fri, 03 Apr 2026 12:35:46 -0500

Azure blob storage permissions and management

Tempo requires the following configuration to authenticate to and access Azure blob storage:

Storage Account name specified in the configuration file as storage_account_name or in the environment variable AZURE_STORAGE_ACCOUNT
Credentials for accessing the Storage Account; can be one of the following
- Storage Account access key specified in the configuration file as storage_account_key or in the environment variable AZURE_STORAGE_KEY
- An Azure Managed Identity; either system or user assigned. To use Azure Managed Identities, you’ll need to set use_managed_identity to true in the configuration file or set user_assigned_id to the client ID for the managed identity you’d like to use.
  - For a system-assigned managed identity, no additional configuration is required.
  - For a user-assigned managed identity, you’ll need to set user_assigned_id to the client ID for the managed identity in the configuration file.
- Via Azure Workload Identity. To use Azure Workload Identity, you’ll need to enable Azure Workload Identity on your cluster, add the required label and annotation to the service account and the required pod label.

Azure blocklist polling

If you are hosting Tempo on Azure, two values may need to be updated to ensure consistent successful blocklist polling. If you are experiencing this issue, we recommend to set blocklist_poll_tenant_index_builders to 1.

Additionally, if you are seeing DNS failures like the ones below, try increasing blocklist_poll_jitter_ms. Discussion here.

reading storage container: Head "https://tempoe**************.blob.core.windows.net/tempo/single-tenant/d8aafc48-5796-4221-ac0b-58e001d18515/meta.compacted.json?timeout=61": dial tcp: lookup tempoe**************.blob.core.windows.net on 10.0.0.10:53: dial udp 10.0.0.10:53: operation was canceled

Your final config may look something like:

  storage:
    trace:
      blocklist_poll_tenant_index_builders: 1
      blocklist_poll_jitter_ms: 500

(Optional) Storage Account management policy for cleaning up the storage container

The following Storage Account management policy shows an example of cleaning up files from the container after they have been deleted for a period of time.

{
  "id": "/subscriptions/00000000-0000-0000000000000000000000/resourceGroups/resourceGroupName/providers/Microsoft.Storage/storageAccounts/accountName/managementPolicies/default",
  "lastModifiedTime": "2021-11-30T19:19:54.855455+00:00",
  "name": "DefaultManagementPolicy",
  "policy": {
    "rules": [
      {
        "definition": {
          "actions": {
            "baseBlob": {
              "delete": {
                "daysAfterLastAccessTimeGreaterThan": null,
                "daysAfterModificationGreaterThan": 60.0
              },
              "enableAutoTierToHotFromCool": null,
              "tierToArchive": null,
              "tierToCool": null
            },
            "snapshot": null,
            "version": null
          },
          "filters": {
            "blobIndexMatch": null,
            "blobTypes": [
              "blockBlob"
            ],
            "prefixMatch": [
              "tempo-data"
            ]
          }
        },
        "enabled": true,
        "name": "TempoBlobRetention",
        "type": "Lifecycle"
      },
      {
        "definition": {
          "actions": {
            "baseBlob": null,
            "snapshot": null,
            "version": {
              "delete": {
                "daysAfterCreationGreaterThan": 7.0
              },
              "tierToArchive": null,
              "tierToCool": null
            }
          },
          "filters": {
            "blobIndexMatch": null,
            "blobTypes": [
              "blockBlob"
            ],
            "prefixMatch": []
          }
        },
        "enabled": true,
        "name": "VersionRetention",
        "type": "Lifecycle"
      }
    ]
  },
  "resourceGroup": "resource-group-name",
  "type": "Microsoft.Storage/storageAccounts/managementPolicies"
}

Configure TLS communication

Fri, 03 Apr 2026 12:35:46 -0500

Configure TLS communication

Tempo can be configured to communicate between the components using Transport Layer Security, or TLS.

Note
The ciphers and TLS version here are for example purposes only. We are not recommending which ciphers or TLS versions for use in production environments.

Server configuration

This sample TLS server configuration shows supported options.

server:
  tls_cipher_suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  tls_min_version: VersionTLS12

  grpc_tls_config:
    cert_file: /tls/tls.crt
    key_file: /tls/tls.key
    client_auth_type: VerifyClientCertIfGiven
    client_ca_file: /tls/ca.crt
  http_tls_config:
    cert_file: /tls/tls.crt
    key_file: /tls/tls.key
    client_auth_type: VerifyClientCertIfGiven
    client_ca_file: /tls/ca.crt

Valid values for the client_auth_type are documented in the standard crypt/tls package under ClientAuthType here.

Client configuration

Several components of Tempo need to configure the gRPC clients they use to communicate with other components. For example, when the querier contacts the query-frontend to request work, the client in use must enable TLS if the server is serving a TLS endpoint.

The Tempo configuration uses a standard configuration stanza for each of these client configurations. Below is an example of the configuration.

The optional configuration elements tls_min_version, tls_cipher_suites, and tls_insecure_skip_verify may be omitted. The option tls_server_name may or may not be required, depending on the environment.

grpc_client_config:
  tls_enabled: true
  tls_cert_path: /tls/tls.crt
  tls_key_path: /tls/tls.key
  tls_ca_path: /tls/ca.crt
  tls_server_name: tempo.trace.svc.cluster.local
  tls_insecure_skip_verify: false
  tls_cipher_suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  tls_min_version: VersionTLS12

The configuration block needs to be set at the following configuration locations.

ingester_client.grpc_client_config
metrics_generator_client.grpc_client_config
querier.query-frontend.grpc_client_config

Additionally, memberlist must also be configured, but the client configuration is nested directly under memberlist as follows. The same configuration options are available as above.

memberlist:
    tls_enabled: true
    tls_cert_path: /tls/tls.crt
    tls_key_path: /tls/tls.key
    tls_ca_path: /tls/ca.crt
    tls_server_name: tempo.trace.svc.cluster.local
    tls_insecure_skip_verify: false

Receiver TLS

Additional receiver configuration can be added to support TLS communication for traces being sent to Tempo. The receiver configuration is pulled in from the Open Telemetry collector, and is documented upstream here.

An example tls block might look like the following:

tls:
  ca_file: /tls/ca.crt
  cert_file: /tls/tls.crt
  key_file: /tls/tls.key
  min_version: VersionTLS12

The above structure can be set on the following receiver configurations:

distributor.receivers.otlp.protocols.grpc.tls
distributor.receivers.otlp.protocols.http.tls
distributor.receivers.zipkin.tls
distributor.receivers.jaeger.protocols.grpc.tls
distributor.receivers.jaeger.protocols.thrift_http.tls

Run Tempo Distributed with sidecar proxies

Fri, 03 Apr 2026 12:35:46 -0500

Run Tempo Distributed with sidecar proxies

You can route inter-pod gRPC traffic run through a sidecar proxy to meet requirements such as custom security, routing, or logging. Common examples include Envoy, Nginx, Traefik, or service meshes like Istio and Linkerd.

How Tempo pods communicate

Tempo pods communicate using gRPC.

The different components like distributors and ingesters find each other by a shared ring with the list of pods, their roles, and their addresses. Pods advertise their address and listening port into the ring when they start, and deregister themselves when they exit.

The overall network looks like this:

The low-level ring data for ingesters can be viewed by browsing to the /ingester/ring URL on a distributor. It looks like this:

By default, gRPC traffic uses port 9095, but this can be changed by customizing the grpc_listen_port for each pod that needs it.

server:
  grpc_listen_port: 12345

The ring contents reflect the new port:

Run Tempo with proxies

Some installations require that the inter-pod gRPC traffic runs through a sidecar proxy. Running Tempo with proxies requires setting two ports for each pod: one for the Tempo process and one for the sidecar. Additionally, the ring contents must reflect the proxy’s port so that traffic from other pods goes through the proxy.

The overall network looks like this:

This cannot be accomplished by setting the same grpc_listen_port as in the previous example. Instead, we need the ingester to listen on port A but advertise itself on port B. This is done by customizing the ingester’s lifecycler port:

ingester:
   lifecycler:
       port: 12345

Now, the ingester is listening for regular traffic on port 9095, but the distributor will route traffic to it on port 12345.

Metrics-generator proxy

You can customize the lifecyler port in the metrics-generator. To set an instance port for the metrics-generator, use this configuration:

metrics_generator:
  ring:
    instance_port: 12345

Replace 12345 with the correct port number.

Configure IPv6

Fri, 03 Apr 2026 12:35:46 -0500

Configure IPv6

Tempo can be configured to communicate between the components using Internet Protocol Version 6, or IPv6.

Note: In order to support this support this configuration, the underlying infrastructure must support this address family. This configuration may be used in a single-stack IPv6 environment, or in a dual-stack environment where both IPv6 and IPv4 are present. In a dual-stack scenario, only one address family may be configured at a time, and all components must be configured for that address family.

Protocol configuration

This sample listen configuration will allow the gRPC and HTTP servers to listen on IPv6, and configure the various memberlist components to enable IPv6.

memberlist:
  bind_addr:
    - '::'
  bind_port: 7946

compactor:
  ring:
    kvstore:
      store: memberlist
    enable_inet6: true

metrics_generator:
  ring:
    enable_inet6: true

ingester:
  lifecycler:
    address: '::'
    enable_inet6: true

server:
  grpc_listen_address: '[::0]'
  grpc_listen_port: 9095
  http_listen_address: '[::0]'
  http_listen_port: 3200

Kubernetes service configuration

Each service fronting the workloads will need to be configured with with spec.ipFamilies and spec.ipFamilyPolicy set. See this compactor example.

apiVersion: v1
kind: Service
metadata:
  labels:
    name: compactor
  name: compactor
  namespace: trace
spec:
  clusterIP: fccb::31a7
  clusterIPs:
    - fccb::31a7
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv6
  ipFamilyPolicy: SingleStack
  ports:
    - name: compactor-http-metrics
      port: 3200
      protocol: TCP
      targetPort: 3200
  selector:
    app: compactor
    name: compactor
  sessionAffinity: None
  type: ClusterIP

You can check the listening service from within a pod.

❯ k exec -it compactor-55c778b8d9-2kch2 -- sh
/ # apk add iproute2
OK: 12 MiB in 27 packages
/ # ss -ltn -f inet
State   Recv-Q   Send-Q     Local Address:Port     Peer Address:Port  Process
/ # ss -ltn -f inet6
State    Recv-Q   Send-Q     Local Address:Port     Peer Address:Port  Process
LISTEN   0        4096                   *:7946                *:*
LISTEN   0        4096                   *:9095                *:*
LISTEN   0        4096                   *:3200                *:*

Query Tempo with Grafana

Fri, 03 Apr 2026 12:35:46 -0500

Query Tempo with Grafana

Grafana can query Tempo directly. This feature has been enabled since Grafana 7.5.x.

Grafana Cloud comes pre-configured with a Tempo data source.

If you are using Grafana on-prem, you need to set up the Tempo data source.

Configure the data source

To query Tempo with Grafana:

Point the Grafana data source at your Tempo query frontend (or monolithic mode Tempo).
Enter the URL: http://<tempo hostname>:<http port number>. For most of our examples the following works.

The port of 3200 is a common port used in our examples. Tempo default HTTP port is 80.

Prior to Grafana 7.5.x, Grafana was not able to query Tempo directly and required an intermediary, Tempoo-Query. This the Grafana 7.4.x example to explains configuration. The url entered will be http://<tempo-query hostname>:16686/.

Compression and encoding

Fri, 03 Apr 2026 12:35:46 -0500

Compression and encoding

Tempo can compress traces that it pushes to backend storage. This requires extra memory and CPU, but it reduces the quantity of stored data. Anecdotal tests suggest that zstd will cut your storage costs to ~15% of the uncompressed amount. It is highly recommended to use the default zstd.

Compression is configured under storage like so:

storage:
  trace:
    block:
      v2_encoding: zstd

The following options are supported:

none
gzip
lz4-64k
lz4-256k
lz4-1M
lz4
snappy
zstd
s2

Although all of these compression formats are supported in Tempo, at Grafana we use zstd. It’s possible/probable that the other compression algorithms may have issue at scale. File an issue if you have any problems.

WAL

The WAL also supports compression. By default, this is configured to use snappy. This comes with a small performance penalty but reduces disk I/O and and adds checksums to the WAL. All of the above configuration options are supported but only snappy has been tested at scale.

storage:
  trace:
    wal:
      v2_encoding: snappy

Manifest

Fri, 03 Apr 2026 12:35:46 -0500

Manifest

This document is a reference for all Tempo options and their defaults. If you are just getting started with Tempo, refer to Tempo examples and other configuration documentation. Most installations will require only setting 10 to 20 of these options.

It was generated by running Tempo with a minimal configuration and accessing the /status/config endpoint:

go run ./cmd/tempo --storage.trace.backend=local --storage.trace.local.path=/tmp/tempo/traces --storage.trace.wal.path=/tmp/tempo/wal

Complete configuration

Note
This manifest was generated on 2023-06-01.

target: all
http_api_prefix: ""
server:
    http_listen_network: tcp
    http_listen_address: ""
    http_listen_port: 80
    http_listen_conn_limit: 0
    grpc_listen_network: tcp
    grpc_listen_address: ""
    grpc_listen_port: 9095
    grpc_listen_conn_limit: 0
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: true
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 30s
    http_server_write_timeout: 30s
    http_server_idle_timeout: 2m0s
    grpc_server_max_recv_msg_size: 16777216
    grpc_server_max_send_msg_size: 16777216
    grpc_server_max_concurrent_streams: 100
    grpc_server_max_connection_idle: 2562047h47m16.854775807s
    grpc_server_max_connection_age: 2562047h47m16.854775807s
    grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
    grpc_server_keepalive_time: 2h0m0s
    grpc_server_keepalive_timeout: 20s
    grpc_server_min_time_between_pings: 10s
    grpc_server_ping_without_stream_allowed: true
    log_format: logfmt
    log_level: info
    log_source_ips_enabled: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_headers: false
    log_request_at_info_level_enabled: false
    log_request_exclude_headers_list: ""
    http_path_prefix: ""
internal_server:
    http_listen_network: tcp
    http_listen_address: localhost
    http_listen_port: 3101
    http_listen_conn_limit: 0
    grpc_listen_network: ""
    grpc_listen_address: ""
    grpc_listen_port: 0
    grpc_listen_conn_limit: 0
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: false
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 30s
    http_server_write_timeout: 30s
    http_server_idle_timeout: 2m0s
    grpc_server_max_recv_msg_size: 0
    grpc_server_max_send_msg_size: 0
    grpc_server_max_concurrent_streams: 0
    grpc_server_max_connection_idle: 0s
    grpc_server_max_connection_age: 0s
    grpc_server_max_connection_age_grace: 0s
    grpc_server_keepalive_time: 0s
    grpc_server_keepalive_timeout: 0s
    grpc_server_min_time_between_pings: 0s
    grpc_server_ping_without_stream_allowed: false
    log_format: ""
    log_level: ""
    log_source_ips_enabled: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_headers: false
    log_request_at_info_level_enabled: false
    log_request_exclude_headers_list: ""
    http_path_prefix: ""
    enable: false
distributor:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 5m0s
        instance_id: local-instance
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
    receivers: {}
    override_ring_key: distributor
    log_received_traces: false
    forwarders: []
    extend_writes: true
ingester_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
metrics_generator_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
querier:
    search:
        query_timeout: 30s
        prefer_self: 10
        external_hedge_requests_at: 8s
        external_hedge_requests_up_to: 2
        external_backend: ""
        google_cloud_run: null
        external_endpoints: []
    trace_by_id:
        query_timeout: 10s
    max_concurrent_queries: 20
    frontend_worker:
        frontend_address: 127.0.0.1:9095
        dns_lookup_duration: 10s
        parallelism: 2
        match_max_concurrent: true
        id: ""
        grpc_client_config:
            max_recv_msg_size: 104857600
            max_send_msg_size: 16777216
            grpc_compression: gzip
            rate_limit: 0
            rate_limit_burst: 0
            backoff_on_ratelimits: false
            backoff_config:
                min_period: 100ms
                max_period: 1s
                max_retries: 5
            initial_stream_window_size: 0B
            initial_connection_window_size: 0B
            tls_enabled: false
            tls_cert_path: ""
            tls_key_path: ""
            tls_ca_path: ""
            tls_server_name: ""
            tls_insecure_skip_verify: false
            tls_cipher_suites: ""
            tls_min_version: ""
            connect_timeout: 0s
            connect_backoff_base_delay: 0s
            connect_backoff_max_delay: 0s
    query_relevant_ingesters: false
query_frontend:
    max_outstanding_per_tenant: 2000
    querier_forget_delay: 0s
    max_batch_size: 5
    max_retries: 2
    search:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        default_result_limit: 20
        max_result_limit: 0
        max_duration: 168h0m0s
        query_backend_after: 15m0s
        query_ingesters_until: 30m0s
    trace_by_id:
        query_shards: 50
        hedge_requests_at: 2s
        hedge_requests_up_to: 2
compactor:
    ring:
        kvstore:
            store: ""
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        wait_stability_min_duration: 1m0s
        wait_stability_max_duration: 5m0s
        instance_id: local-instance
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
        enable_inet6: false
        wait_active_instance_timeout: 10m0s
    compaction:
        v2_in_buffer_bytes: 5242880
        v2_out_buffer_bytes: 20971520
        v2_prefetch_traces_count: 1000
        compaction_window: 1h0m0s
        max_compaction_objects: 6000000
        max_block_bytes: 107374182400
        block_retention: 336h0m0s
        compacted_block_retention: 1h0m0s
        retention_concurrency: 10
        max_time_per_tenant: 5m0s
        compaction_cycle: 30s
    override_ring_key: compactor
ingester:
    lifecycler:
        ring:
            kvstore:
                store: inmemory
                prefix: collectors/
                consul:
                    host: localhost:8500
                    acl_token: ""
                    http_client_timeout: 20s
                    consistent_reads: false
                    watch_rate_limit: 1
                    watch_burst_size: 1
                    cas_retry_delay: 1s
                etcd:
                    endpoints: []
                    dial_timeout: 10s
                    max_retries: 10
                    tls_enabled: false
                    tls_cert_path: ""
                    tls_key_path: ""
                    tls_ca_path: ""
                    tls_server_name: ""
                    tls_insecure_skip_verify: false
                    tls_cipher_suites: ""
                    tls_min_version: ""
                    username: ""
                    password: ""
                multi:
                    primary: ""
                    secondary: ""
                    mirror_enabled: false
                    mirror_timeout: 2s
            heartbeat_timeout: 5m0s
            replication_factor: 1
            zone_awareness_enabled: false
            excluded_zones: ""
        num_tokens: 128
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        observe_period: 0s
        join_after: 0s
        min_ready_duration: 15s
        interface_names:
            - en0
        enable_inet6: false
        final_sleep: 0s
        tokens_file_path: ""
        availability_zone: ""
        unregister_on_shutdown: true
        readiness_check_ring_health: true
        address: 127.0.0.1
        port: 0
        id: local-instance
    concurrent_flushes: 4
    flush_check_period: 10s
    flush_op_timeout: 5m0s
    trace_idle_period: 10s
    max_block_duration: 30m0s
    max_block_bytes: 524288000
    complete_block_timeout: 15m0s
    override_ring_key: ring
metrics_generator:
    ring:
        kvstore:
            store: inmemory
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        instance_id: local-instance
        instance_interface_names:
            - eth0
            - en0
        instance_addr: 127.0.0.1
        instance_port: 0
        enable_inet6: false
    processor:
        service_graphs:
            wait: 10s
            max_items: 10000
            workers: 10
            histogram_buckets:
                - 0.1
                - 0.2
                - 0.4
                - 0.8
                - 1.6
                - 3.2
                - 6.4
                - 12.8
            dimensions: []
            peer_attributes:
                - peer.service
                - db.name
                - db.system
            span_multiplier_key: ""
        span_metrics:
            histogram_buckets:
                - 0.002
                - 0.004
                - 0.008
                - 0.016
                - 0.032
                - 0.064
                - 0.128
                - 0.256
                - 0.512
                - 1.024
                - 2.048
                - 4.096
                - 8.192
                - 16.384
            intrinsic_dimensions:
                service: true
                span_name: true
                span_kind: true
                status_code: true
            dimensions: []
            dimension_mappings: []
            enable_target_info: false
            span_multiplier_key: ""
            subprocessors:
                0: true
                1: true
                2: true
            filter_policies: []
            target_info_excluded_dimensions: []
        local_blocks:
            block:
                bloom_filter_false_positive: 0.01
                bloom_filter_shard_size_bytes: 102400
                version: vParquet2
                search_encoding: snappy
                search_page_size_bytes: 1048576
                v2_index_downsample_bytes: 1048576
                v2_index_page_size_bytes: 256000
                v2_encoding: zstd
                parquet_row_group_size_bytes: 100000000
                parquet_dedicated_columns: []
            search:
                chunk_size_bytes: 1000000
                prefetch_trace_count: 1000
                read_buffer_count: 32
                read_buffer_size_bytes: 1048576
                cache_control:
                    footer: false
                    column_index: false
                    offset_index: false
            flush_check_period: 10s
            trace_idle_period: 10s
            max_block_duration: 1m0s
            max_block_bytes: 500000000
            complete_block_timeout: 1h0m0s
            max_live_traces: 0
    registry:
        collection_interval: 15s
        stale_duration: 15m0s
        max_label_name_length: 1024
        max_label_value_length: 2048
    storage:
        path: ""
        wal:
            wal_segment_size: 134217728
            wal_compression: false
            stripe_size: 16384
            truncate_frequency: 2h0m0s
            min_wal_time: 300000
            max_wal_time: 14400000
            no_lockfile: false
        remote_write_flush_deadline: 1m0s
    traces_storage:
        path: ""
        completedfilepath: ""
        blocksfilepath: ""
        v2_encoding: none
        search_encoding: none
        ingestion_time_range_slack: 0s
        version: vParquet2
    metrics_ingestion_time_range_slack: 30s
    query_timeout: 30s
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        wal:
            path: /tmp/tempo/wal
            completedfilepath: /tmp/tempo/wal/completed
            blocksfilepath: /tmp/tempo/wal/blocks
            v2_encoding: snappy
            search_encoding: none
            ingestion_time_range_slack: 2m0s
            version: vParquet2
        block:
            bloom_filter_false_positive: 0.01
            bloom_filter_shard_size_bytes: 102400
            version: vParquet2
            search_encoding: snappy
            search_page_size_bytes: 1048576
            v2_index_downsample_bytes: 1048576
            v2_index_page_size_bytes: 256000
            v2_encoding: zstd
            parquet_row_group_size_bytes: 100000000
            parquet_dedicated_columns: []
        search:
            chunk_size_bytes: 1000000
            prefetch_trace_count: 1000
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576
            cache_control:
                footer: false
                column_index: false
                offset_index: false
        blocklist_poll: 5m0s
        blocklist_poll_concurrency: 50
        blocklist_poll_fallback: true
        blocklist_poll_tenant_index_builders: 2
        blocklist_poll_stale_tenant_index: 0s
        blocklist_poll_jitter_ms: 0
        blocklist_poll_tolerate_consecutive_errors: 1
        backend: local
        local:
            path: /tmp/tempo/traces
        gcs:
            bucket_name: ""
            prefix: ""
            chunk_buffer_size: 10485760
            endpoint: ""
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            insecure: false
            object_cache_control: ""
            object_metadata: {}
        s3:
            tls_cert_path: ""
            tls_key_path: ""
            tls_ca_path: ""
            tls_server_name: ""
            tls_insecure_skip_verify: false
            tls_cipher_suites: ""
            tls_min_version: VersionTLS12
            bucket: ""
            prefix: ""
            endpoint: ""
            region: ""
            access_key: ""
            secret_key: ""
            session_token: ""
            insecure: false
            part_size: 0
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            signature_v2: false
            forcepathstyle: false
            bucket_lookup_type: 0
            tags: {}
            storage_class: ""
            metadata: {}
        azure:
            storage_account_name: ""
            storage_account_key: ""
            use_managed_identity: false
            use_federated_token: false
            user_assigned_id: ""
            container_name: ""
            prefix: ""
            endpoint_suffix: blob.core.windows.net
            max_buffers: 4
            buffer_size: 3145728
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
        cache: ""
        cache_min_compaction_level: 0
        cache_max_block_age: 0s
        background_cache:
            writeback_goroutines: 10
            writeback_buffer: 10000
        memcached: null
        redis: null
overrides:
    ingestion_rate_strategy: local
    ingestion_rate_limit_bytes: 15000000
    ingestion_burst_size_bytes: 20000000
    max_traces_per_user: 10000
    max_global_traces_per_user: 0
    forwarders: []
    metrics_generator_ring_size: 0
    metrics_generator_processors: null
    metrics_generator_max_active_series: 0
    metrics_generator_collection_interval: 0s
    metrics_generator_disable_collection: false
    metrics_generator_forwarder_queue_size: 0
    metrics_generator_forwarder_workers: 0
    metrics_generator_processor_service_graphs_histogram_buckets: []
    metrics_generator_processor_service_graphs_dimensions: []
    metrics_generator_processor_service_graphs_peer_attributes: []
    metrics_generator_processor_span_metrics_histogram_buckets: []
    metrics_generator_processor_span_metrics_dimensions: []
    metrics_generator_processor_span_metrics_intrinsic_dimensions: {}
    metrics_generator_processor_span_metrics_filter_policies: []
    metrics_generator_processor_span_metrics_dimension_mappings: []
    metrics_generator_processor_span_metrics_enable_target_info: false
    metrics_generator_processor_span_metrics_target_info_excluded_dimensions: []
    metrics_generator_processor_local_blocks_max_live_traces: 0
    metrics_generator_processor_local_blocks_max_block_duration: 0s
    metrics_generator_processor_local_blocks_max_block_bytes: 0
    metrics_generator_processor_local_blocks_flush_check_period: 0s
    metrics_generator_processor_local_blocks_trace_idle_period: 0s
    metrics_generator_processor_local_blocks_complete_block_timeout: 0s
    block_retention: 0s
    max_bytes_per_tag_values_query: 5000000
    max_blocks_per_tag_values_query: 0
    max_search_duration: 0s
    max_bytes_per_trace: 5000000
    parquet_dedicated_columns: []
    per_tenant_override_config: ""
    per_tenant_override_period: 10s
    user_configurable_overrides:
        enabled: false
        poll_interval: 1m0s
        client:
            backend: ""
            confirm_versioning: true
            local:
                path: ""
            gcs:
                bucket_name: ""
                prefix: ""
                chunk_buffer_size: 0
                endpoint: ""
                hedge_requests_at: 0s
                hedge_requests_up_to: 0
                insecure: false
                object_cache_control: ""
                object_metadata: {}
            s3:
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                bucket: ""
                prefix: ""
                endpoint: ""
                region: ""
                access_key: ""
                secret_key: ""
                session_token: ""
                insecure: false
                part_size: 0
                hedge_requests_at: 0s
                hedge_requests_up_to: 0
                signature_v2: false
                forcepathstyle: false
                bucket_lookup_type: 0
                tags: {}
                storage_class: ""
                metadata: {}
            azure:
                storage_account_name: ""
                storage_account_key: ""
                use_managed_identity: false
                use_federated_token: false
                user_assigned_id: ""
                container_name: ""
                prefix: ""
                endpoint_suffix: ""
                max_buffers: 0
                buffer_size: 0
                hedge_requests_at: 0s
                hedge_requests_up_to: 0
memberlist:
    node_name: ""
    randomize_node_name: true
    stream_timeout: 10s
    retransmit_factor: 2
    pull_push_interval: 30s
    gossip_interval: 1s
    gossip_nodes: 2
    gossip_to_dead_nodes_time: 30s
    dead_node_reclaim_time: 0s
    compression_enabled: false
    advertise_addr: ""
    advertise_port: 7946
    cluster_label: ""
    cluster_label_verification_disabled: false
    join_members: []
    min_join_backoff: 1s
    max_join_backoff: 1m0s
    max_join_retries: 10
    abort_if_cluster_join_fails: false
    rejoin_interval: 0s
    left_ingesters_timeout: 5m0s
    leave_timeout: 20s
    message_history_buffer_bytes: 0
    bind_addr: []
    bind_port: 7946
    packet_dial_timeout: 2s
    packet_write_timeout: 5s
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
usage_report:
    reporting_enabled: true
    backoff:
        min_period: 100ms
        max_period: 10s
        max_retries: 0

Apache Parquet block format

Fri, 03 Apr 2026 12:35:46 -0500

Apache Parquet block format

Tempo has a default columnar block format based on Apache Parquet. This format is required for tags-based search as well as TraceQL, the query language for traces. The columnar block format improves search performance and enables a large ecosystem of tools to access the underlying trace data.

For more information, refer to the Parquet design document and Issue 1480. Additionally, there is now a Parquet v3 design document.

If you install using the new Helm charts, then Parquet is enabled by default.

Considerations

The Parquet block format is enabled by default in Tempo 2.0. No data conversion or upgrade process is necessary. As soon as the format is enabled, Tempo starts writing data in that format, leaving existing data as-is.

Block formats based on Parquet require more CPU and memory resources than the previous v2 format but provide search and TraceQL functionality.

Choose a different block format

The default block format is vParquet2 which is the latest iteration of Tempo’s Parquet based columnar block format. It is still possible to use the previous format vParquet. To enable it, set the block version option to vParquet in the Storage section of the configuration file.

# block format version. options: v2, vParquet, vParquet2
[version: vParquet]

It is possible to disable Parquet and use the previous v2 block format. This disables all forms of search, but also reduces resource consumption, and may be desired for a high-throughput cluster that does not need these capabilities. Set the block version option to v2 in the Storage section of the configuration file.

# block format version. options: v2, vParquet, vParquet2
[version: v2]

To re-enable the default vParquet2 format, remove the block version option from the Storage section of the configuration file or set the option to vParquet2.

Parquet configuration parameters

Some parameters in the Tempo configuration are specific to Parquet. For more information, refer to the storage configuration documentation.

Trace search parameters

These configuration options impact trace search.

Parameter	Default value	Description
`[read_buffer_size_bytes: <int>]`	`10485676`	Size of read buffers used when performing search on a vParquet block. This value times the `read_buffer_count` is the total amount of bytes used for buffering when performing search on a Parquet block.

`[read_buffer_count: <int>]`	32	Number of read buffers used when performing search on a vParquet block. This value times the `read_buffer_size_bytes` is the total amount of bytes used for buffering when performing search on a Parquet block.

The cache_control section contains the follow parameters for Parquet metadata objects:

Parameter	Default value	Description
`[footer: \| default = false]`	`false`	Specifies if the footer should be cached
`[column_index: <bool> \| default = false]`	`false`	Specifies if the column index should be cached
`[offset_index: <bool> \| default = false]`	`false`	Specifies if the offset index should be cached

Convert to Parquet

If you have used an earlier version of the Parquet format, you can use tempo-cli to convert a Parquet file from its existing schema to the one used in Tempo 2.0.

For instructions, refer to the Parquet convert command documentation.

Polling

Fri, 03 Apr 2026 12:35:46 -0500

Polling

The polling cycle is controlled by a number of configuration options detailed here.

storage:
    trace:
        # How often to repoll the backend for new blocks. Default is 5m
        [blocklist_poll: <duration>]

        # Number of blocks to process in parallel during polling. Default is 50.
        [blocklist_poll_concurrency: <int>]

        # By default components will pull the blocklist from the tenant index. If that fails the component can
        # fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true.
        [blocklist_poll_fallback: <bool>]

        # Maximum number of compactors that should build the tenant index. All other components will download
        # the index.  Default 2.
        [blocklist_poll_tenant_index_builders: <int>]

        # The oldest allowable tenant index. If an index is pulled that is older than this duration,
        # the polling will consider this an error. Note that `blocklist_poll_fallback` applies here.
        # If fallback is true and a tenant index exceeds this duration, it will fall back to listing
        # the bucket contents.
        # Default 0 (disabled).
        [blocklist_poll_stale_tenant_index: <duration>]

Due to the mechanics of the tenant index, the blocklist will be stale by at most 2 times the configured blocklist_poll duration. There are two configuration options that need to be balanced against the blockist_poll to handle this:

The ingester complete_block_timeout is used to hold a block in the ingester for a given period of time after it has been flushed. This allows the ingester to return traces to the queriers while they are still unaware of the newly flushed blocks.

ingester:
  # How long to hold a complete block in the ingester after it has been flushed to the backend.  Default is 15m
  [complete_block_timeout: <duration>]

The compactor compacted_block_retention is used to keep a block in the backend for a given period of time after it has been compacted and the data is no longer needed. This allows queriers with a stale blocklist to access these blocks successfully until they complete their polling cycles and have up to date blocklists. Like the complete_block_timeout, this should be at a minimum 2x the configured blocklist_poll duration.

compactor:
  compaction:
    # How long to leave a block in the backend after it has been compacted successfully.  Default is 1h
    [compacted_block_retention: <duration>]

Additionally, the querier blocklist_poll duration needs to be greater than or equal to the compactor blocklist_poll duration. Otherwise, a querier may not correctly check all assigned blocks and incorrectly return 404. It is recommended to simply set both components to use the same poll duration.

Grafana Agent

Fri, 03 Apr 2026 12:35:46 -0500

Grafana Agent

The Grafana Agent is a telemetry collector for sending metrics, logs, and trace data to the opinionated Grafana observability stack.

It is commonly used as a tracing pipeline, offloading traces from the application and forwarding them to a storage backend. The Grafana Agent tracing stack is built using OpenTelemetry.

The Grafana Agent supports receiving traces in multiple formats: OTLP (OpenTelemetry), Jaeger, Zipkin, and OpenCensus.

On top of receiving and exporting traces, the Grafana Agent contains many features that make your distributed tracing system more robust, and leverages all the data that is processed in the pipeline.

Architecture

The Grafana Agent can be configured to run a set of tracing pipelines to collect data from your applications and write it to Tempo. Pipelines are built using OpenTelemetry, and consist of receivers, processors and exporters. The architecture mirrors that of the OTel Collector’s design. See the configuration reference for all available configuration options. For a quick start, refer to this blog post.

This allows you to configure multiple distinct tracing pipelines, each of which collects separate spans and sends them to different backends.

Receiving traces

The Grafana Agent supports multiple ingestion receivers: OTLP (OpenTelemetry), Jaeger, Zipkin, OpenCensus and Kafka.

Each tracing pipeline can be configured to receive traces in all these formats. Traces that arrive to a pipeline will go through the receivers/processors/exporters defined in it.

Pipeline processing

The Grafana Agent processes tracing data as it flows through the pipeline to make the distributed tracing system more reliable and leverage the data for other purposes such as trace discovery, tail-based sampling, and generating metrics.

Batching

The Agent supports batching of traces. Batching helps better compress the data, reduces the number of outgoing connections, and is a recommended best practice. To configure it, refer to the batch block in the configuration reference.

Attributes manipulation

The Grafana Agent allows for general manipulation of attributes on spans that pass through this agent. A common use may be to add an environment or cluster variable. To configure it, refer to the attributes block in the configuration reference.

Attaching metadata with Prometheus Service Discovery

Prometheus Service Discovery mechanisms enable you to attach the same metadata to your traces as your metrics. For example, for Kubernetes users this means that you can dynamically attach metadata for namespace, pod, and name of the container sending spans.

traces:
  ...
  scrape_configs:
  - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    job_name: kubernetes-pods
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
    - source_labels: [__meta_kubernetes_pod_container_name]
      target_label: container
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: false

This feature isn’t just useful for Kubernetes users, however. All of Prometheus’ various service discovery mechanisms are supported here. This means you can use the same scrape_configs between your metrics, logs, and traces to get the same set of labels, and easily transition between your observability data when moving from your metrics, logs, and traces.

To configure it, refer to the scrape_configs block in the configuration reference.

Trace discovery through automatic logging

Automatic logging writes well formatted log lines to help with trace discovery.

For a closer look into the feature, visit Automatic logging.

Tail-based sampling

The Agent implements tail-based sampling for distributed tracing systems and multi-instance Agent deployments. With this feature, sampling decisions can be made based on data from a trace, rather than exclusively with probabilistic methods.

For a detailed description, go to Tail-based sampling.

Generating metrics from spans

The Agent can take advantage of the span data flowing through the pipeline to generate Prometheus metrics.

Go to Span metrics for a more detailed explanation of the feature.

Service graph metrics

Service graph metrics represent the relationships between services within a distributed system.

This service graphs processor builds a map of services by analyzing traces, with the objective to find edges. Edges are spans with a parent-child relationship, that represent a jump (e.g. a request) between two services. The amount of requests and their duration are recorded as metrics, which are used to represent the graph.

To read more about this processor, go to its section.

Exporting spans

The Grafana Agent can export traces to multiple different backends for every tracing pipeline. Exporting is built using OpenTelemetry Collector’s OTLP exporter. The Agent supports exporting tracing in OTLP format.

Aside from endpoint and authentication, the exporter also provides mechanisms for retrying on failure, and implements a queue buffering mechanism for transient failures, such as networking issues.

To see all available options, refer to the remote_write block in the Agent configuration reference.