Troubleshoot Tempo on Grafana Labs

Distributor refusing spans

Fri, 03 Apr 2026 12:35:46 -0500

Distributor refusing spans

The two most likely causes of refused spans are unhealthy ingesters or trace limits being exceeded.

Unhealthy ingesters

Unhealthy ingesters can be caused by failing OOMs, or scale down events. If you have unhealthy ingesters, your log line will look something like this:

msg="pusher failed to consume trace data" err="at least 2 live replicas required, could only find 1"

In this case you may need to visit the ingester ring page at /ingester/ring on the Distributors and “Forget” the unhealthy ingesters. This will work in the short term, but the long term fix is to stabilize your ingesters.

Trace limits reached

In high volume tracing environments the default trace limits are sometimes not sufficient. These limits exist to protect Tempo and prevent it from OOMing, crashing or otherwise allow tenants to not DOS each other. If you are refusing spans due to limits you will see logs like this at the distributor:

msg="pusher failed to consume trace data" err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (52428800) exceeded while adding 15632 bytes to trace a0fbd6f9ac5e2077d90a19551dd67b6f for tenant single-tenant"
msg="pusher failed to consume trace data" err="rpc error: code = FailedPrecondition desc = LIVE_TRACES_EXCEEDED: max live traces per tenant exceeded: per-user traces limit (local: 60000 global: 0 actual local: 60000) exceeded"
msg="pusher failed to consume trace data" err="rpc error: code = ResourceExhausted desc = RATE_LIMITED: ingestion rate limit (15000000 bytes) exceeded while adding 10 bytes"

You will also see the following metric incremented. The reason label on this metric will contain information about the refused reason.

tempo_discarded_spans_total

In this case use available configuration options to increase limits.

Troubleshoot Grafana Agent

Fri, 03 Apr 2026 12:35:46 -0500

Troubleshoot Grafana Agent

Sometimes it can be difficult to tell what, if anything, the Grafana Agent is sending along to the backend. This document focuses on a few techniques to gain visibility on how many traces are being pushed to the Agent and if they’re making it to the backend. The tracing pipeline is built on top of the OpenTelemetry Collector which does a fantastic job of logging network and other issues.

If your logs are showing no obvious errors try the following:

Metrics

The agent publishes a few Prometheus metrics that are useful to determine how much trace traffic it is receiving and successfully forwarding. These are a good place to start when diagnosing tracing Agent issues.

traces_receiver_accepted_spans
traces_receiver_refused_spans
traces_exporter_sent_spans
traces_exporter_send_failed_spans

Automatic logging

If metrics and logs are looking good, but you are still unable to find traces in Grafana Cloud, you can turn on Automatic Logging. A recommend debug setup is:

traces:
  configs:
  - name: default
    ...
    automatic_logging:
      backend: stdout
      roots: true

This will emit logs to stdout for every root span that the Agent forwards. This can be useful to see exactly which traces are being forwarded to Grafana Cloud.

Unable to find traces

Fri, 03 Apr 2026 12:35:46 -0500

Unable to find traces

The two main causes of missing traces are:

Issues in ingestion of the data into Tempo. Spans are either not being sent correctly to Tempo or they are not getting sampled.
Issues querying for traces that have been received by Tempo.

Section 1: Diagnose and fix ingestion issues

The first step is to check whether the application spans are actually reaching Tempo.

Add the following flag to the distributor container - distributor.log-received-traces.

This flag enables debug logging of all the traces received by the distributor. These logs can help check if Tempo is receiving any traces at all.

You can also check the following metrics:

tempo_distributor_spans_received_total
tempo_ingester_traces_created_total

The value of both metrics should be greater than 0 within a few minutes of the application spinning up. You can check both metrics using either

The metrics page exposed from Tempo at http://<tempo-address>:<tempo-http-port>/metrics or
In Prometheus, if it is being used to scrape metrics.

Case 1 - tempo_distributor_spans_received_total is 0

If the value of tempo_distributor_spans_received_total is 0, possible reasons are:

Use of incorrect protocol/port combination while initializing the tracer in the application.
Tracing records not getting picked up to send to Tempo by the internal sampler.
Application is running inside docker and sending traces to an incorrect endpoint.

Receiver specific traffic information can also be obtained using tempo_receiver_accepted_spans which has a label for the receiver (protocol used for ingestion. Ex: jaeger-thrift).

Solutions

There are three possible solutions: protocol or port problems, sampling issues, or incorrect endpoints.

To fix protocol or port problems:

Find out which communication protocol is being used by the application to emit traces. This is unique to every client SDK. For instance: Jaeger Golang Client uses Thrift Compact over UDP by default.
Check the list of supported protocols and their ports and ensure that the correct combination is being used.

To fix sampling issues:

These issues can be tricky to determine because most SDKs use a probabilistic sampler by default. This may lead to just one in a 1000 records being picked up.
Check the sampling configuration of the tracer being initialized in the application and make sure it has a high sampling rate.
Some clients also provide metrics on the number of spans reported from the application, for example jaeger_tracer_reporter_spans_total. Check the value of that metric if available and make sure it is greater than zero.
Another way to diagnose this problem would be to generate lots and lots of traces to see if some records make their way to Tempo.

To fix an incorrect endpoint issue:

If the application is also running inside docker, make sure the application is sending traces to the correct endpoint (tempo:<receiver-port>).

Case 2 - tempo_ingester_traces_created_total is 0

If the value of tempo_ingester_traces_created_total is 0, this can indicate network issues between distributors and ingesters.

Checking the metric tempo_request_duration_seconds_count{route='/tempopb.Pusher/Push'} exposed from the ingester which indicates that it is receiving ingestion requests from the distributor.

Solution

Check logs of distributors for a message like msg="pusher failed to consume trace data" err="DoBatch: IngesterCount <= 0". This is likely because no ingester is joining the gossip ring, make sure the same gossip ring address is supplied to the distributors and ingesters.

Diagnose and fix sampling and limits issues

If you are able to query some traces in Tempo but not others, you have come to the right section!

This could happen because of a number of reasons and some have been detailed in this blog post - Where did all my spans go? A guide to diagnosing dropped spans in Jaeger distributed tracing. This is useful if you are using the Jaeger Agent.

If you are using the Grafana Agent, continue reading the following section for metrics to monitor.

Diagnose the issue

Check if the pipeline is dropping spans. The following metrics on Grafana Agent help determine this:

tempo_exporter_send_failed_spans. The value of this metric should be 0.
tempo_receiver_refused_spans. This value of this metric should be 0.
tempo_processor_dropped_spans. The value of this metric should be 0.

If the pipeline is not reporting any dropped spans, check whether application spans are being dropped by Tempo. The following metrics help determine this:

tempo_receiver_refused_spans. The value of tempo_receiver_refused_spans should be 0.

Grafana Agent and Tempo share the same metric. Make sure to check the value of the metric from both services. If the value of tempo_receiver_refused_spans is greater than 0, then the possible reason is the application spans are being dropped due to rate limiting.

Solution

If the pipeline (Grafana Agent) is found to be dropping spans, the deployment may need to be scaled up. Look for a message like too few agents compared to the ingestion rate in the agent logs.
There might also be issues with connectivity to Tempo backend, check the agent for logs like error sending batch, will retry and make sure the Tempo endpoint and credentials are correctly configured.
If Tempo is found to be dropping spans, then the possible reason is the application spans are being dropped due to rate limiting. The rate limiting may be appropriate and does not need to be fixed. The metric simply explained the cause of the missing spans, and there is nothing more to be done.
If more ingestion volume is needed, increase the configuration for the rate limiting, by adding this CLI flag to Tempo at startup - https://github.com/grafana/tempo/blob/78f3554ca30bd5a4dec01629b8b7b2b0b2b489be/modules/overrides/limits.go#L42

Note
Check the ingestion limits page for further information on limits.

Section 3: Diagnose and fix issues with querying traces

If you have determined that data has been ingested correctly into Tempo, then it is time to investigate possible issues with querying the data.

Check the logs of the Tempo Query Frontend. The Query Frontend pod runs with two containers (Query Frontend and Tempo Query), so lets use the following command to view Query Frontend logs -

kubectl logs -f pod/query-frontend-xxxxx -c query-frontend

The presence of the following errors in the log may explain issues with querying traces:

level=info ts=XXXXXXX caller=frontend.go:63 method=GET traceID=XXXXXXXXX url=/api/traces/XXXXXXXXX duration=5m41.729449877s status=500
no org id
could not dial 10.X.X.X:3200 connection refused
tenant-id not found

Possible reasons for the above errors are:

Tempo Querier is not connected to Tempo Query Frontend. Check the value of the metric cortex_query_frontend_connected_clients exposed by the Query Frontend. It should be > 0, which indicates that Queriers are connected to the Query Frontend.
Grafana Tempo data source is not configured to pass tenant-id in the Authorization header (only applicable to multi-tenant deployments).
Not connected to Tempo Querier correctly.
Insufficient permissions.

Solutions

To fix connection issues:

If the queriers are not connected to the Query Frontend, check the following section in Querier configuration and make sure the address of the Query Frontend is correct
YAML
```
querier:
  frontend_worker:
    frontend_address: query-frontend-discovery.default.svc.cluster.local:9095
```

Confirm that the Grafana data source is configured correctly and debug network issues between Grafana and Tempo.

To fix an insufficient permissions issue:

Verify that the Querier has the LIST and GET permissions on the bucket.

Too many jobs in the queue

Fri, 03 Apr 2026 12:35:46 -0500

Too many jobs in the queue

The error message might also be

queue doesn't have room for 100 jobs
failed to add a job to work queue

You may see this error if the compactor isn’t running and the blocklist size has exploded. Possible reasons why the compactor may not be running are:

Insufficient permissions.
Compactor sitting idle because no block is hashing to it.
Incorrect configuration settings.

Diagnosing the issue

Check metric tempodb_compaction_bytes_written_total If this is greater than zero (0), it means the compactor is running and writing to the backend.
Check metric tempodb_compaction_errors_total If this metric is greater than zero (0), check the logs of the compactor for an error message.

Solutions

Verify that the Compactor has the LIST, GET, PUT, and DELETE permissions on the bucket objects.
- If these permissions are missing, assign them to the compactor container.
- For detailed information, check - https://grafana.com/docs/tempo/latest/configuration/s3/#permissions
If there’s a compactor sitting idle while others are running, port-forward to the compactor’s http endpoint. Then go to /compactor/ring and click Forget on the inactive compactor.
Check the following configuration parameters to ensure that there are correct settings:
- max_block_bytes to determine when the ingester cuts blocks. A good number is anywhere from 100MB to 2GB depending on the workload.
- max_compaction_objects to determine the max number of objects in a compacted block. This should relatively high, generally in the millions.
- retention_duration for how long traces should be retained in the backend.
Check the storage section of the config and increase queue_depth. Do bear in mind that a deeper queue could mean longer waiting times for query responses. Adjust max_workers accordingly, which configures the number of parallel workers that query backend blocks.

storage:
  trace:
    pool:
      max_workers: 100                 # worker pool determines the number of parallel requests to the object store backend
      queue_depth: 10000

Bad blocks

Fri, 03 Apr 2026 12:35:46 -0500

Bad blocks

Queries fail with an error message containing:

error querying store in Querier.FindTraceByID: error using pageFinder (1, 5927cbfb-aabe-48b2-9df5-f4c3302d915f): ...

This might indicate that there is a bad (corrupted) block in the backend.

A block can get corrupted if the ingester crashed while flushing the block to the backend.

Fixing bad blocks

At the moment, a backend block can be fixed if either the index or bloom-filter is corrupt/deleted.

To fix such a block, first download it onto a machine where you can run the tempo-cli.

Next run the tempo-cli’s gen index / gen bloom commands depending on which file is corrupt/deleted. The command will create a fresh index/bloom-filter from the data file at the required location (in the block folder). To view all of the options for this command, see the cli docs.

Finally, upload the generated index or bloom-filter onto the object store backend under the folder for the block.

Removing bad blocks

If the above step on fixing bad blocks reveals that the data file is corrupt, the only remaining solution is to delete the block, which can result in some loss of data.

The mechanism to remove a block from the backend is backend-specific, but the block to remove will be at:

<tenant ID>/<block ID>

Tag search

Fri, 03 Apr 2026 12:35:46 -0500

Tag search

An issue occurs while searching for traces in Grafana Explore. The Service Name and Span Name drop down lists are empty, and there is a No options found message.

HTTP requests to Tempo query frontend endpoint at /api/search/tag/service.name/values would respond with an empty set.

Root cause

The introduction of a cap on the size of tags causes this issue.

Configuration parameter max_bytes_per_tag_values_query causes the return of an empty result when a query exceeds the configured value.

Solutions

There are two main solutions to this issue:

Reduce the cardinality of tags pushed to Tempo. Reducing the number of unique tag values will reduce the size returned by a tag search query.
Increase the max_bytes_per_tag_values_query parameter in the overrides block of your Tempo configuration to a value as high as 50MB.

Response larger than the max

Fri, 03 Apr 2026 12:35:46 -0500

Response larger than the max

The error message will take a similar form to the following:

500 Internal Server Error Body: response larger than the max (<size> vs <limit>)

This error indicates that the response received or sent is too large. This can happen in multiple places, but it’s most commonly seen in the query path, with messages between the querier and the query frontend.

Solutions

Tempo server (general)

Tempo components communicate with each other via gRPC requests. To increase the maximum message size, you can increase the gRPC message size limit in the server block.

server:
  grpc_server_max_recv_msg_size: <size>
  grpc_server_max_send_msg_size: <size>

The server config block is not synchronized across components. Most likely you will need to increase the message size limit in multiple components.

Querier

Additionally, querier workers can be configured to use a larger message size limit.

querier:
    frontend_worker:
        grpc_client_config:
            max_send_msg_size: <size>

Ingestion

Lastly, message size is also limited in ingestion and can be modified in the distributor block.

distributor:
  receivers:
    otlp:
      grpc:
        max_recv_msg_size_mib: <size>

Troubleshoot metrics-generator

Fri, 03 Apr 2026 12:35:46 -0500

Troubleshoot metrics-generator

If you are concerned with data quality issues in the metrics-generator, we’d first recommend:

Reviewing your telemetry pipeline to determine the number of dropped spans. We are only looking for major issues here.
Reviewing the service graph documentation to understand how they are built.

If everything seems ok from these two perspectives, consider the following topics to help resolve general issues with all metrics and span metrics specifically.

All metrics

Dropped spans in the distributor

The distributor has a queue of outgoing spans to the metrics-generators. If that queue is full then the distributor will drop spans before they reach the generator. Use the following metric to determine if that is happening:

sum(rate(tempo_distributor_queue_pushes_failures_total{}[1m]))

Failed pushes to the generator

For any number of reasons, the distributor can fail a push to the generators. Use the following metric to determine if that is happening:

sum(rate(tempo_distributor_metrics_generator_pushes_failures_total{}[1m]))

Discarded spans in the generator

Spans are rejected from being considered by the metrics-generator by a configurable slack time as well as due to user configurable filters. You can see the number of spans rejected by reason using this metric:

sum(rate(tempo_metrics_generator_spans_discarded_total{}[1m])) by (reason)

If a lot of spans are dropped in the metrics-generator due to your filters, you will need to adjust them. If spans are dropped due to the ingestion slack time, consider adjusting this setting:

metrics_generator:
  metrics_ingestion_time_range_slack: 30s

If spans are regularly exceeding this value you may want to consider reviewing your tracing pipeline to see if you have excessive buffering. Note that increasing this value allows the generator to consume more spans, but does reduce the accuracy of metrics because spans farther away from “now” are included.

Max active series

The generator protects itself and your remote-write target by having a maximum number of series the generator produces. Use the sum below to determine if series are being dropped due to this limit:

sum(rate(tempo_metrics_generator_registry_series_limited_total{}[1m]))

Use the following setting to update the limit:

overrides:
  metrics_generator_max_active_series: 0

Note that this value is per metrics generator. The actual max series remote written will be <# of metrics generators> * <metrics_generator_max_active_series>.

Remote write failures

For any number of reasons, the generator may fail a write to the remote write target. Use the following metrics to determine if that is happening:

sum(rate(prometheus_remote_storage_samples_failed_total{}[1m]))
sum(rate(prometheus_remote_storage_samples_dropped_total{}[1m]))
sum(rate(prometheus_remote_storage_exemplars_failed_total{}[1m]))
sum(rate(prometheus_remote_storage_exemplars_dropped_total{}[1m]))

Service graph metrics

Service graphs have additional configuration which can impact the quality of the output metrics.

Expired edges

The following metrics can be used to determine how many edges are failing to find a match.

Rate of edges that have expired without a match:

sum(rate(tempo_metrics_generator_processor_service_graphs_expired_edges{}[1m]))

Rate of all edges:

sum(rate(tempo_metrics_generator_processor_service_graphs_edges{}[1m]))

If you are seeing a large number of edges expire without a match, consider adjusting the wait setting. This controls how long the metrics generator waits to find a match before it gives up.

metrics_generator:
  processor:
    service_graphs:
      wait: 10s

Service graph max items

The service graph processor has a maximum number of edges it will track at once to limit the total amount of memory the processor uses. To determine if edges are being dropped due to this limit, check:

sum(rate(tempo_metrics_generator_processor_service_graphs_dropped_spans{}[1m]))

Use max_items to adjust the maximum amount of edges tracked:

metrics_generator:
  processor:
    service_graphs:
      max_items: 10000