Documentationbreadcrumb arrow Grafana Tempobreadcrumb arrow Troubleshootbreadcrumb arrow Queryingbreadcrumb arrow Response larger than the max
Open source

Response larger than the max

The error message is similar to the following:

500 Internal Server Error Body: response larger than the max (<size> vs <limit>)

This error indicates that the response received or sent is too large. This can happen in multiple places, but it’s most commonly seen in the query path, with messages between the querier and the query frontend.

Solutions

Query frontend streaming responses

For gRPC streaming search, tag, and TraceQL metrics responses, the query frontend can split streamed diffs and final responses into smaller packets. Use query_frontend.max_grpc_streaming_packet_size to set the target packet size in bytes. The default is 2097152 bytes, or 2 MiB.

YAML 
query_frontend:
  max_grpc_streaming_packet_size: 2097152

This setting applies to gRPC streaming responses from the query frontend. It doesn’t increase the process-wide gRPC message limit for other Tempo component communication.

Tempo server (general)

Tempo components communicate with each other via gRPC requests. To increase the maximum message size, you can increase the gRPC message size limit in the server block.

YAML 
server:
  grpc_server_max_recv_msg_size: <size>
  grpc_server_max_send_msg_size: <size>

The server config block is not synchronized across components. Most likely you will need to increase the message size limit in multiple components.

Querier

Additionally, querier workers can be configured to use a larger message size limit.

YAML 
querier:
    frontend_worker:
        grpc_client_config:
            max_send_msg_size: <size>

Ingestion

Lastly, message size is also limited in ingestion and can be modified in the distributor block.

YAML 
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          max_recv_msg_size_mib: <size>