Tempo HTTP API on Grafana Labs

Metrics summary API

Fri, 03 Apr 2026 20:44:05 +0000

Metrics summary API

Warning
The Metrics summary API is an experimental feature that is disabled by default. To enable it, adjust your configuration as suggested below.

This document explains how to use the metrics summary API in Tempo. This API returns RED metrics (span count, erroring span count, and latency information) for kind=server spans sent to Tempo in the last hour, grouped by a user-specified attribute.

Configuration

To enable the experimental metrics summary API, you must turn on the local blocks processor in the metrics generator. Be aware that the generator will use considerably more resources, including disk space, if this is enabled:

overrides:
  metrics_generator_processors: [..., 'local-blocks']

Request

To make a request to this API, use the following endpoint on the query-frontend:

GET http://<tempo>/api/metrics/summary

Query Parameters

All query parameters must be URL-encoded to preserve non-URL-safe characters in the query such as &.

Name	Examples	Definition	Required?
`q`	`{ resource.service.name = "foo" && span.http.status_code != 200 }`	The TraceQL query with full syntax. All spans matching this query are included in the calculations. Any valid TraceQL query is supported.	Yes
`groupBy`	`name` `.foo` `resource.namespace` `span.http.url,span.http.status_code`	The TraceQL value(s) to group by. Any valid intrinsic or attribute with scope. To group by multiple values use a comma-delimited list.	Yes
`start`	1672549200	Start of time range in unix seconds. If not specified, then all recent data is queried.	No
`end`	1672549200	End of the time range in unix seconds. If not specified, then all recent data is queried.	No

Example:

curl "$URL/api/metrics/summary" --data-urlencode 'q={resource.service.name="checkout-service"}' --data-urlencode 'groupBy=name'

Response

The Tempo response is a SpanMetricsSummary object defined in tempo.proto, relevant section pasted below:

message SpanMetricsSummaryResponse {
  repeated SpanMetricsSummary summaries = 1;
}

message SpanMetricsSummary {
  uint64 spanCount = 1;
  uint64 errorSpanCount = 2;
  TraceQLStatic static = 3;
  uint64 p99 = 4;
  uint64 p95 = 5;
  uint64 p90 = 6;
  uint64 p50 = 7;
}

message TraceQLStatic {
  int32 type = 1;
  int64 n = 2;
  double f = 3;
  string s = 4;
  bool b = 5;
  uint64 d = 6;
  int32 status = 7;
  int32 kind = 8;
}

The response is returned as JSON following standard protobuf->JSON mapping rules.

Note
The uint64 fields cannot be fully expressed by JSON numeric values so the fields are serialized as strings.

Example:

{
   "summaries": [
       {
           "spanCount": "20",
           "series" : [
               {
                   "key": ".attr1",
                   "value": {
                       "type": 5,
                       "s": "foo"
                   },
               },
               ...
           ],
           "p99": "68719476736",
           "p95": "1073741824",
           "p90": "1017990479",
           "p50": "664499239"
       },

Field	Notes
`summaries`	The list of metrics per group.
`.spanCount`	Number of spans in this group.
`.errorSpanCount`	Number of spans with `status`=`error`. (This field will not be present if the value is `0`.)
`.series`	The unique values for this group. A key/value pair will be returned for each entry in `groupBy`.
`.key`	Key name.
`.value`	Value with TraceQL underlying type.
`.type`	Data type enum defined here (This field will not be present if the value is `0`.) 0 = nil 3 = integer 4 = float 5 = string 6 = bool 7 = duration 8 = span status 9 = span kind
`.n`	Populated if this is an integer value.
`.s`	Populated if this is a string value.
`.f`	Populated if this is a float value.
`.b`	Populated if this is a boolean value.
`.d`	Populated if this is a duration value.
`.status`	Populated if this is a span status value.
`.kind`	Populated if this is a span kind value.
`.p99`	The p99 latency of this group in nanoseconds.
`.p95`	The p95 latency of this group in nanoseconds.
`.p90`	The p90 latency of this group in nanoseconds.
`.p50`	The p50 latency of this group in nanoseconds.

Push spans with HTTP

Fri, 03 Apr 2026 20:44:05 +0000

Push spans with HTTP

Sometimes using a tracing system is intimidating because it seems like you need complex application instrumentation or a span ingestion pipeline in order to push spans. This guide aims to show an extremely basic technique for pushing spans with HTTP/JSON from a Bash script using the OpenTelemetry receiver.

Before you begin

This procedure uses an example Docker Compose setup to run Tempo, so you do not need an existing installation. The Docker image also includes a Grafana container which will allow us to visualize traces.

To use this procedure, you need to have Docker and docker compose installed.

Start Tempo using the quick start

Use the instructions in the Quick start for Tempo documentation to start a local instance of Tempo and Grafana.

Push spans with OTLP

Now that Tempo is running and listening on port 4318 for OTLP spans, let’s push a span to it using curl.

Before you can use this example, you need to update the start and end time as instructed.

curl -X POST -H 'Content-Type: application/json' http://localhost:4318/v1/traces -d '
{
	"resourceSpans": [{
    	"resource": {
        	"attributes": [{
            	"key": "service.name",
            	"value": {
                	"stringValue": "my.service"
            	}
        	}]
    	},
    	"scopeSpans": [{
        	"scope": {
            	"name": "my.library",
            	"version": "1.0.0",
            	"attributes": [{
                	"key": "my.scope.attribute",
                	"value": {
                    	"stringValue": "some scope attribute"
                	}
            	}]
        	},
        	"spans": [
        	{
            	"traceId": "5B8EFFF798038103D269B633813FC700",
            	"spanId": "EEE19B7EC3C1B100",
            	"name": "I am a span!",
            	"startTimeUnixNano": 1689969302000000000,
            	"endTimeUnixNano": 1689970000000000000,
            	"kind": 2,
            	"attributes": [
            	{
                	"key": "my.span.attr",
                	"value": {
                    	"stringValue": "some value"
                	}
            	}]
        	}]
    	}]
	}]
}'

Note that the startTimeUnixNano field is in nanoseconds and can be obtained by any tool that provides the epoch date in nanoseconds (for example, under Linux, date +%s%8N). The endTimeUnixNano field is also in nanoseconds, where 100000000 nanoseconds is 100 milliseconds.

Copy and paste the curl command into a text editor.
Replace startTimeUnixNano and endTimeUnixNano with current values for the last 24 hours to allow you to search for them using a 24 hour relative time range. You can get this in seconds and milliseconds from the following link. Multiple the milliseconds value by 1,000,000 to turn it into nanoseconds. You can do this from a bash terminal with:
Bash
```
echo $((<epochTimeMilliseconds> * 1000000))
```
Copy the updated curl command to a terminal window and run it.
View the trace in Grafana:
1. Open a browser window to http://localhost:3000.
2. Open the Explorer page and select the Tempo data source.
3. Select the Search query type.
4. Select Run query to list available traces.
5. Select the trace ID (yellow box) to view details about the trace and its spans.

Retrieve traces

The easiest way to get the trace is to execute a simple curl command to Tempo. The returned format is OTLP.

Replace the trace ID in the curl command with the trace ID that was generated from the push. This information is is in the data that’s sent with the curl. You could use Grafana’s Explorer page to find this, as shown in the previous section.

curl http://localhost:3200/api/traces/5b8efff798038103d269b633813fc700

{"batches":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]},"scopeSpans":[{"scope":{"name":"my.library","version":"1.0.0"},"spans":[{"traceId":"W47/95gDgQPSabYzgT/HAA==","spanId":"7uGbfsPBsQA=","name":"I am a span!","kind":"SPAN_KIND_SERVER","startTimeUnixNano":"1689969302000000000","endTimeUnixNano":"1689970000000000000","attributes":[{"key":"my.span.attr","value":{"stringValue":"some value"}}],"status":{}}]}]}]}

Copy and paste the updated curl command into a terminal window.

Use TraceQL to search for a trace

Alternatively, you can also use TraceQL to search for the trace that was pushed. You can search by using the unique trace attributes that were set:

curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ .service.name = "my.service" }'

{"traces":[{"traceID":"5b8efff798038103d269b633813fc700","rootServiceName":"my.service","rootTraceName":"I am a span!","startTimeUnixNano":"1694718625557000000","durationMs":10000,"spanSet":{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1},"spanSets":[{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1}]}],"metrics":{"inspectedBytes":"292781","completedJobs":1,"totalJobs":1}}

To format this in a more human-readable output, consider using a tool such as jq, which lets you to run the same curl command and pipe it to jq to format the block. For example:

curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ .service.name = "my.service" }' | jq

Spans from everything!

Tracing is not limited to enterprise languages with complex frameworks. As you can see it’s easy to store and track events from your js, python or bash scripts. You can use Tempo/distributed tracing today to trace CI pipelines, long running bash processes, python data processing flows, or anything else you can think of.

Happy tracing!