Deploy Tempo with Tempo Operator on Grafana Labs

Quickstart

Fri, 03 Apr 2026 12:35:46 -0500

Quickstart

One page summary on how to start with Tempo Operator and TempoStack.

Requirements

The easiest way to start with the Tempo Operator is to use Kubernetes kind.

Deploy

To install the operator in an existing cluster, make sure you have cert-manager installed and run:

kubectl apply -f https://github.com/grafana/tempo-operator/releases/latest/download/tempo-operator.yaml

Once you have the operator deployed you need to install a storage backend. For this quick start guide, we will install MinIO as follows:

kubectl apply -f https://raw.githubusercontent.com/grafana/tempo-operator/main/minio.yaml

After minio was deployed, create a secret for MinIO in the namespace you are using:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: minio-test
stringData:
  endpoint: http://minio.minio.svc:9000
  bucket: tempo
  access_key_id: tempo
  access_key_secret: supersecret
type: Opaque
EOF

Then create Tempo CR:

kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
EOF

After create the TempoStack CR, you should see a some pods on the namespace. Wait for the stack to stabilize.

The stack deployed above is configured to receive Jaeger, Zipkin, and OpenTelemetry (OTLP) protocols. Because the Jaeger Query is enabled, you can also use the Jaeger UI to inspect the data.

To do a quick test, deploy a Job that generates some traces.

kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: tracegen
spec:
  template:
    spec:
      containers:
        - name: tracegen
          image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:latest
          command:
            - "./tracegen"
          args:
            - -otlp-endpoint=tempo-simplest-distributor:4317
            - -otlp-insecure
            - -duration=30s
            - -workers=1
      restartPolicy: Never
  backoffLimit: 4
EOF

Forward the Jaeger Query port to see the traces:

kubectl port-forward svc/tempo-simplest-query-frontend 16686:16686

Visit http://localhost:16686 to view the results.

Object storage

Fri, 03 Apr 2026 12:35:46 -0500

Object storage

Tempo Operator supports AWS S3, Azure, GCS, Minio and OpenShift Data Foundation for TempoStack object storage.

AWS S3

Requirements

Create a bucket on AWS.

Installation

Deploy the Tempo Operator to your cluster.

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-s3 \
  --from-literal=bucket="<BUCKET_NAME>" \
  --from-literal=endpoint="<AWS_BUCKET_ENDPOINT>" \
  --from-literal=access_key_id="<AWS_ACCESS_KEY_ID>" \
  --from-literal=access_key_secret="<AWS_ACCESS_KEY_SECRET>"

where tempostack-dev-s3 is the secret name.

Create an instance of TempoStack by referencing the secret name and type as s3:
YAML
```
spec:
  storage:
    secret:
      name: tempostack-dev-s3
      type: s3
```

Azure

Requirements

Create a bucket on Azure.

Installation

Deploy the Tempo Operator to your cluster.

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-azure \
  --from-literal=container="<AZURE_CONTAINER_NAME>" \
  --from-literal=account_name="<AZURE_ACCOUNT_NAME>" \
  --from-literal=account_key="<AZURE_ACCOUNT_KEY>"

where tempostack-dev-azure is the secret name.

Create an instance of TempoStack by referencing the secret name and type as azure:
YAML
```
spec:
  storage:
    secret:
      name: tempostack-dev-azure
      type: azure
```

Google Cloud Storage

Requirements

Create a project on Google Cloud Platform.
Create a bucket under same project.
Create a service account under same project for GCP authentication.

Installation

Deploy the Tempo Operator to your cluster.
Copy the service account credentials received from GCP into a file name key.json.
Create an Object Storage secret with keys bucketname and key.json as follows:
console
```
kubectl create secret generic tempostack-dev-gcs \
  --from-literal=bucketname="<BUCKET_NAME>" \
  --from-file=key.json="<PATH/TO/KEY.JSON>"
```
where tempostack-dev-gcs is the secret name, <BUCKET_NAME> is the name of bucket created in requirements step and <PATH/TO/KEY.JSON> is the file path where the key.json was copied to.
Create an instance of TempoStack by referencing the secret name and type as gcs:
YAML
```
spec:
  storage:
    secret:
      name: tempostack-dev-gcs
      type: gcs
```

MinIO

Requirements

Deploy MinIO on your cluster, e.g. using the MinIO Operator or another method.
Create a bucket on MinIO using the CLI.

Installation

Deploy the Tempo Operator to your cluster.

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-minio \
  --from-literal=bucket="<BUCKET_NAME>" \
  --from-literal=endpoint="<MINIO_BUCKET_ENDPOINT>" \
  --from-literal=access_key_id="<MINIO_ACCESS_KEY_ID>" \
  --from-literal=access_key_secret="<MINIO_ACCESS_KEY_SECRET>"

where tempostack-dev-minio is the secret name.

Create an instance of TempoStack by referencing the secret name and type as s3:
YAML
```
spec:
  storage:
    secret:
      name: tempostack-dev-minio
      type: s3
```

OpenShift Data Foundation

Requirements

Deploy the OpenShift Data Foundation on your cluster.
Create a bucket via an ObjectBucketClaim.

Installation

Deploy the Tempo Operator to your cluster.

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-odf \
  --from-literal=bucket="<BUCKET_NAME>" \
  --from-literal=endpoint="https://s3.openshift-storage.svc" \
  --from-literal=access_key_id="<ACCESS_KEY_ID>" \
  --from-literal=access_key_secret="<ACCESS_KEY_SECRET>"

where tempostack-dev-odf is the secret name. You can copy the values for BUCKET_NAME, ACCESS_KEY_ID and ACCESS_KEY_SECRET from your ObjectBucketClaim’s accompanied secret.

Create an instance of TempoStack by referencing the secret name and type as s3:
YAML
```
spec:
  storage:
    secret:
      name: tempostack-dev-odf
      type: s3
```

Enable multi-tenancy

Fri, 03 Apr 2026 12:35:46 -0500

Enable multi-tenancy

Tempo is a multi-tenant distributed tracing backend. It supports multi-tenancy through the use of a header: X-Scope-OrgID. Refer to multi-tenancy docs for more details. This document outlines how to deploy and use multi-tenant Tempo with the Operator.

Multi-tenancy without authentication

The following Kubernetes Custom Resource (CR) deploys a multi-tenant Tempo instance.

Note
Jaeger query is not tenant aware and therefore is not supported in this configuration.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  tenants: {}
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m

OIDC authentication with static RBAC

On Kubernetes, a multi-tenant Tempo instance uses OIDC authentication and static RBAC authorization defined in the CR. The instance should be accessed through service tempo-simplest-gateway, which handles authentication and authorization. The service exposes Jaeger query API and OpenTelemetry gRPC (OTLP) for trace ingestion. The Jaeger UI can be accessed at http://<exposed gateway service>:8080/api/traces/v1/<tenant-name>/search.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
    gateway:
      enabled: true
  storage:
    secret:
      type: s3
      name: minio-test
  storageSize: 200M
  tenants:
    mode: static
    authentication:
      - tenantName: test-oidc
        tenantId: test-oidc
        oidc:
          issuerURL: http://dex.default.svc.cluster.local:30556/dex
          redirectURL: http://tempo-simplest-gateway.default.svc.cluster.local:8080/oidc/test-oidc/callback
          usernameClaim: email
          secret:
            name: oidc-test
    authorization:
      roleBindings:
      - name: "test"
        roles:
        - read-write
        subjects:
        - kind: user
          name: "admin@example.com"
      roles:
      - name: read-write
        permissions:
        - read
        - write
        resources:
        - traces
        tenants:
        - test-oidc

The secret oidc-test defines fields clientID, clientSecret and issuerCAPath.
The RBAC gives tenant test-oidc read and write access for traces.

OpenShift

On OpenShift, the authentication and authorization does not require any third-party service dependencies. The authentication uses OpenShift OAuth (the user is redirected to the OpenShift login page) and authorization is handled through SubjectAccessReview (SAR).

The instance should be accessed through service tempo-simplest-gateway, which handles authentication and authorization. The service exposes Jaeger query API and OpenTelemetry gRPC (OTLP) for trace ingestion. The Jaeger UI can be accessed at http://<exposed gateway service>:8080/api/traces/v1/<tenant-name>/search.

apiVersion: tempo.grafana.com/v1alpha1
kind:  TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: object-storage
      type: s3
  storageSize: 1Gi
  tenants:
    mode: openshift
    authentication:
      - tenantName: dev
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa"
      - tenantName: prod
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"
  template:
    gateway:
      enabled: true
    queryFrontend:
      jaegerQuery:
        enabled: true

ClusterRole and ClusterRoleBinding objects have to be created to enable reading and writing the data.

RBAC for reading the data

The following RBAC gives authenticated users access to read trace data for dev and prod tenants.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-reader
rules:
  - apiGroups:
      - 'tempo.grafana.com'
    resources:
      - dev
      - prod
    resourceNames:
      - traces
    verbs:
      - 'get'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-reader
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated

RBAC for writing data

The following RBAC gives service account otel-collector write access for trace data for dev tenant.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: otel
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-write
rules:
  - apiGroups:
      - 'tempo.grafana.com'
    resources:
      - dev
    resourceNames:
      - traces
    verbs:
      - 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-write
subjects:
  - kind: ServiceAccount
    name: otel-collector
    namespace: otel

OpenTelemetry collector CR configuration with authentication for dev tenant.

spec:
    serviceAccount: otel-collector
    config: |
        extensions:
          bearertokenauth:
            filename: "/var/run/secrets/kubernetes.io/serviceaccount/token"
        exporters:
          # Export the dev tenant traces to a Tempo instance
          otlp/dev:
            endpoint: tempo-simplest-gateway.tempo.svc.cluster.local:8090
            tls:
              insecure: false
              ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
            auth:
              authenticator: bearertokenauth
            headers:
              X-Scope-OrgID: "dev"

Monitor TempoStack instances

Fri, 03 Apr 2026 12:35:46 -0500

Monitor TempoStack instances

The configuration for monitoring TempoStack instances is exposed in the CR:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
spec:
  observability:
    metrics:
      createServiceMonitors: true
      createPrometheusRules: true
    tracing:
      sampling_fraction: 1.0
      jaeger_agent_endpoint: localhost:6831

Configure distributed tracing of operands

All Tempo components as well as the Tempo Gateway support the export of traces in thrift_compact format.

Deploy OpenTelemetry collector sidecar

To deploy the OpenTelemetry collector, follow these steps:

Install OpenTelemetry Operator into the cluster.
Create an OpenTelemetryCollector CR that receives trace data in Jaeger Thrift format and exports data via OTLP to the desired trace backend.
Optional: Deploy tracing backend to store trace data.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: sidecar-for-tempo
spec:
  mode: sidecar
  config: |
    receivers:
      jaeger:
        protocols:
          thrift_compact:

    exporters:
      otlp:
        endpoint: <otlp-endpoint>:4317
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          receivers: [jaeger]
          exporters: [otlp]

Send trace data to OpenTelemetry sidecar

Finally, create a TempoStack instance that sets jaeger_agent_endpoint to report trace data to the localhost. The Tempo operator sets the OpenTelemetry inject annotation sidecar.opentelemetry.io/inject": "true to all TempoStack pods. The OpenTelemetry Operator will recognize the annotation, and it will inject a sidecar into all TempoStack pods.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simple-stack
spec:
  template:
    queryFrontend:
      jaegerQuery:
        enabled:
  storage:
    secret:
      type: s3
      name: minio-test
  storageSize: 200M
  observability:
    tracing:
      sampling_fraction: "1.0"
      jaeger_agent_endpoint: localhost:6831