---
title: "Link traces and database queries | Database Observability documentation"
description: "Identify and trace queries back to your application code through trace correlation."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Link traces and database queries

Trace linkage connects your application traces to the database queries they run, so you can move between the two in either direction:

- **From a trace to a query.** In Explore or Traces Drilldown, a database span can show a Database Observability link. Selecting it takes you to that query in Database Observability, where you can see its samples, latency, and history.
- **From a query back to a trace.** When a query sample carries trace context, you can jump from the query back to the trace of the request that ran it.

## Why this is useful

When an application request is slow, a trace usually shows that a database call was slow, but not *why*. Trace linkage lets you cross from that span into the query’s full picture in Database Observability: whether it’s habitually slow, what it runs against, and how often it executes. Going the other way, when you find a problem query, you can trace it back to the user-facing request that triggered it, instead of guessing which part of the application is responsible.

## Before you begin

Trace linkage requires an application instrumented with an OpenTelemetry SDK that sends traces to Grafana Cloud.

The database spans must include:

- Query text in `db.query.text` or `db.statement`
- A supported database system in `db.system.name` or `db.system`: `mysql` or `postgresql`

## What to expect

Trace linkage is best-effort correlation, not a guaranteed link on every span. Database Observability matches in two ways:

- **Query text match** is the default and needs no additional application changes. It matches the span’s query text against the queries Database Observability has collected.
- **Exact trace match** is more precise but requires sqlcommenter instrumentation. It can link to the single query execution behind a specific span using its trace ID and span ID.

The following video shows an exact trace match opening the corresponding query in Database Observability.

When Database Observability matches a span to a query, it opens the Query samples tab in the Query details view. If multiple database instances contain the query, you may first be asked to select an instance.

### How to enable exact trace matching

Exact trace match is an advanced optional enhancement to trace linkage matching. Exact trace match only improves the precision of the matches, not how often a link resolves.

- Enable [sqlcommenter](https://open-telemetry.github.io/opentelemetry-sqlcommenter/) in your application’s OpenTelemetry SDK instrumentation. sqlcommenter appends a W3C `traceparent` comment to SQL queries. For the setting to turn on in each language, refer to [Enable sqlcommenter in your application](https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-infrastructure/database-observability/monitor/link-traces/#enable-sqlcommenter-in-your-application).
- Use Alloy 1.17.0 or later. If you can’t upgrade from an earlier version, set `disable_query_redaction = true` in the `query_samples` block as a fallback.

## Enable sqlcommenter in your application

sqlcommenter appends a comment to each SQL statement before your application sends it to the database. The comment holds `key='value'` pairs. Exact trace match uses the `traceparent` pair, which is the W3C trace context of the request:

SQL ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```sql
SELECT * FROM orders WHERE customer_id = ? /*traceparent='00-5bd66ef5095369c7b0d1f8f4bd33716a-c532cb4098ac3dd2-01'*/
```

The `traceparent` value carries the trace ID and span ID that was active when the comment was added.

The OpenTelemetry semantic conventions treat sqlcommenter as an opt-in feature, so no instrumentation adds the comment by default.

The following snippets show only the setting that adds the comment. Keep the rest of your OpenTelemetry setup as it is.

### Python

The SQLAlchemy instrumentation creates the spans and adds the comment. Set `enable_commenter=True` when you instrument the engine. The `traceparent` pair is included by default. `commenter_options` only lets you opt out of pairs you don’t want.

Python ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```python
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

SQLAlchemyInstrumentor().instrument(
    engine=engine,
    enable_commenter=True,
)
```

Call it once per engine, right after `create_engine()`. The DB-API driver instrumentations (`Psycopg2Instrumentor`, `PsycopgInstrumentor`, `MySQLClientInstrumentor`, and `PyMySQLInstrumentor`) accept `enable_commenter=True` as well.

### Node.js

The `mysql2` and `pg` instrumentations create the spans and take a built-in sqlcommenter option. Set `addSqlCommenterCommentToQueries: true` on each.

JavaScript ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```javascript
const { MySQL2Instrumentation } = require('@opentelemetry/instrumentation-mysql2');
const { PgInstrumentation } = require('@opentelemetry/instrumentation-pg');

const instrumentations = [
  new MySQL2Instrumentation({ addSqlCommenterCommentToQueries: true }),
  new PgInstrumentation({ addSqlCommenterCommentToQueries: true }),
];
```

The SDK has to patch `mysql2` and `pg` before your application code loads them, so register the instrumentations in a file you preload, for example with `node -r ./otel.js`. These instrumentations don’t add a comment to a query that already contains `--` or `/* */`, even inside a string literal.

### Go

`otelsql` creates a database client span per query and has a built-in `WithSQLCommenter` option that appends the `traceparent` comment using the global text map propagator. The following example uses the `mysql` driver. For PostgreSQL, use the `postgres` driver and `semconv.DBSystemNamePostgreSQL`.

Go ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```go
import (
    "github.com/XSAM/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
)

db, err := otelsql.Open("mysql", dsn,
    otelsql.WithAttributes(semconv.DBSystemNameMySQL),
    otelsql.WithSQLCommenter(true),
)
```

Make sure the global propagator includes `propagation.TraceContext{}`, for example with `otel.SetTextMapPropagator(propagation.TraceContext{})`.

### Java

The OpenTelemetry Java agent (2.21.0 or later) includes an experimental JDBC sqlcommenter option. Enable it with a system property or environment variable:

Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```bash
-Dotel.instrumentation.jdbc.experimental.sqlcommenter.enabled=true
# or via environment variable:
export OTEL_INSTRUMENTATION_JDBC_EXPERIMENTAL_SQLCOMMENTER_ENABLED=true
```

If you are on an older agent version or prefer to add the comment in your data access layer, use a helper that reads the current span, URL-encodes the value per the sqlcommenter specification, and appends the `traceparent` to the SQL string before passing it to JDBC:

java ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```java
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanContext;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;

public static String annotate(String sql) {
    SpanContext ctx = Span.current().getSpanContext();
    if (!ctx.isValid()) {
        return sql;
    }
    String traceparent = "00-" + ctx.getTraceId() + "-" + ctx.getSpanId() + "-" + ctx.getTraceFlags().asHex();
    String encoded = URLEncoder.encode(traceparent, StandardCharsets.UTF_8);
    return sql + " /*traceparent='" + encoded + "'*/";
}
```

Values in the comment must be URL-encoded and single-quoted, as the [sqlcommenter specification](https://google.github.io/sqlcommenter/spec/) requires. The `sqlcommenter` project also publishes Hibernate and Spring integrations for Java.

### Ruby

The OpenTelemetry `mysql2` (`>= 0.34.1`), `pg` (`>= 0.37.1`), and `trilogy` (`>= 0.69.1`) instrumentation gems support sqlcommenter propagation natively. Set `propagator: 'tracecontext'` when configuring the SDK:

ruby ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```ruby
require 'mysql2'
require 'pg'

OpenTelemetry::SDK.configure do |c|
  c.use 'OpenTelemetry::Instrumentation::Mysql2', propagator: 'tracecontext'
  c.use 'OpenTelemetry::Instrumentation::PG', propagator: 'tracecontext'
end
```

In Ruby on Rails, Active Record executes queries through those underlying driver gems (`pg`, `mysql2`, or `trilogy`). If your Rails initializer uses `use_all`, pass `propagator: 'tracecontext'` for your database adapter in `config/initializers/opentelemetry.rb`:

ruby ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```ruby
OpenTelemetry::SDK.configure do |c|
  c.use_all(
    'OpenTelemetry::Instrumentation::Mysql2' => { propagator: 'tracecontext' },
    'OpenTelemetry::Instrumentation::PG' => { propagator: 'tracecontext' },
    'OpenTelemetry::Instrumentation::Trilogy' => { propagator: 'tracecontext' },
  )
end
```

Alternatively, in Rails 7+ you can use Active Record’s built-in [`QueryLogs`](https://api.rubyonrails.org/classes/ActiveRecord/QueryLogs.html) (which uses the sqlcommenter format by default in Rails 7.1+) and add a `traceparent` tag in `config/application.rb`:

ruby ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```ruby
config.active_record.query_log_tags_enabled = true
config.active_record.query_log_tags = [
  :application,
  :controller,
  :action,
  {
    traceparent: -> {
      ctx = ::OpenTelemetry::Trace.current_span.context
      next unless ctx.valid?
      flags = format("%02x", ctx.trace_flags.sampled? ? 1 : 0)
      "00-#{ctx.hex_trace_id}-#{ctx.hex_span_id}-#{flags}"
    }
  }
]
```

Require the `mysql2` or `pg` gem before your ORM (such as Sequel) loads it so the OpenTelemetry patches apply.

### Considerations

Some application and database settings affect whether the comment reaches Alloy intact and whether an exact match finds the sample.

- **Which span ID the comment carries.** Exact trace match keys on the trace ID and the span ID. The span ID in the comment is the span that was active when the comment was added. When the comment is added before the database client span opens, for example in a helper called at the query call site or an outer `database/sql` wrapper, the comment carries the parent span’s ID and the match resolves to the trace rather than to the database span. Instrumentations that add the comment from inside the database client span, such as the built-in Python, Node.js, `otelsql` (`WithSQLCommenter`), Java agent, and Ruby instrumentation options above, carry the database span’s ID.
- **Prepared statements.** Server-side prepared statements (where the database parses the SQL once on `PREPARE` and subsequent `EXECUTE` calls only pass a statement ID and parameters, such as Node.js `mysql2`’s `execute()` or Go’s `db.PrepareContext()`) do not carry a per-execution `traceparent`: the comment is either skipped or fixed to the trace that originally prepared the statement.
- **Long statements.** The database stores a limited length of statement text for each sample, and sqlcommenter appends `traceparent` at the end. If a statement exceeds that limit, the comment is truncated. Refer to the [MySQL troubleshooting guide](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/database-observability/troubleshoot/mysql/#confirm-sql-text-length-for-samples-optional) and [PostgreSQL troubleshooting guide](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/database-observability/troubleshoot/postgres/#confirm-track_activity_query_size-value) to check and update the database engine settings correctly.
- **Span attribute sanitization.** OpenTelemetry SDKs commonly sanitize or omit literals and comments in `db.query.text` / `db.statement` on the span (for example, replacing quoted values with `?`). Database Observability normalizes span query text and reads `traceparent` from the database sample collected by Alloy, so span sanitization doesn’t affect trace linkage.

The [fullstack-o11y-examples](https://github.com/grafana/fullstack-o11y-examples) repository shows a working setup for each language.

## Linking a query sample back to a trace

Database Observability shows **View trace in Explore** links directly on query sample tooltips. Selecting it opens Explore Traces with the trace ID and span ID from the sample’s `traceparent`.

The link requires:

- a W3C `traceparent` comment appended to your SQL text with [sqlcommenter](https://open-telemetry.github.io/opentelemetry-sqlcommenter/). Refer to [Enable sqlcommenter in your application](https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-infrastructure/database-observability/monitor/link-traces/#enable-sqlcommenter-in-your-application).
- Alloy 1.17.0 or later

On older versions of Alloy, when a query sample includes a `traceparent`, you can copy the trace ID from the sample and search for it in Explore or Traces Drilldown.

## Troubleshoot trace matching

If selecting a link doesn’t take you to a specific query, use the checks below for the type of matching you expect.

### If query text matching doesn’t find a query

Query text matching relies on the queries Database Observability has already collected. If a link sends you to Queries overview instead of a specific query, check the following:

- **Confirm the query text is collected.** Database Observability refreshes the queries based on the sampling rate configured in Alloy, so a query it hasn’t collected yet can’t be matched. Wait for the next collection and try again. To check whether a query is available for matching, query the logs under `op="query_association"` in your logs data source and confirm your query text appears.
- **The text must match after normalization.** Matching uses normalized query text, so unusual literals or formatting can prevent a match.

### If exact matching doesn’t find a sample

Exact matching only succeeds if Alloy sampled the specific execution behind the span and captured its `traceparent`. Alloy samples running queries periodically rather than capturing every execution, so most executions are never sampled and exact matching remains best-effort even when sqlcommenter is configured correctly.

- **Confirm the `traceparent` is logged.** In your logs data source, query the logs under `op="query_sample"` and confirm the log lines include a `traceparent` field. If it’s missing, sqlcommenter isn’t appending the `traceparent` or Alloy is below 1.17.0. A matching log line looks like this:

logfmt ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```logfmt
level="info" datname="shop" pid="1089" leader_pid="" user="shop-api" app="" client="203.0.113.10:54321" backend_type="client backend" state="idle" xid="0" xmin="0" xact_time="" query_time="4.294ms" queryid="5810693783667443671" traceparent="00-5bd66ef5095369c7b0d1f8f4bd33716a-c532cb4098ac3dd2-01"
```

- **Lower `collect_interval` with care.** The `query_samples` block has a `collect_interval` option that controls how often Alloy samples queries from the database. A shorter interval samples more executions and raises the chance that the one you selected was captured, but it cannot ensure every execution is captured from the database. Shorter intervals also increase telemetry volume and add load to your database, so lower `collect_interval` gradually and watch the impact. For broader coverage, rely on query text matching.

### Use Adaptive Traces to keep query spans

If you use Adaptive Traces, you can choose to retain traces that contain database query spans. Retaining those traces keeps the trace available, so you can reliably match from a query sample back to its trace.

Configure an Adaptive Traces policy of type `and`. The policy retains traces only when a span has both a supported database system and query text:

JSON ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "and_sub_policy": [
    {
      "name": "db-system-check",
      "type": "ottl_condition",
      "ottl_condition": {
        "span": [
          "attributes[\"db.system.name\"] == \"mysql\" or attributes[\"db.system.name\"] == \"postgresql\" or attributes[\"db.system\"] == \"mysql\" or attributes[\"db.system\"] == \"postgresql\""
        ],
        "error_mode": "ignore"
      }
    },
    {
      "name": "has-query-text",
      "type": "ottl_condition",
      "ottl_condition": {
        "span": [
          "attributes[\"db.query.text\"] != nil or attributes[\"db.statement\"] != nil"
        ],
        "error_mode": "ignore"
      }
    }
  ]
}
```
