You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by "binarycat-dremio (via GitHub)" <gi...@apache.org> on 2026/07/22 20:50:10 UTC

[D] [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation [arrow-adbc]

GitHub user binarycat-dremio created a discussion: [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation

Hi team,

I'd like to clarify the intended tracing model for ADBC FlightSQL.

In a typical DB-API flow like:

```python
with dbapi.connect(...) as conn:
    with conn.cursor() as cur:
        cur.execute(sql1)
        cur.fetchall()

        cur.execute(sql2)
        cur.fetchall()
```

there seem to be a few possible interpretations of how tracing should behave, and I'm not sure which one is the expected design.

Questions:
1. Should all query-related spans created from a single connection be grouped under the `FlightSQLDatabase.Open` span/trace?
2. Or should each statement / `execute(...)` operation start its own logical trace?
3. If query operations are intentionally separate, is the preferred correlation mechanism something like a stable `connection_id` attribute on statement/cursor spans rather than parent-child trace linkage?

From an observability perspective, I can see two reasonable models:

- Connection-oriented model: `Open` is the parent context, and later `Prepare` / `ExecuteQuery` spans for that connection are grouped beneath it.
- Query-oriented model: each logical query execution gets its own trace/span tree, and cross-query correlation happens through attributes such as `connection_id`.

I'm mainly trying to understand the intended behavior before proposing any broader change.

Related:
- Issue: https://github.com/apache/arrow-adbc/issues/4568
- PR: https://github.com/apache/arrow-adbc/pull/4570


GitHub link: https://github.com/apache/arrow-adbc/discussions/4572

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org


Re: [D] [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation [arrow-adbc]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
GitHub user zeroshade added a comment to the discussion: [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation

> 1. Should all query-related spans created from a single connection be grouped under the FlightSQLDatabase.Open span/trace?

I don't know about grouping them all under the same span/trace, but if we don't group them under the same span/trace then we should at least connect with them with some sort of ID metadata.

> 2. Or should each statement / execute(...) operation start its own logical trace?

I think it might make sense for everything under a given statement being one logical trace, most definitely.

> 3. If query operations are intentionally separate, is the preferred correlation mechanism something like a stable connection_id attribute on statement/cursor spans rather than parent-child trace linkage?

My personal preference would be a stable connection_id attribute on the statement/cursor spans rather than parent-child trace linkage since it's common that a connection might be used for multiple queries/statements.

GitHub link: https://github.com/apache/arrow-adbc/discussions/4572#discussioncomment-17756652

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org


Re: [D] [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation [arrow-adbc]

Posted by "binarycat-dremio (via GitHub)" <gi...@apache.org>.
GitHub user binarycat-dremio added a comment to the discussion: [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation

Thanks for sharing your opinion. Given the current discussion and the OTel DB span conventions, I think it may make sense to introduce a dedicated span attribute for connection-level correlation, for example `adbc.connection.id`.

The intent would not be to force all operations from a single connection into one trace, but to provide an explicit correlation key for users who want to group/query spans originating from the same logical connection.

That seems consistent with the query-oriented tracing model: individual statements/operations remain separate spans/traces, while users who need stronger correlation can opt into it via span attributes rather than implicit parent-child linkage.
If this direction sounds reasonable, I can open a follow-up issue proposing `adbc.connection.id` as an opaque, stable-for-the-lifetime-of-the-connection attribute.

GitHub link: https://github.com/apache/arrow-adbc/discussions/4572#discussioncomment-17796062

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org


Re: [D] [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation [arrow-adbc]

Posted by "binarycat-dremio (via GitHub)" <gi...@apache.org>.
GitHub user binarycat-dremio edited a discussion: [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation

Hello colleagues,

I'd like to clarify the intended tracing model for ADBC FlightSQL.

In a typical DB-API flow like:

```python
with dbapi.connect(...) as conn:
    with conn.cursor() as cur:
        cur.execute(sql1)
        cur.fetchall()

        cur.execute(sql2)
        cur.fetchall()
```

there seem to be a few possible interpretations of how tracing should behave, and I'm not sure which one is the expected design.

Questions:
1. Should all query-related spans created from a single connection be grouped under the `FlightSQLDatabase.Open` span/trace?
2. Or should each statement / `execute(...)` operation start its own logical trace?
3. If query operations are intentionally separate, is the preferred correlation mechanism something like a stable `connection_id` attribute on statement/cursor spans rather than parent-child trace linkage?

From an observability perspective, I can see two reasonable models:

- Connection-oriented model: `Open` is the parent context, and later `Prepare` / `ExecuteQuery` spans for that connection are grouped beneath it.
- Query-oriented model: each logical query execution gets its own trace/span tree, and cross-query correlation happens through attributes such as `connection_id`.

I'm mainly trying to understand the intended behavior before proposing any broader change.

Related:
- Issue: https://github.com/apache/arrow-adbc/issues/4568
- PR: https://github.com/apache/arrow-adbc/pull/4570

cc: @lidavidm @xborder @zeroshade @Mandukhai-Alimaa @amoeba 

---
The current state with splitted traces.

<img width="1439" height="1059" alt="Screenshot 2026-07-22 at 19 18 18" src="https://github.com/user-attachments/assets/c4702caf-2f8c-4898-a7ff-4c384685e49c" />


---
The possible implementation with grouped query spans under Open entry.

<img width="1439" height="1059" alt="Screenshot 2026-07-22 at 19 17 31" src="https://github.com/user-attachments/assets/26c7fe76-a7a1-4244-977f-a73b042d1b37" />


GitHub link: https://github.com/apache/arrow-adbc/discussions/4572

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org


Re: [D] [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
GitHub user lidavidm added a comment to the discussion: [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation

> * Should all query-related spans created from a single connection be grouped under the `FlightSQLDatabase.Open` span/trace?

It appears OpenTelemetry has only specified what to do for individual calls to the database.
https://opentelemetry.io/docs/specs/semconv/db/database-spans/#database-client-span-duration

> * Or should each statement / `execute(...)` operation start its own logical trace?

By the above spec, yes.

I think we should align with the official spec wherever possible.

> * If query operations are intentionally separate, is the preferred correlation mechanism something like a stable `connection_id` attribute on statement/cursor spans rather than parent-child trace linkage?

It's not clear to me that this is expected of clients in the first place.

GitHub link: https://github.com/apache/arrow-adbc/discussions/4572#discussioncomment-17776255

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org


Re: [D] [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation [arrow-adbc]

Posted by "binarycat-dremio (via GitHub)" <gi...@apache.org>.
GitHub user binarycat-dremio edited a discussion: [DISCUSS] FlightSQL tracing: expected relationship between Open, statement/query spans, and connection-level correlation

Hi team,

I'd like to clarify the intended tracing model for ADBC FlightSQL.

In a typical DB-API flow like:

```python
with dbapi.connect(...) as conn:
    with conn.cursor() as cur:
        cur.execute(sql1)
        cur.fetchall()

        cur.execute(sql2)
        cur.fetchall()
```

there seem to be a few possible interpretations of how tracing should behave, and I'm not sure which one is the expected design.

Questions:
1. Should all query-related spans created from a single connection be grouped under the `FlightSQLDatabase.Open` span/trace?
2. Or should each statement / `execute(...)` operation start its own logical trace?
3. If query operations are intentionally separate, is the preferred correlation mechanism something like a stable `connection_id` attribute on statement/cursor spans rather than parent-child trace linkage?

From an observability perspective, I can see two reasonable models:

- Connection-oriented model: `Open` is the parent context, and later `Prepare` / `ExecuteQuery` spans for that connection are grouped beneath it.
- Query-oriented model: each logical query execution gets its own trace/span tree, and cross-query correlation happens through attributes such as `connection_id`.

I'm mainly trying to understand the intended behavior before proposing any broader change.

Related:
- Issue: https://github.com/apache/arrow-adbc/issues/4568
- PR: https://github.com/apache/arrow-adbc/pull/4570


---
The current state with splitted traces.

<img width="1439" height="1059" alt="Screenshot 2026-07-22 at 19 18 18" src="https://github.com/user-attachments/assets/c4702caf-2f8c-4898-a7ff-4c384685e49c" />


---
The possible implementation with grouped query spans under Open entry.

<img width="1439" height="1059" alt="Screenshot 2026-07-22 at 19 17 31" src="https://github.com/user-attachments/assets/26c7fe76-a7a1-4244-977f-a73b042d1b37" />


GitHub link: https://github.com/apache/arrow-adbc/discussions/4572

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org