You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "aiguofer (via GitHub)" <gi...@apache.org> on 2023/07/07 19:02:07 UTC
[GitHub] [arrow-adbc] aiguofer opened a new issue, #887: adbc/flightsql: Connection management and per request headers
aiguofer opened a new issue, #887:
URL: https://github.com/apache/arrow-adbc/issues/887
For applications that make requests to a FlightSql server, how should connections be handled?
My initial assumption was that since there is no state associated with the connection and it's simply making gRPC requests, that I could just create a connection on each request. However, I'm noticing this adds ~100ms latency per request. Although it's not the end of the world, this seems a little excessive. Digging into it a little, I noticed that when I establish a connection it makes a `getFlightInfoSqlInfo` request (oddly enough, it doesn't also make the `getStreamSqlInfo` request). This left me with a few questions:
- Should we maintain long lived connections?
- What kind of state or socket connection is associated with a connection?
- If so, is a single connection thread safe, or should we have a connection per worker thread (or some sort of pool to manage this)?
- If we're maintaining a long lived connection and we want to pass a specific header when we execute a query, is there a way to pass a specific header only for that query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] lidavidm commented on issue #887: python/adbc_driver_flightsql: Connection management and per request headers
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #887:
URL: https://github.com/apache/arrow-adbc/issues/887#issuecomment-1626043111
While Flight SQL is *mostly* stateless (note that some contributors are attempting to change this!) it's still best to reuse connections where possible, both because there is state we are requesting from the server, and because that lets gRPC maintain a single TCP connection instead of having to constantly set up and tear down more.
For Flight SQL specifically, the connection should be thread safe, but ADBC does not guarantee this (so connection-per-worker or a pool is better).
You can set the header on the statement; this won't affect other connections (and will go away when you close the statement). Statements are fairly cheap (it's just a bit of state tracking the query string etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] aiguofer commented on issue #887: python/adbc_driver_flightsql: Connection management and per request headers
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on issue #887:
URL: https://github.com/apache/arrow-adbc/issues/887#issuecomment-1631515264
Aahhh it expects kwargs, that worked! thanks a bunch. I'll go ahead and close this and will likely follow up with a docs/docstring PR to make discoverability of these topics easier!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] aiguofer closed issue #887: python/adbc_driver_flightsql: Connection management and per request headers
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer closed issue #887: python/adbc_driver_flightsql: Connection management and per request headers
URL: https://github.com/apache/arrow-adbc/issues/887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] aiguofer commented on issue #887: python/adbc_driver_flightsql: Connection management and per request headers
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on issue #887:
URL: https://github.com/apache/arrow-adbc/issues/887#issuecomment-1631124290
Great, thanks for the info and context! I think we have a way forward using the SQLAlchemy built in pool, which should be usable with any dbapi compliant connection.
As for adding headers to a statement, thanks for the info! was able to add DataDog tracing to a query with:
```
def _set_tracing_headers(self, cursor: Cursor) -> None:
current_context = tracer.current_trace_context()
if current_context:
headers: Dict[str, str] = {}
HTTPPropagator.inject(current_context, headers)
cursor.adbc_statement.set_options(
{f"{StatementOptions.RPC_CALL_HEADER_PREFIX.value}{k}": v for k, v in headers.items()}
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] lidavidm commented on issue #887: python/adbc_driver_flightsql: Connection management and per request headers
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #887:
URL: https://github.com/apache/arrow-adbc/issues/887#issuecomment-1631401163
I think that's missing a `**` in front of the dict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] aiguofer commented on issue #887: python/adbc_driver_flightsql: Connection management and per request headers
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on issue #887:
URL: https://github.com/apache/arrow-adbc/issues/887#issuecomment-1631363727
hmmm, actually this does not work:
```
cursor.adbc_statement.set_options(
{f"{StatementOptions.RPC_CALL_HEADER_PREFIX.value}{k}": v for k, v in headers.items()}
)
```
I'm getting `TypeError: set_options() takes exactly 0 positional arguments (1 given)`. How should I be setting those?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org