You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2025/04/15 14:10:55 UTC
[I] format: explicit cursors and backpressure? [arrow-adbc]
lidavidm opened a new issue, #2709:
URL: https://github.com/apache/arrow-adbc/issues/2709
### What feature or improvement would you like to see?
We have this implicitly from ArrowArrayStream but maybe we need something more explicit? Or would the async stream and its backpressure handle this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] format: explicit cursors and backpressure? [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2709:
URL: https://github.com/apache/arrow-adbc/issues/2709#issuecomment-2817849554
Putting things together we need the following:
- Executing a SQL query can either give you partitions or (multiple) result sets
- Maybe this can be unified to giving you a list of "promise" objects that can be either immediately consumed to get a (partition of a) result set, or serialized into a partition (if supported)
- This does lead to more API calls, though...
- Each result set needs to have a requested schema bound (so the driver can convert if it's able to/if there are potentially multiple valid Arrow types for a column)
- Result streams should use the async device interface, which handles backpressure already
- Cancelling the statement will cancel the entire query?
- Dropping a substream shouldn't cancel the entire query?
- Do we need some way to potentially retry a stream, and ideally retrying a stream from a given (row) offset?
- Some of these questions may be specific to Flight SQL, but I think it generally applies to any sort of newfangled system with a REST or gRPC API instead of an old-school database protocol
- We won't support seeking through streams (aka no actual cursors)
- We should standardize that readers should be safe to use if ADBC objects are released (but they may error instead of producing data and/or ADBC objects may error on release if there are unclosed streams)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] format: explicit cursors and backpressure? [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #2709:
URL: https://github.com/apache/arrow-adbc/issues/2709#issuecomment-2806196134
I assume that the Async idea and it's backpressure would handle this, at least IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org