You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2025/10/27 15:34:13 UTC
[PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
paleolimbot opened a new pull request, #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623
This PR proposes a 1.2 spec addition `AdbcStatementRequestSchema()`. There are a few decisions here and while I put specific language in I think any version of these is still helpful:
- I wrote this up as this best-effort (like the PyCapsule `requested_schema`). I think this is better for negotiation (e.g., the caller calls execute schema, requests a schema with all strings/binary/list columns as views, and the driver produces string/binary views but not list views because it doesn't implement those). A strict version is also useful and might generate better errors.
- Maybe `struct ArrowSchema*` should be `const ArrowSchema*`. I'm not sure this matters (mostly the C struct will be populated specifically for this call and they are easy to copy).
- No opinions on `RequestSchema` as the name if there's a better one
The capability to do this is built in to the Postgres driver already for some types (the COPY reader was designed to always accept an arbitrary ArrowSchema, which we currently just infer before constructing the reader).
Other 1.2 spec additions in https://github.com/apache/arrow-adbc/pull/3607 (which also contains the infrastructure for a new ADBC minor version).
Closes #1514 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3960045513
I updated the base branch to spec-1.2.0. Are there any updates needed to reflect the multi-result? I don't have a personal use case for requesting a schema from a multi result set but I'm also happy to add the requisite definitions if we want to do that while we're here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3453816347
Thanks for this!
I wonder if we also want to somehow allow things like "read 'field_name' as string view, but leave other fields alone" or "read all 'database_type' as string view, but leave other fields alone"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3957444238
Sorry, this slipped my notice...I think it's ok to put what you had back into this PR, and then we can review/merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3555491528
I can add that bit back in too...whatever is less confusing! We can also experiment with schema negotiation afterward (I don't think it affects the behaviour of the function but I could be missing something). I think options are orthogonal...options affect the guessing of the schema in the absence of a request and we could have those, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3615500567
My concern with options is that I'd want them to be reasonably consistent across drivers. Maybe it doesn't have to be part of this function, but I also don't want every driver to have its own slightly different way of choosing between string/string_view/large_string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-5134551630
> I think one followup is that we need a way to do this for ReadPartition too
Without judgement, is there any connector (other than Flight) which supports `AdbcStatementExecutePartition` today?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3519905815
Mostly I'm just trying to cut down the number of extra functions to create. If the idea is that we wouldn't need to create a new function to handle multiple result sets, then I'm okay with that. And that just leaves my first question
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#discussion_r3236667194
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2293,6 +2296,36 @@ AdbcStatusCode AdbcStatementExecuteSchema(struct AdbcStatement* statement,
struct ArrowSchema* schema,
struct AdbcError* error);
+/// \brief Request the schema of the next statement execution
+///
+/// Allows the caller to request a specific schema based on prior
Review Comment:
Should we explicitly define the behavior if columns in the result schema are missing from the requested schema and the reverse (columns in the requested schema missing from the result schema)?
What about column ordering? I'm not making any suggestions about what the behavior should be, just that we should explicitly state what the expected behavior should be in those cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-5134739530
I believe it was just added to DataFusion (but in that case, the "serialized" bit is a fully serialized physical plan, which does have an identical schema per partition)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-5134510910
> I think one followup is that we need a way to do this for ReadPartition too
Sorry, I lost track of this. Should it be that `AdbcStatementExecutePartition` takes the requested schema for the statement into account rather than the connection executing the partition? (I suppose I don't see any restriction that partitions share a schema)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3521890311
> Is the expectation that the driver itself would cast/convert the types as they come back from the source?
Yes, it's intended as guidance where a driver would otherwise have to guess. I wrote it up here as best-effort (same as PyCapsule `requested_schema`)...the other choice would be that the driver is required to produce that schema. I don't happen think that's particularly useful (whatever schema is produced by the output stream is always the stream that should be used to interpret the data), but for the immediate use cases (SQLite and Postgres drivers) it would also not be a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3617230265
Agreed (that's what I have always liked about schema negotiation).
Should I put that bit back into this PR or is that a separate documentation update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3518831086
two questions:
1. Is the expectation that the driver itself would cast/convert the types as they come back from the source?
2. Since we're also working on and discussing handling multiple result sets, how would this work/handle the case of multiple result sets? Would we need an entirely new function once we add support for multiple result sets to handle this for that case? Is there any way we can adapt this function so that it can work in both cases somehow instead of needing to create an entirely new function afterwards?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#discussion_r3236851674
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2293,6 +2296,36 @@ AdbcStatusCode AdbcStatementExecuteSchema(struct AdbcStatement* statement,
struct ArrowSchema* schema,
struct AdbcError* error);
+/// \brief Request the schema of the next statement execution
+///
+/// Allows the caller to request a specific schema based on prior
Review Comment:
Good point...I will try to update the text tonight. Off the top of my head the ability to request a schema with more columns than the source or in a different order is useful, as is widening the allowed nullability. Requesting fewer columns is fishy. DataFusion has quite a nice programmatic definition of this (designed for casting struct columns to match each other, possibly from multiple Parquet files).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3555270014
Well, maybe we want to cover cases like explicitly choosing between string/large_string/string_view (in principle the driver would have picked one, but maybe the user wants one of the other types and this can avoid a later conversion). Or is the thought that we need to add driver-specific flags for all of that? @CurtHagenlocher had commented before that with JDBC/ODBC, you get to pick the type you want when you get the data - that's obviously too much flexibility for an Arrow stream, but I don't see why we should limit this to only cases where it is ambiguous to the driver. (In the first place, the underlying data isn't Arrow much of the time; even if it is, ideally the request would be pushed back to the backend.)
The driver could certainly ignore requests to cast, of course, especially if it's actually just going to be a cast internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3518006425
This looks pretty good. On one hand, it's not the most ergonomic way to handle this but it's the most regular with the existing API. Am I right that the most convenient way to use this API may be to first call `AdbcStatementExecuteSchema` (and hope it's implemented), then modify the returned schema to my desire, call `AdbcStatementRequestSchema` with the modified schema, and then execute my statement?
Another question, should it be spelled out that the schema can be partial or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3518246126
I think the reason for adding this would be for a user who executed and collected a query but got an error along the lines of "can't append string to column of type int64", which is what would happen today if someone issued a query with the sqllite driver and first few thousand values were null.
The second reason is for people who already know the arrow type they want, perhaps because they're using a tool like ibis that already knows or because they executed the query and the driver guessed something they didn't like or that was inconsistent with a previous query against the same data.
I should probably take out the negotiation bit since it seems to be distracting from the intent...it's orthogonal to options, which would only affect the default guessing (but still have no way to specifically specify type parameters or specific problematic columns).
(These are all roughly the same reason why a requested schema is provided for a csv reader.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#discussion_r3612546953
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2293,6 +2296,36 @@ AdbcStatusCode AdbcStatementExecuteSchema(struct AdbcStatement* statement,
struct ArrowSchema* schema,
struct AdbcError* error);
+/// \brief Request the schema of the next statement execution
+///
+/// Allows the caller to request a specific schema based on prior
Review Comment:
What's the status here?
Personally I'm fine saying that you have to match (at least to start), though allowing widened nullability seems fine, and if there's a use case we can define the rest. Reordering columns and such to me starts to feel like #13 a bit. If only everyone supported Substrait, things would be easy :P
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-5403261518
Another followup is perhaps we want a way to specify that the caller is OK with fallible conversions (e.g. NUMERIC to decimal) resulting in a null instead of failing the call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3454380931
> I wonder if we also want to somehow allow things like "read 'field_name' as string view, but leave other fields alone" or "read all 'database_type' as string view, but leave other fields alone"
Sure, or some version of an option get/set with of a list of supported or preferred types. Happy to take out the negotiation bit if that's confusing (the main purpose here is to help the driver make better choices for loose or row-based typing).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-3519320223
Wouldn't you just call this in between NextResultSetSchema and NextResultSet? We'd just generalize NextResultSetSchema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-5042220607
I think one followup is that we need a way to do this for ReadPartition too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-5137079759
> > I think one followup is that we need a way to do this for ReadPartition too
>
> Without judgement, is there any connector (other than Flight) which supports `AdbcStatementExecutePartition` today?
Yeah, it's not well supported. In theory I believe Snowflake & BigQuery could also support it.
> Should it be that AdbcStatementExecutePartition takes the requested schema for the statement into account rather than the connection executing the partition? (I suppose I don't see any restriction that partitions share a schema)
I think this would be fine, I guess it would be less flexible but that may not matter. Presumably the driver would just serialize the schema into the partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#issuecomment-5138120364
As written here, the requested schema is just a suggestion (although we could say that the driver *should* consider it for `AdbcStatementExecutePartitions()` to put it on any potential implementor's radar if it applies to their model of partitioned execution).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/include/arrow-adbc): Add AdbcStatementRequestSchema [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #3623:
URL: https://github.com/apache/arrow-adbc/pull/3623#discussion_r3615056007
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2293,6 +2296,36 @@ AdbcStatusCode AdbcStatementExecuteSchema(struct AdbcStatement* statement,
struct ArrowSchema* schema,
struct AdbcError* error);
+/// \brief Request the schema of the next statement execution
+///
+/// Allows the caller to request a specific schema based on prior
Review Comment:
I forgot about this...I'll put the language in sometime today. The intention is to align types in place for drivers where the inferred destination type is arbitrary, not to reorder (which places responsibility on the driver that's currently tricky to satisfy in C or C++).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org