You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "felipecrv (via GitHub)" <gi...@apache.org> on 2025/04/23 14:13:24 UTC
[I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
felipecrv opened a new issue, #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739
### What feature or improvement would you like to see?
As discussed in https://github.com/apache/arrow-adbc/pull/2525#issuecomment-2822702472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2996562870
I opened this issue after comments in #2525. Do I understand it right, that what's been decided here is that something along the lines of what #2525 does can be merged.
> (at least at this time the latest datafusion and adbc are based on the same arrow-rs version, no additional work needed)
@eitsupi that's not enough. I want to use `adbc_core` from slightly older versions of Arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #2739: rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires
URL: https://github.com/apache/arrow-adbc/issues/2739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2996998611
I see your point.
In that case perhaps an additional test on CI using the `-Z minimal-versions` option is needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2826283337
For what it's worth, I asked at the DataFusion community call and it seems this type of dependency weirdness between Arrow, Data Fusion, and other tooling happens everywhere. When adbc_core and the datafusion driver are a bit more fleshed out I think we can approach those communities to move the projects to minimize the weirdness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2976750322
Thanks for clarifying!
> We would also want to decouple the MSRV: [#2525 (comment)](https://github.com/apache/arrow-adbc/pull/2525#issuecomment-2705299051)
Actually, it seems that MSRV is already isolated. (ideally they should be tested on CI)
https://github.com/apache/arrow-adbc/blob/24950853d2398fd75601656fbf2bc2ddeff7d87c/rust/Cargo.toml#L26
https://github.com/apache/arrow-adbc/blob/24950853d2398fd75601656fbf2bc2ddeff7d87c/rust/driver/datafusion/Cargo.toml#L23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2828680819
> ...weirdness between Arrow, Data Fusion, and other tooling happens everywhere
True, but `adbc_core` is probably one of the easiest to detangle since it depends only on the Array C Data Interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2976605529
Sorry if I am not fully understanding the discussion, but is this problem not simply a matter of making the datafusion driver's arrow dependency not inherit the workspace version?
I mean just change
https://github.com/apache/arrow-adbc/blob/24950853d2398fd75601656fbf2bc2ddeff7d87c/rust/driver/datafusion/Cargo.toml#L35-L37
to
```toml
arrow-array = "55.0.0"
arrow-buffer = "55.0.0"
arrow-schema = "55.0.0"
```
or something.
(Of course, since it will be across different ARROW versions, it will also need to be rewritten to use FFI)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2976726395
I think that would work, along with updating the dependency for `adbc_core` to be something like `>=53,<55` (for example) to allow a bit more flexibility. That way you can more likely use the version of arrow-rs that your application uses. (We shouldn't be needing anything too complex from arrow-rs and we could eventually test a few versions just to be sure.)
We would also want to decouple the MSRV: https://github.com/apache/arrow-adbc/pull/2525#issuecomment-2705299051
(Somewhat tangential, but I'm also thinking it's weird that adbc_core has the FFI, driver manager, and driver exporter bits; I would have expected those to all be separate crates and not part of core.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-3000755279
After loosening the version requirements for arrow in the workspace as like #2525, I was able to confirm that the test passed on arrow 53.1.0 by running the command as follows.
```sh
cargo +nightly generate-lockfile -Z direct-minimal-versions
cargo test --all-features --package adbc_core
```
For now I have opened a PR to update this kind of thing to be done on CI #3017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2993930081
> I think that would work, along with updating the dependency for `adbc_core` to be something like `>=53,<55` (for example) to allow a bit more flexibility. That way you can more likely use the version of arrow-rs that your application uses. (We shouldn't be needing anything too complex from arrow-rs and we could eventually test a few versions just to be sure.)
Surely this is a simpler solution: after 2025, cargo will recognize the rust-version field and download the dependency, so we may not have to worry about higher versions that break MSRV accidentally getting mixed in.
(I have never tried it, so I don't know if it really works well in that case.)
As a note, a general note on such version designations can be found here.
https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-metadata
It may be useful if we want to test with minimum versions of the dependencies.
https://doc.rust-lang.org/cargo/reference/unstable.html#minimal-versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2994130448
Ok. It sounds like we don't need to split workspaces, at least. We can just relax `adbc_core`'s version bound (https://github.com/apache/arrow-adbc/issues/2524) and have `adbc_datafusion` pin the same version of Arrow as DataFusion itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "eitsupi (via GitHub)" <gi...@apache.org>.
eitsupi commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2994031457
I think we can close this. (at least at this time the latest datafusion and adbc are based on the same arrow-rs version, no additional work needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] rust: split workspaces so that dependencies of adbc_core aren't tied to what the datafusion driver requires [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2739:
URL: https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2994028082
@eitsupi so do you think we can just close this (now that you've added an MSRV test)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org