You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "liurenjie1024 (via GitHub)" <gi...@apache.org> on 2025/03/27 04:46:54 UTC
[PR] Add datafusion cli for iceberg [iceberg-rust]
liurenjie1024 opened a new pull request, #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143
## Which issue does this PR close?
- Closes #1142.
## What changes are included in this PR?
Initial check in iceberg cli.
## Are these changes tested?
Yes, ut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on code in PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2018187420
##########
rust-toolchain.toml:
##########
Review Comment:
Maybe we can exclude `crates/integrations` from workspace, and have separate `rust-toolchain.toml` for them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on code in PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2032308752
##########
rust-toolchain.toml:
##########
Review Comment:
I don't have strong opinions on this. It would be nice to have more frequent iceberg-rs releases.
But it seems currently we want to wait for some internal milestones to be finished before release. So if some user/developer wants to upgrade datafusion, they have to wait for it. This slowness seems a little unnecessary to me. 🤔
To summarize a bit,
- approach 1: use separate toolchain/msrv for integrations. (currently used)
- approach 2: use the same toolchain/msrv
- to do this, we might either need to bump msrv on demand (not only on release ), or release more frequently.
also cc @Fokko @Xuanwo @sdd @c-thiel for opinions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2031156783
##########
rust-toolchain.toml:
##########
Review Comment:
> (imagine datafusion releases more frequently, and we can't bump MSRV until next release.)
Do we need to catch up with every latest datafusion release? In this case I would prefer to release iceberg-rust more often.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 closed pull request #1143: Add datafusion cli for iceberg
URL: https://github.com/apache/iceberg-rust/pull/1143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#issuecomment-2796350792
In favor #1193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#issuecomment-2781418926
> > * Closes [An iceberg cli tool #1142](https://github.com/apache/iceberg-rust/issues/1142).
>
> Maybe we could have a tracking issue and create subtasks for feature ideas?
Do you mean to have a tracking issue for all feature ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on code in PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2030391170
##########
rust-toolchain.toml:
##########
Review Comment:
Because currently we already use different MSRV for the datafusion integration.
https://github.com/apache/iceberg-rust/blame/e1de63d3ec47ee97d2b22cd193634e9aa9ff6ed8/crates/integrations/datafusion/Cargo.toml#L23-L26
And in previous discussion:
> We currently only have `msrv` for `iceberg`. I'm not sure if our goal is to maintain `msrv` for all our integration crates, such as `datafusion`.
_Originally posted by @Xuanwo in https://github.com/apache/iceberg-rust/issues/849#issuecomment-2562880940_
You can see e.g., this PR is blocked by this. Actually we can upgrade the toolchain version used by the datafusion integration, without upgrading the project MSRV.
And e.g., for the cli project, since it depends on the datafusion integration, it needs to use `max(project_msrv, datafusion_integration_msrv)`. That's why I proposed to just have a separate toolchain for all the integrations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#issuecomment-2781902388
> > > * Closes [An iceberg cli tool #1142](https://github.com/apache/iceberg-rust/issues/1142).
> >
> >
> > Maybe we could have a tracking issue and create subtasks for feature ideas?
>
> Do you mean to have a tracking issue for all feature ideas?
Yes
> Seems we have can't merge this since datafusion-cli requires newer version of rustc. We should postpone it after we release 0.5.0.
I replied some thoughts on https://github.com/apache/iceberg-rust/pull/1143#discussion_r2030391170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#issuecomment-2760624467
> * Closes [An iceberg cli tool #1142](https://github.com/apache/iceberg-rust/issues/1142).
Maybe we could have a tracking issue and create subtasks for features?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2030149688
##########
rust-toolchain.toml:
##########
Review Comment:
What's the benefit of this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#issuecomment-2781427563
Seems we have can't merge this since datafusion-cli requires newer version of rustc. We should postpone it after we release 0.5.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add datafusion cli for iceberg [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on code in PR #1143:
URL: https://github.com/apache/iceberg-rust/pull/1143#discussion_r2030393097
##########
rust-toolchain.toml:
##########
Review Comment:
But it seems our new MSRV policy (3 month) can cover datafusion's 4 month policy, so maybe there's no need to have separate MSRV any more.
However, if we only upgrade MSRV "on release", that may not be enough. (imagine datafusion releases more frequently, and we can't bump MSRV until next release.) Maybe we could just bump MSRV "on demand" in this case? We just need to guarantee at any time, the 3 month time range is covered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org