You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datafusion.apache.org by "gabotechs (via GitHub)" <gi...@apache.org> on 2026/09/09 16:04:27 UTC
[PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
gabotechs opened a new pull request, #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2
This PRs does a basic setup so that the project starts compiling and tests start passing.
It still does not include any CI or Github workflow.
The main changes are:
- Cargo.toml: point to specific deps from crates.io and a recent commit hash from `iceberg-rust`
- .gitignore: so that some folders do not get unintentionally committed to git
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r4040060981
##########
crates/playground/Cargo.toml:
##########
@@ -17,33 +17,32 @@
[package]
description = "Apache iceberg client"
-edition.workspace = true
-homepage.workspace = true
-license.workspace = true
+edition = { workspace = true }
+homepage = { workspace = true }
+license = { workspace = true }
name = "iceberg-playground"
publish = false
readme = "README.md"
-repository.workspace = true
-rust-version.workspace = true
-version.workspace = true
+repository = { workspace = true }
+version = { workspace = true }
[dependencies]
anyhow = { workspace = true }
-clap = { workspace = true }
+clap = { version = "4.5.48", features = ["derive", "cargo"] }
datafusion = { workspace = true }
datafusion-cli = { workspace = true }
-dirs = { workspace = true }
-fs-err = { workspace = true }
-home = { workspace = true }
+dirs = "6"
+fs-err = "3.1.0"
+home = "0.5.12"
iceberg = { workspace = true }
-iceberg-catalog-rest = { workspace = true }
-iceberg-datafusion = { workspace = true }
-mimalloc = { workspace = true }
-stacker = { workspace = true }
+iceberg-catalog-rest = { git = "https://github.com/apache/iceberg-rust", rev = "665c64e48e8d33797ecb1a421f327edd9b024879" }
Review Comment:
can move this to use workspace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "konjac (via GitHub)" <gi...@apache.org>.
konjac commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r3991384773
##########
Cargo.toml:
##########
@@ -16,34 +16,36 @@
# under the License.
[package]
-edition = { workspace = true }
Review Comment:
same question to dependencies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#issuecomment-5730746040
Thanks everyone! I'll work on getting some CI set up soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r4047145370
##########
crates/playground/Cargo.toml:
##########
@@ -17,33 +17,32 @@
[package]
description = "Apache iceberg client"
-edition.workspace = true
-homepage.workspace = true
-license.workspace = true
+edition = { workspace = true }
+homepage = { workspace = true }
+license = { workspace = true }
name = "iceberg-playground"
publish = false
readme = "README.md"
-repository.workspace = true
-rust-version.workspace = true
-version.workspace = true
+repository = { workspace = true }
+version = { workspace = true }
[dependencies]
anyhow = { workspace = true }
-clap = { workspace = true }
+clap = { version = "4.5.48", features = ["derive", "cargo"] }
datafusion = { workspace = true }
datafusion-cli = { workspace = true }
-dirs = { workspace = true }
-fs-err = { workspace = true }
-home = { workspace = true }
+dirs = "6"
+fs-err = "3.1.0"
+home = "0.5.12"
iceberg = { workspace = true }
-iceberg-catalog-rest = { workspace = true }
-iceberg-datafusion = { workspace = true }
-mimalloc = { workspace = true }
-stacker = { workspace = true }
+iceberg-catalog-rest = { git = "https://github.com/apache/iceberg-rust", rev = "665c64e48e8d33797ecb1a421f327edd9b024879" }
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "konjac (via GitHub)" <gi...@apache.org>.
konjac commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r3991367397
##########
Cargo.toml:
##########
@@ -16,34 +16,36 @@
# under the License.
[package]
-edition = { workspace = true }
Review Comment:
why does the PR stop using the workspace feature to share common meta entries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r4007220735
##########
crates/datafusion/Cargo.toml:
##########
@@ -16,34 +16,36 @@
# under the License.
[package]
-edition = { workspace = true }
Review Comment:
It was just due to a mechanical replace, just moved all the common stuff to the workspace-level Cargo.toml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r3972611592
##########
Cargo.toml:
##########
@@ -16,34 +16,33 @@
# under the License.
[package]
-edition = { workspace = true }
-homepage = { workspace = true }
-name = "iceberg-datafusion"
+edition = "2024"
+homepage = "https://rust.iceberg.apache.org/"
Review Comment:
ok for now, we may want to move this later
##########
src/physical_plan/repartition.rs:
##########
@@ -76,7 +76,7 @@ use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
///
/// ```ignore
/// use std::num::NonZeroUsize;
-/// use iceberg_datafusion::physical_plan::project_with_partition;
+/// use datafusion_iceberg::physical_plan::project_with_partition;
Review Comment:
😄
##########
Cargo.toml:
##########
@@ -16,34 +16,33 @@
# under the License.
[package]
-edition = { workspace = true }
-homepage = { workspace = true }
-name = "iceberg-datafusion"
+edition = "2024"
+homepage = "https://rust.iceberg.apache.org/"
+name = "datafusion-iceberg"
publish = true
-rust-version = { workspace = true }
-version = { workspace = true }
+version = "0.10.1"
categories = ["database"]
description = "Apache Iceberg DataFusion Integration"
keywords = ["iceberg", "integrations", "datafusion"]
-license = { workspace = true }
-repository = { workspace = true }
+license = "Apache-2.0"
+repository = "https://github.com/datafusion-contrib/datafusion-iceberg"
[dependencies]
-anyhow = { workspace = true }
-async-trait = { workspace = true }
-dashmap = { workspace = true }
-datafusion = { workspace = true }
-futures = { workspace = true }
-iceberg = { workspace = true }
-parquet = { workspace = true }
-tokio = { workspace = true }
-uuid = { workspace = true }
+anyhow = "1.0.72"
+async-trait = "0.1.89"
+dashmap = "6"
+datafusion = "55.0.0"
+futures = "0.3"
+iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "665c64e48e8d33797ecb1a421f327edd9b024879" }
Review Comment:
do we want to pin to hash or use the latest release?
https://crates.io/crates/iceberg/versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r3989815942
##########
Cargo.toml:
##########
@@ -16,34 +16,33 @@
# under the License.
[package]
-edition = { workspace = true }
-homepage = { workspace = true }
-name = "iceberg-datafusion"
+edition = "2024"
+homepage = "https://rust.iceberg.apache.org/"
+name = "datafusion-iceberg"
publish = true
-rust-version = { workspace = true }
-version = { workspace = true }
+version = "0.10.1"
categories = ["database"]
description = "Apache Iceberg DataFusion Integration"
keywords = ["iceberg", "integrations", "datafusion"]
-license = { workspace = true }
-repository = { workspace = true }
+license = "Apache-2.0"
+repository = "https://github.com/datafusion-contrib/datafusion-iceberg"
[dependencies]
-anyhow = { workspace = true }
-async-trait = { workspace = true }
-dashmap = { workspace = true }
-datafusion = { workspace = true }
-futures = { workspace = true }
-iceberg = { workspace = true }
-parquet = { workspace = true }
-tokio = { workspace = true }
-uuid = { workspace = true }
+anyhow = "1.0.72"
+async-trait = "0.1.89"
+dashmap = "6"
+datafusion = "55.0.0"
+futures = "0.3"
+iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "665c64e48e8d33797ecb1a421f327edd9b024879" }
Review Comment:
The latest release of `iceberg-rust` is 0.10 https://crates.io/crates/iceberg-rust/0.10.0/code/Cargo.toml, which still pins to `arrow@58`.
However, `iceberg-rust@main` already depends on `arrow@59`, so if we want to port the full commit history, we cannot depend on whatever is published to crates.io.
This should be fine though, this will just be a problem when we release this project first to crates.io, which I don't expect to happen soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#issuecomment-5605235672
I think the same will apply on this pr as mentioned here https://github.com/apache/datafusion-iceberg/pull/1#issuecomment-5605055826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2#discussion_r3989777325
##########
Cargo.toml:
##########
@@ -16,34 +16,33 @@
# under the License.
[package]
-edition = { workspace = true }
-homepage = { workspace = true }
-name = "iceberg-datafusion"
+edition = "2024"
+homepage = "https://rust.iceberg.apache.org/"
Review Comment:
I actually can move this now, should not be a problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org
Re: [PR] Adapt project for new repo (make it compile + tests passing) [datafusion-iceberg]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs merged PR #2:
URL: https://github.com/apache/datafusion-iceberg/pull/2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datafusion.apache.org
For additional commands, e-mail: dev-help@datafusion.apache.org