You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "EmilyMatt (via GitHub)" <gi...@apache.org> on 2025/04/23 12:43:15 UTC
[PR] chore: Update viable crates [datafusion-comet]
EmilyMatt opened a new pull request, #1677:
URL: https://github.com/apache/datafusion-comet/pull/1677
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Reduce the amount of duplicate crates due to crates that use outdated versions, thereby improving compile times and reducing binary size.
Some of the crates(such as rand) have also made changes due to upcoming Rust Compiler changes that would give them compilation errors(reserved keywords, in this case, I believe)
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Update crate versions, removed some unused crates, removed crate versions when using "path"(for less maintenance), and moved to use the crates.io version of datafusion now that its released(it was left on the git version)
For the rand crate, updated the usage based on the deprecation warnings.
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Firstly by compilation, then all the tests passing normally should be enough.
As for timing and crate count:
branch `main`
Binary size - 75MBs (release build)
branch `update-crates`
Binary size - 73MBs(release build)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Update viable crates [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1677:
URL: https://github.com/apache/datafusion-comet/pull/1677#discussion_r2056028940
##########
native/Cargo.toml:
##########
@@ -38,16 +38,16 @@ arrow = { version = "55.0.0", features = ["prettyprint", "ffi", "chrono-tz"] }
async-trait = { version = "0.1" }
bytes = { version = "1.10.0" }
parquet = { version = "55.0.0", default-features = false, features = ["experimental"] }
-datafusion = { git = "https://github.com/apache/datafusion", rev = "47.0.0", default-features = false, features = ["unicode_expressions", "crypto_expressions", "nested_expressions", "parquet"] }
-datafusion-comet-spark-expr = { path = "spark-expr", version = "0.8.0" }
-datafusion-comet-proto = { path = "proto", version = "0.8.0" }
Review Comment:
It is usally necessary to keep the versions as well otherwise `cargo publish` will fail. However, we do not publish this crate so it is probably fine to remove them.
##########
native/Cargo.toml:
##########
@@ -38,16 +38,16 @@ arrow = { version = "55.0.0", features = ["prettyprint", "ffi", "chrono-tz"] }
async-trait = { version = "0.1" }
bytes = { version = "1.10.0" }
parquet = { version = "55.0.0", default-features = false, features = ["experimental"] }
-datafusion = { git = "https://github.com/apache/datafusion", rev = "47.0.0", default-features = false, features = ["unicode_expressions", "crypto_expressions", "nested_expressions", "parquet"] }
-datafusion-comet-spark-expr = { path = "spark-expr", version = "0.8.0" }
-datafusion-comet-proto = { path = "proto", version = "0.8.0" }
Review Comment:
It is usually necessary to keep the versions as well otherwise `cargo publish` will fail. However, we do not publish this crate so it is probably fine to remove them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Update viable crates [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #1677:
URL: https://github.com/apache/datafusion-comet/pull/1677#discussion_r2056074078
##########
.github/workflows/miri.yml:
##########
@@ -38,6 +38,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
+ - name: Install Build Dependencies
+ shell: bash
+ run: |
+ apt-get update
+ apt-get install -y protobuf-compiler
Review Comment:
Previous version of prost carried a fallback binary in case protoc was not found, but I think its both better to not download a binary executable with a crate, as well as ensure the miri tests use the same exact code as the regular build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Update viable crates [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #1677:
URL: https://github.com/apache/datafusion-comet/pull/1677#discussion_r2056071541
##########
native/Cargo.toml:
##########
@@ -38,16 +38,16 @@ arrow = { version = "55.0.0", features = ["prettyprint", "ffi", "chrono-tz"] }
async-trait = { version = "0.1" }
bytes = { version = "1.10.0" }
parquet = { version = "55.0.0", default-features = false, features = ["experimental"] }
-datafusion = { git = "https://github.com/apache/datafusion", rev = "47.0.0", default-features = false, features = ["unicode_expressions", "crypto_expressions", "nested_expressions", "parquet"] }
-datafusion-comet-spark-expr = { path = "spark-expr", version = "0.8.0" }
-datafusion-comet-proto = { path = "proto", version = "0.8.0" }
Review Comment:
Ah thats good to know
I can revert this change if you like, I just feel its nicer not having to do manual work everywhere when updating a version^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Update viable crates [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1677:
URL: https://github.com/apache/datafusion-comet/pull/1677#discussion_r2056187263
##########
native/Cargo.toml:
##########
@@ -38,16 +38,16 @@ arrow = { version = "55.0.0", features = ["prettyprint", "ffi", "chrono-tz"] }
async-trait = { version = "0.1" }
bytes = { version = "1.10.0" }
parquet = { version = "55.0.0", default-features = false, features = ["experimental"] }
-datafusion = { git = "https://github.com/apache/datafusion", rev = "47.0.0", default-features = false, features = ["unicode_expressions", "crypto_expressions", "nested_expressions", "parquet"] }
-datafusion-comet-spark-expr = { path = "spark-expr", version = "0.8.0" }
-datafusion-comet-proto = { path = "proto", version = "0.8.0" }
Review Comment:
It is fine to remove it. If we ever decide to start publishing this crate, we can add it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Update viable crates [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1677:
URL: https://github.com/apache/datafusion-comet/pull/1677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Update viable crates [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1677:
URL: https://github.com/apache/datafusion-comet/pull/1677#issuecomment-2825234249
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1677?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 56.62%. Comparing base [(`f09f8af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`bc8d6c6`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/bc8d6c663c42ff64728053e994943a28d6e6746b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 161 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #1677 +/- ##
============================================
+ Coverage 56.12% 56.62% +0.50%
- Complexity 976 1014 +38
============================================
Files 119 125 +6
Lines 11743 12591 +848
Branches 2251 2346 +95
============================================
+ Hits 6591 7130 +539
- Misses 4012 4286 +274
- Partials 1140 1175 +35
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1677?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org