You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/08/18 21:48:32 UTC
[PR] ci: Check datafusion-spark feature combinations to catch publish errors [datafusion]
alamb opened a new pull request, #24475:
URL: https://github.com/apache/datafusion/pull/24475
## Which issue does this PR close?
- Closes #24474
## Rationale for this change
Publishing `datafusion-spark` 55.0.0 failed with
```
error[E0433]: cannot find module or crate `datafusion` in this scope
--> src/function/string/quote.rs:20:5
```
because the library referenced the optional `datafusion` dependency (only enabled by the `core` feature) from code that is not gated on that feature.
CI did not catch this because `datafusion-cli` and `datafusion-sqllogictest` depend on `datafusion-spark` with the `core` feature enabled, so workspace-wide builds always compile `datafusion-spark` with the optional `datafusion` dependency present due to feature unification. `cargo publish` builds the crate standalone with only default features (which are empty), where the dependency is absent.
The underlying import error was already fixed on `main` in #24351, so this PR adds the missing CI coverage to prevent the problem from reoccurring.
## What changes are included in this PR?
Add a `cargo check datafusion-spark features` CI job, following the pattern of the existing per-crate feature check jobs (e.g. `datafusion-functions`), that checks the crate standalone:
- default features (`--all-targets`)
- `--no-default-features` (what `cargo publish` verifies)
- `--no-default-features --features=core`
I verified locally that the `--no-default-features` check reproduces the exact `E0433` error from #24474 when the pre-#24351 import is reintroduced, and that all three checks pass on `main`.
## Are these changes tested?
Yes, by CI itself (the new job runs on this PR). The commands were also run locally and pass.
## Are there any user-facing changes?
No, CI only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Check datafusion-spark feature combinations to catch publish errors [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #24475:
URL: https://github.com/apache/datafusion/pull/24475#discussion_r3808213628
##########
.github/workflows/rust.yml:
##########
@@ -287,6 +287,35 @@ jobs:
- name: Check datafusion-functions (unicode_expressions)
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=unicode_expressions
+ # Check datafusion-spark crate features
Review Comment:
It is strange to see an entirely new job for this, but it mirrors what the other check jobs do 🤷
Maybe we can consolidate some of the check jobs to reduce CI cost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Check datafusion-spark feature combinations to catch publish errors [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #24475:
URL: https://github.com/apache/datafusion/pull/24475#discussion_r3808234428
##########
.asf.yaml:
##########
@@ -66,6 +66,7 @@ github:
- "cargo check datafusion-proto features"
- "cargo check datafusion features"
- "cargo check datafusion-functions features"
+ - "cargo check datafusion-spark features"
Review Comment:
mirror the other jobs and make it a required check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Check datafusion-spark feature combinations to catch publish errors [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24475:
URL: https://github.com/apache/datafusion/pull/24475#issuecomment-5360606150
> maybe we could explore cargo matrix: https://crates.io/crates/cargo-matrix
That is a good idea -- thank you @Jefffrey
I filed
- https://github.com/apache/datafusion/issues/24533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Check datafusion-spark feature combinations to catch publish errors [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #24475:
URL: https://github.com/apache/datafusion/pull/24475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Check datafusion-spark feature combinations to catch publish errors [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #24475:
URL: https://github.com/apache/datafusion/pull/24475#discussion_r3817997887
##########
.github/workflows/rust.yml:
##########
@@ -287,6 +287,35 @@ jobs:
- name: Check datafusion-functions (unicode_expressions)
run: cargo check --profile ci --no-default-features -p datafusion-functions --features=unicode_expressions
+ # Check datafusion-spark crate features
Review Comment:
maybe we could explore cargo matrix: https://crates.io/crates/cargo-matrix
though figure out a way to only test a subset of the features instead of every combination (like to be quite expensive?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Check datafusion-spark feature combinations to catch publish errors [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24475:
URL: https://github.com/apache/datafusion/pull/24475#issuecomment-5334761448
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24475?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 81.22%. Comparing base ([`f6ad781`](https://app.codecov.io/gh/apache/datafusion/commit/f6ad7810d05748bac745f677eee575b3d06ba50e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`62ee1ee`](https://app.codecov.io/gh/apache/datafusion/commit/62ee1eedc0493dde29d0bb4b6f53e115c00d8592?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24475 +/- ##
========================================
Coverage 81.22% 81.22%
========================================
Files 1113 1113
Lines 392250 392508 +258
Branches 392250 392508 +258
========================================
+ Hits 318591 318832 +241
- Misses 54914 54916 +2
- Partials 18745 18760 +15
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24475?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org