You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2026/10/23 08:09:03 UTC
[PR] ci: stop re-running the workspace test suite in the extended tests job [datafusion]
jayzhan211 opened a new pull request, #26649:
URL: https://github.com/apache/datafusion/pull/26649
## Which issue does this PR close?
- None. CI-only change to cut extended test time.
## Rationale for this change
The `extended_tests` job runs for every merge queue entry (778 times in the last 30 days, on 32-CPU runners), and most of what it runs is a second copy of the workspace test suite. The `cargo test (amd64)` job already runs that suite on the same commit in the merge queue.
In a recent merge queue run (39907001762) the job spent 555s running tests:
| | time |
|---|---|
| `tests/fuzz.rs` (only in this job) | 266s |
| sqllogictests (also in `cargo test (amd64)`) | 103s |
| physical-plan unit tests (also in `cargo test (amd64)`, except the 1000-iteration spill pool fuzz) | 56s |
| everything else (also in `cargo test (amd64)`) | ~130s |
## What changes are included in this PR?
The `extended` xtask step now runs only the tests that the `workspace` step does not:
- the fuzz tests and `memory_limit_validation`, which are behind the `extended_tests` feature
- `spill_pool_scenario_fuzz`, which this job runs with 1000 iterations
It builds only `datafusion` and `datafusion-physical-plan` test targets and uses test name filters to skip their other tests. The job name and required check are unchanged.
Nothing that ran before is lost:
- `extended_tests` is only defined in `datafusion` and only used by `tests/fuzz.rs` and `tests/memory_limit/mod.rs`.
- The `workspace` step runs every other test with a superset of the features. `recursive_protection` is a default feature, and `json` only applies to the proto crates, which the `workspace` step tests with `json`.
- `ffi_example_table_provider`, which the `workspace` step excludes, has no tests.
### Tradeoff
The step now lists the tests it runs by name. If a new test is put behind the `extended_tests` feature outside `fuzz_cases` or `memory_limit_validation`, it also needs a filter in this step, otherwise it never runs in CI. A comment next to the filters says this.
## What is the testing strategy for this PR?
The xtask snapshot test is updated. Running `cargo xtask ci step test extended` locally runs exactly the expected 156 tests and filters out the rest:
| binary | ran | filtered out |
|---|---|---|
| `datafusion` lib | 0 | 490 |
| `core_integration` | 21 | 1453 |
| `fuzz` | 134 | 0 |
| `datafusion-physical-plan` lib | 1 | 2576 |
Without the filters, the same targets run 4675 tests, and all pass.
Based on the run above, this should cut the job's test time from about 555s to about 325s, plus a smaller build. That is roughly 4 minutes per merge queue run.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: stop re-running the workspace test suite in the extended tests job [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26649:
URL: https://github.com/apache/datafusion/pull/26649#discussion_r4361260575
##########
xtask/src/ci_steps.rs:
##########
@@ -397,23 +397,32 @@ impl StepContext {
}
}
"extended" => {
+ // Run only the tests that the `workspace` step does not. The
+ // merge queue runs that step on the same commit.
command
.args([
"test",
"--profile",
"ci",
- "--exclude",
- "datafusion-examples",
- "--exclude",
- "datafusion-benchmarks",
- "--exclude",
- "datafusion-cli",
- "--workspace",
+ "-p",
+ "datafusion",
+ "-p",
+ "datafusion-physical-plan",
"--lib",
- "--tests",
- "--bins",
+ "--test",
+ "fuzz",
+ "--test",
+ "core_integration",
"--features",
- "avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption",
+ "avro,backtrace,extended_tests,parquet_encryption",
+ "--",
+ // Tests behind the `extended_tests` feature. A new test
Review Comment:
this is a good idea
##########
xtask/src/ci_steps.rs:
##########
@@ -397,23 +397,32 @@ impl StepContext {
}
}
"extended" => {
+ // Run only the tests that the `workspace` step does not. The
+ // merge queue runs that step on the same commit.
command
.args([
"test",
"--profile",
"ci",
- "--exclude",
- "datafusion-examples",
- "--exclude",
- "datafusion-benchmarks",
- "--exclude",
- "datafusion-cli",
- "--workspace",
+ "-p",
+ "datafusion",
+ "-p",
+ "datafusion-physical-plan",
"--lib",
- "--tests",
- "--bins",
+ "--test",
+ "fuzz",
+ "--test",
+ "core_integration",
"--features",
- "avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption",
+ "avro,backtrace,extended_tests,parquet_encryption",
+ "--",
+ // Tests behind the `extended_tests` feature. A new test
Review Comment:
But I worry about this list and the names of the tests getting out of sync.
Maybe as a follow on PR we could move all these "extended" tests into their own binary rather than having to have a feature flag *and* a list here
```shell
cargo test --test core_integration_extended
```
Or something like that 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: stop re-running the workspace test suite in the extended tests job [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #26649:
URL: https://github.com/apache/datafusion/pull/26649#issuecomment-6369397226
good, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: stop re-running the workspace test suite in the extended tests job [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #26649:
URL: https://github.com/apache/datafusion/pull/26649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: stop re-running the workspace test suite in the extended tests job [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26649:
URL: https://github.com/apache/datafusion/pull/26649#issuecomment-6378131800
Thanks @alamb @comphead . I’ll check if we can avoid maintaining this list manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: stop re-running the workspace test suite in the extended tests job [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26649:
URL: https://github.com/apache/datafusion/pull/26649#issuecomment-6423349654
> ci: run extended tests from one test target instead of name filters #26694
Update: was done in
- https://github.com/apache/datafusion/pull/26694
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org