You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/09/17 22:48:49 UTC
[PR] bench: keep the expect_plan check out of the measured region [datafusion]
adriangb opened a new pull request, #25441:
URL: https://github.com/apache/datafusion/pull/25441
## Which issue does this PR close?
- N/A. This is a benchmark harness fix found while measuring https://github.com/apache/datafusion/pull/25339.
## Rationale for this change
A benchmark file can assert a string in the physical plan with the `expect_plan` directive. That check ran on every iteration, inside the measured region, and it rendered the plan with `{:#?}`.
A derived `Debug` for an in-memory source prints every `RecordBatch` the source holds. Thus, for a benchmark whose tables come from `CREATE TABLE ... AS SELECT`, the check builds a multi-megabyte string per iteration and then searches it. For the `null_aware_join` suite, whose tables are built from `range()`, that string is 1.85 MB and the check is most of the measured time: the suite reports 17.6 ms for Q01, while `datafusion-cli` runs the same query in 4 ms.
This inflates absolute numbers and, more importantly, dilutes the A/B difference the benchmark exists to show.
## What changes are included in this PR?
The check now renders the plan as `EXPLAIN` displays it, through `DisplayableExecutionPlan`. It also runs on the first iteration only, because a plan does not change between iterations.
`null_aware_join`, release build, both sides at the same commit (`c4f5a9e0f2`), median of 30 iterations:
| Query | before | after |
|---|---|---|
| Q01 | 17.6 ms | 4.6 ms |
| Q02 | 15.3 ms | 2.6 ms |
| Q03 | 14.7 ms | 2.1 ms |
| Q04 | 0.9 ms | 0.4 ms |
| Q05 | 0.9 ms | 0.4 ms |
| Q06 | 0.8 ms | 0.3 ms |
| Q07 | 0.9 ms | 0.3 ms |
| Q08 | 1.2 ms | 0.5 ms |
A failed check now also prints an 8-line plan instead of a 1.8 MB dump.
There are 96 `expect_plan` strings in the suites. 89 are operator names, which both forms print. The 6 `null_aware_join` strings that read `null_aware: true` become `null_aware`, which is how `HashJoinExec` displays the flag; this PR changes those 6 lines. The one remaining string, h2o's `output_ordering=[pk@0 ASC NULLS LAST, ob@1 DESC]`, holds in the display form as well.
## What is the testing strategy for this PR?
- New unit test `run_checks_expect_plan_once_per_benchmark`: the first run checks the plan, and a later run does not repeat the check. The existing tests still cover the accept and reject paths of a first run.
- I ran the suites that need no downloaded data, and all their `expect_plan` strings still hold: `null_aware_join` (8 queries), `smj` (26 queries, which include the 3 `LeftMark` checks), `nlj` (4 queries) and `array_agg_distinct`.
- For the h2o `output_ordering` string, which needs data I do not have, I built an equivalent case: a Parquet table with `WITH ORDER`, a window query over it, and that `expect_plan` line. It passes.
- Counterfactual check: with a string that is not in the plan, the first iteration still fails, and the message names the string.
- `cargo test -p datafusion-benchmarks --lib` passes (143 tests). `cargo fmt` and clippy are clean.
## Are there any user-facing changes?
No. This changes the benchmark harness and its documentation only.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] bench: keep the expect_plan check out of the measured region [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #25441:
URL: https://github.com/apache/datafusion/pull/25441#issuecomment-5736073404
Thanks @adriangb this is a nice improvement!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] bench: keep the expect_plan check out of the measured region [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25441:
URL: https://github.com/apache/datafusion/pull/25441#issuecomment-5722434631
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25441?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.33%. Comparing base ([`c4f5a9e`](https://app.codecov.io/gh/apache/datafusion/commit/c4f5a9e0f202af0231287ec77774b2e7601e018d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ae18f6b`](https://app.codecov.io/gh/apache/datafusion/commit/ae18f6b31327e486c4dd621e9a2809539287b6ab?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25441 +/- ##
==========================================
- Coverage 82.33% 82.33% -0.01%
==========================================
Files 1137 1137
Lines 432391 432530 +139
Branches 432391 432530 +139
==========================================
+ Hits 356029 356141 +112
- Misses 54845 54847 +2
- Partials 21517 21542 +25
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25441?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] bench: keep the expect_plan check out of the measured region [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #25441:
URL: https://github.com/apache/datafusion/pull/25441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org