You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/03/26 02:24:18 UTC
[I] [DRAFT, EPIC] Benchmark improvements [datafusion]
adriangb opened a new issue, #21165:
URL: https://github.com/apache/datafusion/issues/21165
I'm opening this epic to track improvements / changes we want to our benchmarking setup.
I'll start by collecting some relevant issues:
- https://github.com/apache/datafusion/issues/15511
- https://github.com/apache/datafusion/issues/5504
- https://github.com/apache/datafusion/issues/13446
- #21034
I think we should discuss in this issue what we want from our benchmarking setup and use that to guide how we improve it.
## Trackable over time
This is important for gating releases and catching regressions early. Currently we only ever really run benchmarks in a PR to compare to main or when we go to update ClickBench.
I think we should target Codspeed compatibility for this.
## Can run slow/complex SQL benchmarks
Think ClickBench. I think this might discard the use of criterion at least for this style of benchmark, see https://github.com/bheisler/criterion.rs/issues/320. We can use criterion for smaller, faster benchmarks.
We also need a harness that supports loading data, can give us both cold and hot numbers.
## Can do a "quick run"
We want to be able to run benchmarks just to verify the results are correct, or as a test during development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "vito1317 (via GitHub)" <gi...@apache.org>.
vito1317 commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4967194124
Re: **Trackable over time** — the variability concern (@alamb's point that without real statistical analysis "we'll probably just make a bunch of data we never look at") is what I've tried to address in [alamb/datafusion-benchmarking#16](https://github.com/alamb/datafusion-benchmarking/pull/16): per-query significance thresholds learned from each query's own night-over-night noise (IQR fences, as rustc-perf does) plus E-divisive-means changepoint detection with a permutation significance test (as MongoDB does), running over nightly result history and annotating the dashboard. Validated against the existing 383-revision `results_metal` history, it independently re-discovers the `collect_statistics` changepoint (`2d7ae0926`) on 7 queries. More detail in my comment on #5504.
The detectors are stdlib-only Python and not tied to that repo's CSV format. If useful, I could adapt them into something like `benchmarks/analyze.py` in this repo, consuming a directory of historical `dfbench` JSON results and complementing `compare.py`'s pairwise comparison. Would that be welcome here, or is the preference to keep the analysis layer in the external benchmarking repo (or go the Codspeed route)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4154394698
> Bringing my comment from [#21184](https://github.com/apache/datafusion/pull/21184):
>
> > Also I'm now confused where I should add benchmarks. [#20464](https://github.com/apache/datafusion/pull/20464) added Criterion SMJ benchmarks for sort-merge join , but it's missing scenarios from dfbench's smj benchmarks, which I further extend here. Any help?
>
> Each PR I work on SMJ I add more benchmarks to represent problem scenarios we're seeing. But after [#20464](https://github.com/apache/datafusion/pull/20464) I am now confused. Where should I be adding benchmarks, and what will we be using to catch regressions?
Yeah, I think this is a great point -- it is not easy to add new benchmarks to the existing runner infrastructure, so it doesn't get done often
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4154389289
> My proposal would be that (cost permitting) we run benchmarks on every merge to main and post a comment in the PR if they regressed (so 1 run per PR vs. every commit) or at the very least we run them on RC branches comparing to the previous release
This would be great. My biggest concern with this approach is that the variability between runs is pretty high so we would have to invest in analysis / time to review the results. Otherwise we'll probably just make a bunch of data we never look at
> Currently writing benchmarks requires using dataframe APIs and quite a bit of ceremony (recent example: https://github.com/apache/datafusion/pull/21180).
> I would like it to be possible to write SQL benchmarks, including with some SQL or non-SQL setup (could be a bash script to download data), even if a bit of rust is required (e.g. sql_bench!("../q1/")).
Yes, I 100% agree with the value of SQL based benchmarks
I would personally love to see something with a similar UX to datafusion-cli (interactive REPL and the ability to run scripts in files).
One thought I had would be a special `datafusion-cli` build (maybe we can make `datafusion-bench` or something like what @Omega359 is describing, that was mostly the same as `datafusion-cli` (it is already a crate) with extra:
1. commands to create / generate data or setup
2. Better / easier to script output options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4139727285
> Can write SQL benchmarks
>
> Currently writing benchmarks requires using dataframe APIs and quite a bit of ceremony (recent example: https://github.com/apache/datafusion/pull/21180).
> I would like it to be possible to write SQL benchmarks, including with some SQL or non-SQL setup (could be a bash script to download data), even if a bit of rust is required (e.g. sql_bench!("../q1/")).
>
> This has several advantages:
>
> Less code / boilerplate.
> Benchmarks are more in line with real world usage.
> Can tweak benchmarks without recompiling.
>
> Proposal
>
> Macros/harness code to easily point at a directory of SQL files and generate test cases with setup, etc.
I know @adriangb has a [draft poc ](https://github.com/apache/datafusion/pull/20911) for porting dfbench to criterion however I got nerd sniped by this and decided to have a go at something similar. The idea is essentially that the benchmarks run by bench.sh/dfbench will mostly be converted from being code based to being almost completely sql based.
Thus, I'm in the process of doing a poc for converting [DuckDB's sql benchmark suite](https://github.com/duckdb/duckdb/tree/main/benchmark) to work for Datafusion (rewritten in Rust). I've got it working for a few benchmark suites (imdb, taxi) and I'm working on adding the the various clickbench benchmarks now. It's criterion based though adding/switching to Divan shouldn't be much of an issue. License will be the same as DuckDB since it's largely converted from it using LLM's (MIT license, compatible with Apache). I think I'm about 3 weeks away from having something that I feel would be ready for a draft PR.
It's likely that I'll use the bench.sh/dfbench as a basis for making it easy to run a benchmark (with on demand download of the data, etc) and datafusion-cli for any conversion/transformation of data (csv -> parquet for example). Currently it's 100% sql based except for downloading of data.
`BENCH_NAME=imdb cargo bench --bench sql`
ConfigOptions can be set in sql either during init so it's valid across the whole benchmark suite, per benchmark, or via the env via the use of `SessionConfig::from_env()` for on demand;
Lifecycle of a benchmark is currently
- `initialize` (load + init),
- `assert` (verify conditions via sql if you want),
- `run` (the part that is executed/benchmarked by criterion),
- `verify` (optional result validation)
- `cleanup` (drop data, etc).
I'm looking at adding auto creation of the result data similar to how the sqlogictests behave. since manually doing it is not entirely feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4137328410
Bringing my comment from #21184:
> Also I'm not confused where I should add benchmarks. @andygrove added Criterion SMJ benchmarks in #20464, but they're missing ones from dfbench's `smj` benchmarks, which I further extend here. Any help?
Each PR I work on SMJ I add more benchmarks to represent problem scenarios we're seeing. But after #20464 I am now confused. Where should I be adding benchmarks, and what will we be using to catch regressions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4137538127
I added https://github.com/adriangb/datafusion-benchmarking/pull/2 to the list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4137574161
> My proposal would be that (cost permitting) we run benchmarks on every merge to main and post a comment in the PR if they regressed (so 1 run per PR vs. every commit) or at the very least we run them on RC branches comparing to the previous release.
One possible intermediate cadence is a nightly run of the benchmarks (once per 24 hour period). That could be less frequent than every merge to main, but would catch regressions much earlier than testing only in RCs.
> We don't really have _any_ benchmarks for memory use and there have been multiple memory use regressions.
+1 on this. divan's support looks really promising.
In addition to allocation tracking, it might also be good to track IO used in queries. That's not necessarily something you might care about as much as runtime or peak memory use, but it helps in identifying where runtime regressions might have come from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4137603709
> In addition to allocation tracking, it might also be good to track IO used in queries. That's not necessarily something you might care about as much as runtime or peak memory use, but it helps in identifying where runtime regressions might have come from.
We could certainly do that via `ObjectStore`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4154398763
> The idea is essentially that the benchmarks run by bench.sh/dfbench will mostly be converted from being code based to being almost completely sql based.
@Omega359 I really like this idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4135513001
cc @Omega359 @alamb @cetra3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4139080061
I would love it if we could run the benchmarks on other processor families besides ARM. There are some code changes that may run better on ARM, but on my AMD processor I consistently get better results with some changes.
Case in point, this particular PR: https://github.com/apache/arrow-rs/pull/9407 while not related to datafusion directly, when running the benchmark locally vs main, it is consistently faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Benchmark improvements [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4274606199
My PR for the initial code and benchmark conversion for tpch is up @ #21706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org