Re: [I] [EPIC] Benchmark improvements [datafusion]

["vito1317 (via GitHub)" <gi...@apache.org>]

vito1317 commented on issue #21165:
URL: https://github.com/apache/datafusion/issues/21165#issuecomment-4967194124

   Re: **Trackable over time** — the variability concern (@alamb's point that without real statistical analysis "we'll probably just make a bunch of data we never look at") is what I've tried to address in [alamb/datafusion-benchmarking#16](https://github.com/alamb/datafusion-benchmarking/pull/16): per-query significance thresholds learned from each query's own night-over-night noise (IQR fences, as rustc-perf does) plus E-divisive-means changepoint detection with a permutation significance test (as MongoDB does), running over nightly result history and annotating the dashboard. Validated against the existing 383-revision `results_metal` history, it independently re-discovers the `collect_statistics` changepoint (`2d7ae0926`) on 7 queries. More detail in my comment on #5504.
   
   The detectors are stdlib-only Python and not tied to that repo's CSV format. If useful, I could adapt them into something like `benchmarks/analyze.py` in this repo, consuming a directory of historical `dfbench` JSON results and complementing `compare.py`'s pairwise comparison. Would that be welcome here, or is the preference to keep the analysis layer in the external benchmarking repo (or go the Codspeed route)?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org