Re: [I] Run DataFusion benchmarks regularly and track performance history over time [datafusion]

["vito1317 (via GitHub)" <gi...@apache.org>]

vito1317 commented on issue #5504:
URL: https://github.com/apache/datafusion/issues/5504#issuecomment-4962527462

   I took a stab at the remaining piece of this issue — running benchmarks regularly and tracking history with automated regression detection — building directly on @alamb's datafusion-benchmarking repo: alamb/datafusion-benchmarking#16
   
   It adds a cron-first nightly orchestrator (no CI-vendor dependency, per the guidance earlier in this thread) plus a statistical analysis layer addressing the "we'll probably just make a bunch of data we never look at" concern: per-query noise-adaptive thresholds (rustc-perf style IQR fences) and E-divisive changepoint detection (MongoDB's approach), both stdlib-only Python. Detected changes are annotated on the existing dashboard automatically.
   
   As a sanity check, the detector independently re-discovers the `collect_statistics` change (2d7ae09) as a changepoint on 7 ClickBench queries when run over the historical per-commit dataset.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org