You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/07 17:23:17 UTC
[GitHub] [arrow-datafusion] isidentical commented on pull request #4128: Combined TPCH runs & uniformed summaries for benchmarks
isidentical commented on PR #4128:
URL: https://github.com/apache/arrow-datafusion/pull/4128#issuecomment-1305943969
While playing with this, I've also written a little Python script to function like a benchmark comparison UI (poor man's conbench): https://gist.github.com/isidentical/4e3fff1350e9d49672e15d54d9e8299f
It is quite basic, but I think it can automate a few stuff for https://github.com/datafusion-contrib/benchmark-automation/tree/main. E.g. an example comparison between
```
$ ./target/release/tpch benchmark datafusion --path /opt/data-parquet --format parquet --iterations 3 -o /tmp/benchmarks --disable-statistics
$ ./target/release/tpch benchmark datafusion --path /opt/data-parquet --format parquet --iterations 3 -o /tmp/benchmarks
$ python t.py compare /tmp/benchmarks/file1.json /tmp/benchmarks/file2.json
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ Baseline ┃ Comparison ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 702.18ms │ 687.86ms │ no change │
│ Q2 │ 413.74ms │ 302.22ms │ +1.37x faster │
│ Q3 │ 392.94ms │ 395.34ms │ no change │
│ Q4 │ 111.28ms │ 97.01ms │ +1.15x faster │
│ Q5 │ 465.81ms │ 487.92ms │ no change │
│ Q6 │ 402.94ms │ 402.48ms │ no change │
│ Q7 │ 868.18ms │ 889.51ms │ no change │
│ Q8 │ 499.98ms │ 468.68ms │ +1.07x faster │
│ Q9 │ 827.54ms │ 837.67ms │ no change │
│ Q10 │ 503.22ms │ 492.29ms │ no change │
│ Q11 │ 221.30ms │ 167.37ms │ +1.32x faster │
│ Q12 │ 204.10ms │ 170.99ms │ +1.19x faster │
│ Q13 │ 441.50ms │ 423.67ms │ no change │
│ Q14 │ 373.42ms │ 383.57ms │ no change │
│ Q15 │ 356.24ms │ 352.67ms │ no change │
│ Q16 │ 115.38ms │ 117.98ms │ no change │
│ Q17 │ 2099.22ms │ 2209.00ms │ 1.05x slower │
│ Q18 │ 1255.95ms │ 1285.39ms │ no change │
│ Q19 │ 656.93ms │ 660.46ms │ no change │
│ Q20 │ 640.30ms │ 624.94ms │ no change │
│ Q21 │ 697.55ms │ 685.22ms │ no change │
│ Q22 │ 84.20ms │ 81.76ms │ no change │
└──────────────┴──────────────┴──────────────┴───────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org