You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/12/23 11:52:22 UTC
[I] Benchmarks for planning queries [arrow-datafusion]
alamb opened a new issue, #8638:
URL: https://github.com/apache/arrow-datafusion/issues/8638
### Is your feature request related to a problem or challenge?
DataFusion has a variety of benchmarks we use for query *execution* -- that is how long it takes to run a query.
There are no equivalent benchmark suite for how long it takes to *plan* a query, an area that many people have highlighted as an area of DataFusion they would like to improve. (see https://github.com/apache/arrow-datafusion/issues/5637 for various ideas)
Recently we have had some PRs such as https://github.com/apache/arrow-datafusion/pull/7942 and https://github.com/apache/arrow-datafusion/pull/7870 that propose some non trivial planning change, including some micro benchmarks that show good promise. However, we don't have an agreed upon way to measure the changes overall impacts
### Describe the solution you'd like
As suggested by @Dandandan https://github.com/apache/arrow-datafusion/pull/7942#issuecomment-1864376725
> I suggest to also add some benchmarking. We could take for example TCP-H and TCP-DS (which we already have in the benchmarks / tests) and benchmark the time it takes to plan/optimize the queries rather than execute them.
Specifically, I propose adding benchmarks (with documentation about why they are included) in
https://github.com/apache/arrow-datafusion/blob/03c2ef46f2d88fb015ee305ab67df6d930b780e2/datafusion/core/benches/sql_planner.rs
The code would basically do
1. Create the schema
2. Plan the relevant query (create the physical plan) but not execute it
Contents:
* All 21 TPCH queries (can use the ones from [source location](https://github.com/apache/arrow-datafusion/tree/03c2ef46f2d88fb015ee305ab67df6d930b780e2/benchmarks/queries)),
* Clickbench queries ([source location here](https://github.com/apache/arrow-datafusion/tree/03c2ef46f2d88fb015ee305ab67df6d930b780e2/benchmarks/queries/clickbench))
* TPC-DS queries ([source location](https://github.com/apache/arrow-datafusion/tree/03c2ef46f2d88fb015ee305ab67df6d930b780e2/datafusion/core/tests/tpc-ds))
* Other relevant community supplied cases
### Describe alternatives you've considered
On alternative could be to update the dfbench tests so they can just plan but not run the queries:
> It seems it might not be much work adding an option to the benchmark code to only perform the planning rather than executing the queries.
The dfbench code is here: https://github.com/apache/arrow-datafusion/blob/main/benchmarks/src/bin/dfbench.rs
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Benchmarks for planning queries [arrow-datafusion]
Posted by "matthewmturner (via GitHub)" <gi...@apache.org>.
matthewmturner commented on issue #8638:
URL: https://github.com/apache/arrow-datafusion/issues/8638#issuecomment-1868524146
I will work on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Benchmarks for planning queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8638: Benchmarks for planning queries
URL: https://github.com/apache/arrow-datafusion/issues/8638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Benchmarks for planning queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8638:
URL: https://github.com/apache/arrow-datafusion/issues/8638#issuecomment-2050310079
FWIW I think we have a good set of benchmarks now (`sql_planner` benchmark) so let's close this for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Benchmarks for planning queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8638:
URL: https://github.com/apache/arrow-datafusion/issues/8638#issuecomment-1868277849
I recommend picking one of these test suites (perhaps TPCH or ClickBench) and figuring out the pattern for a benchmark test, and then working on the others
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org