You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/25 20:15:07 UTC
[GitHub] [arrow-datafusion] Dandandan opened a new issue #78: Address performance/execution plan of TPCH query 19
Dandandan opened a new issue #78:
URL: https://github.com/apache/arrow-datafusion/issues/78
**Describe the bug**
TPC-H Query 9 consumes a lot of memory and takes forever in memory on a 8 core machine on SF=1 in memory (with 100%cpu usage across cores) to execute. That likely has to do with the plan misses a equi-join and turns it into a cross join.
**To Reproduce**
Run query 19, check plan and execution time.
**Expected behavior**
The query is expected to finish faster (<1s) and shouldn't need a cross join.
**Additional context**
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #78: Address performance/execution plan of TPCH query 19
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #78:
URL: https://github.com/apache/arrow-datafusion/issues/78#issuecomment-827528710
Cool, thanks @alamb !
I guess there are also some more opportunities for rewriting in the above example:
`l_shipmode in (‘AIR’, ‘AIR REG’)` and `and l_shipinstruct = ‘DELIVER IN PERSON` are repeated and could benefit from being simplified and then pushed down.
`p_container in ( ‘SM CASE’, ‘SM BOX’, ‘SM PACK’, ‘SM PKG’)`, `p_container in (‘MED BAG’, ‘MED BOX’, ‘MED PKG’, ‘MED PACK’)` ,`p_container in ( ‘LG CASE’, ‘LG BOX’, ‘LG PACK’, ‘LG PKG’)`
could be adding a filter with all the items.
`p_size between 1 and 15` , ` p_size between 1 and 10`, `p_size between 1 and 5` could generate a filter with the least restrictive filter `p_size between 1 and 15`
> Shall I file an issue to track this kind of rewrtite?
Would be nice to have some tracking issue :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #78: Address performance/execution plan of TPCH query 19
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #78:
URL: https://github.com/apache/arrow-datafusion/issues/78#issuecomment-827318099
@alamb I am wondering if you think it would be the case to introduce this boolean optimization (Distributivity of and over or):
(p and q1) OR (p and q2) should be replaced by p AND (q1 or q2)
p in this example could be our join predicate.
After this optimization, the physical planner should be able to see the predicate in the filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #78: Address performance/execution plan of TPCH query 19
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #78:
URL: https://github.com/apache/arrow-datafusion/issues/78#issuecomment-828457951
Filed https://github.com/apache/arrow-datafusion/issues/217 to track the idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #78: Address performance/execution plan of TPCH query 19
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #78:
URL: https://github.com/apache/arrow-datafusion/issues/78#issuecomment-827513734
@Dandandan -- yes, I think the "classic" thing to do is a "predicate rewrite" pass that rearranges predicates for further downstream operations
The goal is basically to get the predicate into a form of
`good_predicate1` AND `good_predicate2` AND ...
Where `good_predicate` means the predicate has special support in the execution engine.
Since OR is not typically handled specially, rewrites to AND are helpful
Rewrite 1: (p and q1) OR (p and q2) OR (p and ..) ==> p AND (q1 or q2)
Another common rewrite I have seen is
(col1 = A) OR (col1 = B) OR (col1 = C) ==> col1 IN (A, B, C)
Which then the execution engine can treat like a single column predicate (push down to scan) and build a hash table for `(A, B, C)` and do fast filtering.
Shall I file an issue to track this kind of rewrtite?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on issue #78: Address performance/execution plan of TPCH query 19
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on issue #78:
URL: https://github.com/apache/arrow-datafusion/issues/78#issuecomment-827318099
@alamb I am wondering if you think it would be the solution to introduce this boolean optimization (Distributivity of and over or):
(p and q1) OR (p and q2) should be replaced by p AND (q1 or q2)
p in this example could be our join predicate.
After this optimization, the physical planner should be able to see the predicate in the filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org