[GitHub] [spark] gengliangwang commented on issue #24783: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

[GitBox <gi...@apache.org>]

gengliangwang commented on issue #24783: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24783#issuecomment-498659343
 
 
   @IvanVergiliev I think the code in this PR is much simpler and readable. 
   
   The PR #24068 introduces two `ActionType`: `TrimUnconvertibleFilters` and `BuildSearchArgument`, and handles the two actions in one function:
   1. For the `And`/`Or`/`Not` nodes, the logic is complex to understand in one function
   2. For most of the leaf nodes, the convertible result is always Some(node), we can abstract it like this PR.
   
   This PR build a fully convertible tree first, and then convert the tree to SearchArgument very straightforwardly.  Putting the two procedures into two functions makes the logic cleaner. We can also see that the method `convertibleFilters` is quite short because is reuse the leaf code handling in method `createBuilder`.
   
   With respect, this PR still uses the benchmark in #24068, and it will be co-authored with you. I know there is a lot of work in #24068, but I prefer the simple implementation in this one.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org