[GitHub] [spark] peter-toth commented on pull request #41119: [SPARK-42551][SQL] Support more subexpression elimination cases

["peter-toth (via GitHub)" <gi...@apache.org>]

peter-toth commented on PR #41119:
URL: https://github.com/apache/spark/pull/41119#issuecomment-1600417186

   @wankunde, I didn't foget about this PR just realized that we can improve the current subexpression elimination logic in terms of recognizing the surely evaluated subexpressions and keeping track of conditionally evaluated ones.
   
   IMO if surely evaluated count is >= 2 or surely evaluated count = 1 but there is also a certain probability of evaluating the subexpression conditionally then we can include the subexpression for elimination. (This is not a new idea, basically @cloud-fan already mentioned this in https://github.com/apache/spark/pull/32987#issuecomment-1258168996). So I opened a PR for tracking the expected count of sure and conditional evaluations here: https://github.com/apache/spark/pull/41677. (Need to add a few more tests so it is just WIP now...)
   
   If that PR gets accepted we can revisit those subexpressions that have only conditional evaluations at 2+ places and apply your lazy construct idea proposed in this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org