[GitHub] [spark] cloud-fan commented on pull request #36921: [SPARK-39481][SQL] Do not push down complex filter condition

[GitBox <gi...@apache.org>]

cloud-fan commented on PR #36921:
URL: https://github.com/apache/spark/pull/36921#issuecomment-1230011294

   This reminds me of `CollapseProject`. We need to make decisions between "replicate expensive expressions" vs "have more Project operators". "have more Project operators" is not a big deal so `CollapseProject` is very conservative and only collapse project if only cheap expressions are replicated. However, filter pushdown is a more complicated case as a predicate may reduce the input size dramatically, and replicating it is still worth it.
   
   @wangyum can you put more effort to justify your heuristic in this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org