[GitHub] [spark] cloud-fan commented on pull request #33903: [SPARK-36656][SQL] CollapseProject should not collapse correlated scalar subqueries

[GitBox <gi...@apache.org>]

cloud-fan commented on pull request #33903:
URL: https://github.com/apache/spark/pull/33903#issuecomment-916878091


   I've been thinking about it for a while, and this PR reminds me of the complaints to `CollapseProject` in the past. Correlated scalar subquery is not the only problem, `CollapseProject` may duplicate other expensive expressions and degrade the performance.
   
   When `CollapseProject` was added, there was no whole stage codegen, and a `Project` node was quite expensive that we'd better try our best to eliminate it. Now we have whole stage codegen, and an extra `Project` node is really not an issue. Duplicated expressions are a bigger problem today. I think we can just define an allowlist of "cheap" expressions, and only allow to duplicate them in `CollapseProject`: https://github.com/apache/spark/pull/33958


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org