You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/09 16:30:09 UTC

[GitHub] [spark] cloud-fan commented on pull request #36405: [SPARK-39065][SQL] DS V2 Limit push-down should avoid out of memory

cloud-fan commented on PR #36405:
URL: https://github.com/apache/spark/pull/36405#issuecomment-1121320934

   Please elaborate on the idea more. I don't get how to avoid OOM by not pushing down the limit. The tasks on executors will pull all the result set from data source anyway, isn't it? Then tasks will send the local limit result to the driver and do a final limit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org