You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/06 15:01:01 UTC
[GitHub] [spark] mridulm edited a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
mridulm edited a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894320564
@srowen You are right that the total computation is slightly worse (an additional fold with zero value at driver and ofcourse the extra shuffle).
The main issue is what @venkata91 mentioned - at executor, it is done in context of shuffle, where memory is better managed (not necessarily because executor has more memory).
At driver, it is essentially fetching all partitions (single value per partition) as part of the `fold`'s `runJob` : for nontrivial values, this causes OOM at driver, failing the application (memory usage for io + N serde in addition to the fold).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org