[GitHub] [spark] LantaoJin commented on pull request #29869: [SPARK-32994][CORE] Update external heavy accumulators before they entering into listener event loop

[GitBox <gi...@apache.org>]

LantaoJin commented on pull request #29869:
URL: https://github.com/apache/spark/pull/29869#issuecomment-701717670


   Thanks @tgravescs , actually I agree with your comment. I just told you what I thought. Current code add a user facing changes in deed. I know current patch is not good. Actually to fix this problem I hardcode to filter out the delta SetAccumulator by name in our internal Spark. Open this PR is to help expose the issue and find a common solution. Now looks like there are four ways:
   1. Add a new prefix like “heavy” to merge them before entering loop. (Current patch, and no one like it)
   2. Merge accumulators which not start with  MATRICS_PREFIX before entering loop and document it.
   3. Merge accum which size over a threshold before entering ...
   4. Merge the collection type accum before entering ...
   
   More comments.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org