[jira] [Commented] (SPARK-21424) Blockmanger has memory problems in high concurrent scenes

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088869#comment-16088869 ] 

Sean Owen commented on SPARK-21424:
-----------------------------------

This is pretty vague. Can you provide a clearer description of the problem, like, even a draft pull request? can you provide any numbers to show the performance issue?

> Blockmanger has memory problems in high concurrent scenes
> ---------------------------------------------------------
>
>                 Key: SPARK-21424
>                 URL: https://issues.apache.org/jira/browse/SPARK-21424
>             Project: Spark
>          Issue Type: Improvement
>          Components: Block Manager
>    Affects Versions: 2.1.1
>            Reporter: yiming.xu
>
> * Blockmanger removeBroadcast call entries in high concurrent scenes will copy a lot of hashmap, producing many tuple2, we should redesign the blockmanager.infos.
> * In the use of spark to do the application, will also submit a lot of the job, this time to produce a lot of boardcost, but after the completion of the task, clean up the boardcost need to copy the BlockInfoManager.infos traverse to get their own need to clear the block, which is very Unreasonable design, resulting in the generation of a large number of tuple2. The more tasks, the higher the memory occupied, seriously affected the jvm performance.We should design a new program, can make the block clean up do not need to copy so much data, but also To ensure that multi-threaded at the same time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org