[jira] [Created] (SPARK-21424) Blockmanger has memory problems in high concurrent scenes

["yiming.xu (JIRA)" <ji...@apache.org>]

yiming.xu created SPARK-21424:
---------------------------------

             Summary: Blockmanger has memory problems in high concurrent scenes
                 Key: SPARK-21424
                 URL: https://issues.apache.org/jira/browse/SPARK-21424
             Project: Spark
          Issue Type: Improvement
          Components: Block Manager
    Affects Versions: 2.1.1
            Reporter: yiming.xu


Blockmanger removeBroadcast call entries in high concurrent scenes will copy a lot of hashmap, producing many tuple2, we should redesign the blockmanager.infos.
In the use of spark to do the application, will also submit a lot of the job, this time to produce a lot of boardcost, but after the completion of the task, clean up the boardcost need to copy the infos traverse to get their own need to clear the block, which is very Unreasonable settings, resulting in the generation of a large number of tuple2. The more tasks, the higher the memory occupied, seriously affected the jvm performance.We should design a new program, can make the block clean up do not need to copy so much data, but also To ensure that multi-threaded at the same time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org