You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wang Shuo (JIRA)" <ji...@apache.org> on 2018/12/20 08:11:00 UTC

[jira] [Created] (SPARK-26418) Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService

Wang Shuo created SPARK-26418:
---------------------------------

             Summary: Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService
                 Key: SPARK-26418
                 URL: https://issues.apache.org/jira/browse/SPARK-26418
             Project: Spark
          Issue Type: Bug
          Components: Shuffle
    Affects Versions: 2.4.0
            Reporter: Wang Shuo


In current code path,  OneForOneStreamManager holds StreamState in a Map named streams. 

A StreamState is initialized and put into streams when OpenBlocks request received.

One specific StreamState is removed from streams in two scenarios below:
 # The last chunk of a stream is fetched
 # The connection of ChunkFetch is closed

StreamState will never be clean up, if OpenBlocks request is received without and following  ChunkFetch request. This will cause memory leak in server side, which is harmful for long running service such as ExternalShuffleService.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org