You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Ryan Blue (JIRA)" <ji...@apache.org> on 2016/07/07 16:22:11 UTC

[jira] [Created] (SPARK-16420) UnsafeShuffleWriter leaks compression streams with off-heap memory.

Ryan Blue created SPARK-16420:
---------------------------------

             Summary: UnsafeShuffleWriter leaks compression streams with off-heap memory.
                 Key: SPARK-16420
                 URL: https://issues.apache.org/jira/browse/SPARK-16420
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.0.0
            Reporter: Ryan Blue


When merging spill files using Java file streams, {{UnsafeShuffleWriter}} opens a decompression stream for each shuffle part and [never closes them|https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java#L352-357]. When the compression codec holds off-heap resources, these aren't cleaned up until the finalizer is called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org