[jira] [Created] (SPARK-29055) Memory leak in Spark Driver

["George Papa (Jira)" <ji...@apache.org>]

George Papa created SPARK-29055:
-----------------------------------

             Summary: Memory leak in Spark Driver
                 Key: SPARK-29055
                 URL: https://issues.apache.org/jira/browse/SPARK-29055
             Project: Spark
          Issue Type: Bug
          Components: Block Manager, Spark Core
    Affects Versions: 2.4.4, 2.4.3, 2.4.2, 2.4.1, 2.4.0, 2.3.3
            Reporter: George Papa
         Attachments: image-2019-09-11-16-13-20-588.png

In Spark 2.3.3+ the driver memory is increasing continuously. I don't have this issue with Spark 2.1.1.

In Spark 2.1.1 I see the ContextCleaner runs and cleans the driver and BlockManager removes the broadcast blocks from the memory, as you can see in the following screenshot:

!image-2019-09-11-16-09-06-720.png|width=685,height=89!

But in Spark 2.3.3+ I don't see this cleaning and the driver storage increases!!

*NOTE:* After few hours of use I have application interruption with the following error :

{color:#FF0000}java.lang.OutOfMemoryError: GC overhead limit exceeded{color}

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org