[jira] [Resolved] (SPARK-1042) spark cleans all java broadcast variables when it hits the spark.cleaner.ttl

["Patrick Wendell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Wendell resolved SPARK-1042.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 0.9.2

I think this was fixed back in 0.9.2

> spark cleans all java broadcast variables when it hits the spark.cleaner.ttl 
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-1042
>                 URL: https://issues.apache.org/jira/browse/SPARK-1042
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API, Spark Core
>    Affects Versions: 0.8.0, 0.8.1, 0.9.0
>            Reporter: Tal Sliwowicz
>            Assignee: OuyangJin
>            Priority: Critical
>              Labels: memory_leak
>             Fix For: 0.9.2
>
>
> When setting spark.cleaner.ttl, spark performs the cleanup on time - but it cleans all broadcast variables, not just the ones that are older than the ttl. This creates an exception when the next mapPartitions runs because it cannot find the broadcast variable, even when it was created immediately before running the task.
> Our temp workaround - not set the ttl and suffer from an ongoing memory leak (forces a restart).
> We are using JavaSparkContext and our broadcast variables are Java HashMaps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org