[jira] [Resolved] (SPARK-7121) ClosureCleaner does not handle nesting properly

["Patrick Wendell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Wendell resolved SPARK-7121.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.4.0

> ClosureCleaner does not handle nesting properly
> -----------------------------------------------
>
>                 Key: SPARK-7121
>                 URL: https://issues.apache.org/jira/browse/SPARK-7121
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Andrew Or
>            Assignee: Andrew Or
>             Fix For: 1.4.0
>
>
> For instance, in SparkContext, I tried to do the following:
> {code}
> def scope[T](body: => T): T = body // no-op
> def myCoolMethod(path: String): RDD[String] = scope {
>   parallelize(1 to 10).map { _ => path }
> }
> {code}
> and I got an exception complaining that SparkContext is not serializable. The issue here is that the inner closure is getting its path from the outer closure (the scope), but the outer closure actually references the SparkContext object itself to get the `parallelize` method.
> Note, however, that the inner closure doesn't actually need the SparkContext; it just needs a field from the outer closure. If we modify ClosureCleaner to clean the outer closure recursively while using the fields accessed by the inner closure, then we can serialize the inner closure.
> This is blocking my effort on a visualization task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org