[jira] [Created] (SPARK-7121) ClosureCleaner does not handle nesting properly

["Andrew Or (JIRA)" <ji...@apache.org>]

Andrew Or created SPARK-7121:
--------------------------------

             Summary: ClosureCleaner does not handle nesting properly
                 Key: SPARK-7121
                 URL: https://issues.apache.org/jira/browse/SPARK-7121
             Project: Spark
          Issue Type: Bug
            Reporter: Andrew Or
            Assignee: Andrew Or


For instance, in SparkContext, I tried to do the following:
{code}
def scope[T](body: => T): T = body // no-op
def myCoolMethod(path: String): RDD[String] = scope {
  parallelize(1 to 10).map { _ => path }
}
{code}
and I got an exception complaining that SparkContext is not serializable. The issue here is that the inner closure is getting its path from the outer closure (the scope), but the outer closure actually references the SparkContext object itself to get the `parallelize` method.

Note, however, that the inner closure doesn't actually need the SparkContext; it just needs a field from the outer closure. If we modify ClosureCleaner to clean the outer closure recursively while using the fields accessed by the inner closure, then we can serialize the inner closure.

This is blocking my effort on a visualization task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org