[jira] [Commented] (SPARK-9621) Closure inside RDD doesn't properly close over environment

["Joe Near (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658834#comment-14658834 ] 

Joe Near commented on SPARK-9621:
---------------------------------

Does that problem also affect non-case-classes? For example:

class MyTest() extends Serializable {}
val tv = new MyTest()
val is_true = tv == tv
val also_is_true = ((t: MyTest) => t == tv)(tv)
val res = sc.parallelize(Array((t: MyTest) => t == tv)).first()(tv)

The value "res" is false, while the others are true, as expected.

I'm also seeing the same behavior with a standalone app running locally (i.e. no shell) -- this seems also to be the case for the linked issue. Where exactly *can* case classes be used?


> Closure inside RDD doesn't properly close over environment
> ----------------------------------------------------------
>
>                 Key: SPARK-9621
>                 URL: https://issues.apache.org/jira/browse/SPARK-9621
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 1.4.1
>         Environment: Ubuntu 15.04, spark-1.4.1-bin-hadoop2.6 package
>            Reporter: Joe Near
>
> I expect the following:
> case class MyTest(i: Int)
> val tv = MyTest(1)
> val res = sc.parallelize(Array((t: MyTest) => t == tv)).first()(tv)
> to be "true." It is "false," when I type this into spark-shell. It seems the closure is changed somehow when it's serialized and deserialized.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org