[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

["Jose Soltren (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928660#comment-15928660 ] 

Jose Soltren commented on SPARK-19982:
--------------------------------------

[~cloud_fan] added this test as some work related to SPARK-11954. Wenchen, do you have any thoughts as to why this might be failing intermittently?

Likely not, but I wonder if this has anything to do with outerScopes being a lazy val in object OuterScopes. Then, possibly, in very rare instances, Analyzer.scala:ResolveNewInstance could hit the (outer == null) branch and throw this exception.

We run all the Spark unit tests about a dozen times a night and this has failed on average twice a month since last May (which is as far back as my data goes).

> JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19982
>                 URL: https://issues.apache.org/jira/browse/SPARK-19982
>             Project: Spark
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 2.1.0
>            Reporter: Jose Soltren
>
> JavaDatasetSuite.testJavaBeanEncoder fails sporadically with the error below:
> Unable to generate an encoder for inner class `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to the scope that this class was defined in. Try moving this class out of its parent class.
> From https://spark-tests.appspot.com/test-logs/35475788
> [~vanzin] looked into this back in October and reported:
> I ran this test in a loop (both alone and with the rest of the spark-sql tests) and never got a failure. I even used the same JDK as Jenkins (1.7.0_51).
> Also looked at the code and nothing seems wrong. The errors is when an entry with the parent class name is missing from the map kept in OuterScopes.scala, but the test populates that map in its first line. So it doesn't look like a race nor some issue with weak references (the map uses weak values).
>   public void testJavaBeanEncoder() {
>     OuterScopes.addOuterScope(this);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org