You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Prashant Sharma (JIRA)" <ji...@apache.org> on 2018/05/15 10:49:00 UTC
[jira] [Updated] (SPARK-24279) Incompatible byte code errors, when
using test-jar of spark sql.
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Sharma updated SPARK-24279:
------------------------------------
Description:
Using test libraries available with spark sql streaming, produces weird incompatible byte code errors. It is already tested on a different virtual box instance to make sure, that it is not related to my system environment. A reproducer is uploaded to github.
[https://github.com/ScrapCodes/spark-bug-reproducer]
Doing a clean build reproduces the error.
Verbatim paste of the error.
{code:java}
[INFO] Compiling 1 source files to /home/prashant/work/test/target/test-classes at 1526380360990
[ERROR] error: missing or invalid dependency detected while loading class file 'QueryTest.class'.
[INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[INFO] A full rebuild may help if 'QueryTest.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'.
[INFO] Could not access type PlanTestBase in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtils.class'.
[INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
[ERROR] val inputData = MemoryStream[Int]
[ERROR] ^
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
[ERROR] CheckAnswer(2, 3, 4))
[ERROR] ^
[ERROR] 5 errors found
{code}
was:
Using test libraries available with spark sql streaming, produces weird incompatible byte code errors. It is already tested on a different virtual box instance to make sure, that it is not related to my system environment. A reproducer is uploaded to github.
[https://github.com/ScrapCodes/spark-bug-reproducer]
Doing a clean build reproduces the error.
> Incompatible byte code errors, when using test-jar of spark sql.
> ----------------------------------------------------------------
>
> Key: SPARK-24279
> URL: https://issues.apache.org/jira/browse/SPARK-24279
> Project: Spark
> Issue Type: Bug
> Components: SQL, Structured Streaming
> Affects Versions: 2.3.0, 2.4.0
> Reporter: Prashant Sharma
> Priority: Major
>
> Using test libraries available with spark sql streaming, produces weird incompatible byte code errors. It is already tested on a different virtual box instance to make sure, that it is not related to my system environment. A reproducer is uploaded to github.
> [https://github.com/ScrapCodes/spark-bug-reproducer]
>
> Doing a clean build reproduces the error.
>
> Verbatim paste of the error.
> {code:java}
> [INFO] Compiling 1 source files to /home/prashant/work/test/target/test-classes at 1526380360990
> [ERROR] error: missing or invalid dependency detected while loading class file 'QueryTest.class'.
> [INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
> [INFO] A full rebuild may help if 'QueryTest.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'.
> [INFO] Could not access type PlanTestBase in package org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtils.class'.
> [INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
> [ERROR] val inputData = MemoryStream[Int]
> [ERROR] ^
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
> [ERROR] CheckAnswer(2, 3, 4))
> [ERROR] ^
> [ERROR] 5 errors found
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org