You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/02/28 19:41:04 UTC

[jira] [Updated] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14

     [ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-6069:
-----------------------------
    Priority: Critical  (was: Major)
     Summary: Deserialization Error ClassNotFoundException with Kryo, Guava 14  (was: Deserialization Error ClassNotFound )

To clarify the properties situation, in Spark 1.2.x we have {{spark.files.userClassPathFirst}} _and_ {{spark.yarn.user.classpath.first}}. {{spark.driver.userClassPathFirst}} and {{spark.executor.userClassPathFirst}} are the new more logical versions in 1.3+ only. So ignore those.

{{spark.yarn.user.classpath.first}} is actually what I am setting:
https://github.com/OryxProject/oryx/blob/master/oryx-lambda/src/main/java/com/cloudera/oryx/lambda/BatchLayer.java#L153

But it sounds like you are not using YARN.

Guava 14.0.1 is packaged with the app:
https://github.com/OryxProject/oryx/blob/master/pom.xml#L233

I'm running this on 1.2.0 + YARN, and also local[*] + 1.3.0-SNAPSHOT.

My ? is whether this is perhaps not working for standalone in 1.2 but does in 1.3, since there has been some overhaul to this mechanism since 1.2.

> Deserialization Error ClassNotFoundException with Kryo, Guava 14
> ----------------------------------------------------------------
>
>                 Key: SPARK-6069
>                 URL: https://issues.apache.org/jira/browse/SPARK-6069
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.1
>         Environment: Standalone one worker cluster on localhost, or any cluster
>            Reporter: Pat Ferrel
>            Priority: Critical
>
> A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound.
> The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like:
> spark.executor.extraClassPath      /path/to/some.jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org