You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Saket Kumar <sa...@bgch.co.uk> on 2014/10/27 13:10:39 UTC

Missing java.util.Date class error while running Spark job

Hello all,

I am trying to run a Spark job but while running the job I am getting missing class errors. First I got a missing class error for Joda DateTime, I added the dependency in my build.sbt and rebuilt the assembly jar and tried again. This time I am getting the same error for java.util.Date. I find that error odd as I shouldn’t have to package a core java class in the assembly package. Am I missing something? Shouldn’t it load the class from rt. jar?

4/10/27 11:54:09 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 5, localhost): java.io.IOException: Exception during preparation of SELECT "mpxn", "reading_date", "division", "insert_date", "insert_etl_id", "msn", "register_name", "sourcekey", "t0000", "t0030", "t0100", "t0130", "t0200", "t0230", "t0300", "t0330", "t0400", "t0430", "t0500", "t0530", "t0600", "t0630", "t0700", "t0730", "t0800", "t0830", "t0900", "t0930", "t1000", "t1030", "t1100", "t1130", "t1200", "t1230", "t1300", "t1330", "t1400", "t1430", "t1500", "t1530", "t1600", "t1630", "t1700", "t1730", "t1800", "t1830", "t1900", "t1930", "t2000", "t2030", "t2100", "t2130", "t2200", "t2230", "t2300", "t2330", "trilliant_smart_read_id", "update_date", "update_etl_id" FROM "ser_ingest"."profile_reads" WHERE token("mpxn") > ? AND token("mpxn") <= ? ALLOW FILTERING: class java.util.Date in JavaMirror with org.apache.spark.executor.ExecutorURLClassLoader@6b69dc09 of type class org.apache.spark.executor.ExecutorURLClassLoader with classpath [file:/tmp/transformation.jar] and parent being sun.misc.Launcher$AppClassLoader@2494310b of type class sun.misc.Launcher$AppClassLoader with classpath [file:/etc/spark/spark-1.1.0-bin-hadoop2.4/,file:/etc/spark/spark-1.1.0-bin-hadoop2.4/,file:/etc/spark/spark-1.1.0-bin-hadoop2.4/conf/,file:/etc/spark/spark-1.1.0-bin-hadoop2.4/lib/spark-assembly-1.1.0-hadoop2.4.0.jar] and parent being sun.misc.Launcher$ExtClassLoader@7d1fdbef of type class sun.misc.Launcher$ExtClassLoader with classpath [file:/usr/lib/jvm/jre1.7.0_60/lib/ext/localedata.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/sunec.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/zipfs.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/sunpkcs11.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/sunjce_provider.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/dnsns.jar] and parent being primordial classloader with boot classpath [/usr/lib/jvm/jre1.7.0_60/lib/resources.jar:/usr/lib/jvm/jre1.7.0_60/lib/rt.jar:/usr/lib/jvm/jre1.7.0_60/lib/sunrsasign.jar:/usr/lib/jvm/jre1.7.0_60/lib/jsse.jar:/usr/lib/jvm/jre1.7.0_60/lib/jce.jar:/usr/lib/jvm/jre1.7.0_60/lib/charsets.jar:/usr/lib/jvm/jre1.7.0_60/lib/jfr.jar:/usr/lib/jvm/jre1.7.0_60/classes] not found.


SBT file
import AssemblyKeys._

assemblySettings

jarName in assembly := "transformation.jar"

name := "dataplatform-readings-transformation"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  "joda-time" % "joda-time" % "2.5",
  "org.apache.spark" %% "spark-core" % "1.1.0" % "provided",
  "com.datastax.spark" %%	"spark-cassandra-connector" % "1.1.0-alpha4",
//  "com.datastax.spark" %%	"spark-cassandra-connector" % "1.0.3",
  "org.apache.spark" %% "spark-sql" % "1.1.0"
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
  case PathList("META-INF", "ECLIPSEF.RSA" ) => MergeStrategy.discard
  case PathList("META-INF", "mailcap" ) => MergeStrategy.discard
  case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("plugin.properties" ) => MergeStrategy.discard
  case PathList("META-INF","maven","org.slf4j","slf4j-api","pom.properties" ) => MergeStrategy.discard
  case PathList("META-INF","maven","org.slf4j","slf4j-api","pom.xml" ) => MergeStrategy.discard
  case x => old(x)
}
}

Thanks,
Saket

Re: Missing java.util.Date class error while running Spark job

Posted by Saket Kumar <sa...@bgch.co.uk>.
After trying around with few permutations I was able to get rid of this error by changing the datastax-cassandra-connector version to 1.1.0.alpha. Could it be related to a bug in alpha4? But why would it manifest itself as a class loading error?

On 27 Oct 2014, at 12:10, Saket Kumar <sa...@bgch.co.uk> wrote:

> Hello all,
> 
> I am trying to run a Spark job but while running the job I am getting missing class errors. First I got a missing class error for Joda DateTime, I added the dependency in my build.sbt and rebuilt the assembly jar and tried again. This time I am getting the same error for java.util.Date. I find that error odd as I shouldn’t have to package a core java class in the assembly package. Am I missing something? Shouldn’t it load the class from rt. jar?
> 
> 4/10/27 11:54:09 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 5, localhost): java.io.IOException: Exception during preparation of SELECT "mpxn", "reading_date", "division", "insert_date", "insert_etl_id", "msn", "register_name", "sourcekey", "t0000", "t0030", "t0100", "t0130", "t0200", "t0230", "t0300", "t0330", "t0400", "t0430", "t0500", "t0530", "t0600", "t0630", "t0700", "t0730", "t0800", "t0830", "t0900", "t0930", "t1000", "t1030", "t1100", "t1130", "t1200", "t1230", "t1300", "t1330", "t1400", "t1430", "t1500", "t1530", "t1600", "t1630", "t1700", "t1730", "t1800", "t1830", "t1900", "t1930", "t2000", "t2030", "t2100", "t2130", "t2200", "t2230", "t2300", "t2330", "trilliant_smart_read_id", "update_date", "update_etl_id" FROM "ser_ingest"."profile_reads" WHERE token("mpxn") > ? AND token("mpxn") <= ? ALLOW FILTERING: class java.util.Date in JavaMirror with org.apache.spark.executor.ExecutorURLClassLoader@6b69dc09 of type class org.apache.spark.executor.ExecutorURLClassLoader with classpath [file:/tmp/transformation.jar] and parent being sun.misc.Launcher$AppClassLoader@2494310b of type class sun.misc.Launcher$AppClassLoader with classpath [file:/etc/spark/spark-1.1.0-bin-hadoop2.4/,file:/etc/spark/spark-1.1.0-bin-hadoop2.4/,file:/etc/spark/spark-1.1.0-bin-hadoop2.4/conf/,file:/etc/spark/spark-1.1.0-bin-hadoop2.4/lib/spark-assembly-1.1.0-hadoop2.4.0.jar] and parent being sun.misc.Launcher$ExtClassLoader@7d1fdbef of type class sun.misc.Launcher$ExtClassLoader with classpath [file:/usr/lib/jvm/jre1.7.0_60/lib/ext/localedata.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/sunec.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/zipfs.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/sunpkcs11.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/sunjce_provider.jar,file:/usr/lib/jvm/jre1.7.0_60/lib/ext/dnsns.jar] and parent being primordial classloader with boot classpath [/usr/lib/jvm/jre1.7.0_60/lib/resources.jar:/usr/lib/jvm/jre1.7.0_60/lib/rt.jar:/usr/lib/jvm/jre1.7.0_60/lib/sunrsasign.jar:/usr/lib/jvm/jre1.7.0_60/lib/jsse.jar:/usr/lib/jvm/jre1.7.0_60/lib/jce.jar:/usr/lib/jvm/jre1.7.0_60/lib/charsets.jar:/usr/lib/jvm/jre1.7.0_60/lib/jfr.jar:/usr/lib/jvm/jre1.7.0_60/classes] not found.
> 
> 
> SBT file
> import AssemblyKeys._
> 
> assemblySettings
> 
> jarName in assembly := "transformation.jar"
> 
> name := "dataplatform-readings-transformation"
> 
> version := "1.0"
> 
> scalaVersion := "2.10.4"
> 
> libraryDependencies ++= Seq(
>   "joda-time" % "joda-time" % "2.5",
>   "org.apache.spark" %% "spark-core" % "1.1.0" % "provided",
>   "com.datastax.spark" %%	"spark-cassandra-connector" % "1.1.0-alpha4",
> //  "com.datastax.spark" %%	"spark-cassandra-connector" % "1.0.3",
>   "org.apache.spark" %% "spark-sql" % "1.1.0"
> )
> 
> mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
>   case PathList("META-INF", "ECLIPSEF.RSA" ) => MergeStrategy.discard
>   case PathList("META-INF", "mailcap" ) => MergeStrategy.discard
>   case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
>   case PathList("org", "apache", xs @ _*) => MergeStrategy.last
>   case PathList("plugin.properties" ) => MergeStrategy.discard
>   case PathList("META-INF","maven","org.slf4j","slf4j-api","pom.properties" ) => MergeStrategy.discard
>   case PathList("META-INF","maven","org.slf4j","slf4j-api","pom.xml" ) => MergeStrategy.discard
>   case x => old(x)
> }
> }
> 
> Thanks,
> Saket