You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by John Salvatier <js...@gmail.com> on 2014/01/28 00:39:52 UTC

Sporadic "IOException: Class not found" in ClosureCleaner

I am sporadically getting this error in the process of loading a couple of
data files. Its frequent but not consistent. I tried executing `ulimit -n
16000` before running the script (as recommended
here<https://groups.google.com/forum/#!msg/spark-users/cpi4uAMzjIU/yaJgGlVmdkcJ]>),
but this didn't seem to change anything.

java.io.IOException: Class not found
at org.objectweb.asm.ClassReader.a(Unknown Source)
at org.objectweb.asm.ClassReader.<init>(Unknown Source)
at
spark.ClosureCleaner$.spark$ClosureCleaner$$getClassReader(ClosureCleaner.scala:15)
at spark.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:59)
at spark.ClosureCleaner$.clean(ClosureCleaner.scala:82)
at spark.SparkContext.clean(SparkContext.scala:679)
at spark.RDD.flatMap(RDD.scala:223)

Any suggestions on what to look for? I'm on Spark version 0.7.3.

John