You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Pat Ferrel <pa...@occamsmachete.com> on 2015/03/16 17:01:34 UTC

NoClassDefFoundError: for commons/IOUtils

Trying to write an app that does the following:

    sparkConf.set("spark.kryo.referenceTracking", "false")
      .set("spark.kryoserializer.buffer.mb", "200")// this is default for Mahout optimizer, change it with -D option
    mc = mahoutSparkContext(masterUrl = master, appName = appName, sparkConf = sparkConf)
 
But I get the following runtime error

Connected to the target VM, address: '127.0.0.1:63145', transport: 'socket'
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/common/IOUtils
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:98)
	at com.fb.spark.SparkJobContext$class.setupSparkContext(SparkJobContext.scala:37)
	at com.fb.drivers.CooccurrenceDriver$.setupSparkContext(CooccurrenceDriver.scala:35)
	at com.fb.drivers.CooccurrenceDriver$delayedInit$body.apply(CooccurrenceDriver.scala:61)

IOUTils are used in the mahoutSparkContext call and I’ve tried using them in my own code, which works fine.

commons.oi is in the Mahout main pom and for good measure I put it in my own app’s build.sbt in an identical manner so I’m a bit baffled. It’s as if something in the mahoutSparkContext call itself is somehow removing the jar with common.io in it.

Any ideas?

Re: NoClassDefFoundError: for commons/IOUtils

Posted by Pat Ferrel <pa...@occamsmachete.com>.
Hmm, it’s because MAHOUT_HOME is not discovered even though it is set in the environment.


Nothing to do with commons, it’s just that some exceptions leave the classpath messed up and so unable to perform the “finally"

On Mar 16, 2015, at 9:01 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:

Trying to write an app that does the following:

   sparkConf.set("spark.kryo.referenceTracking", "false")
     .set("spark.kryoserializer.buffer.mb", "200")// this is default for Mahout optimizer, change it with -D option
   mc = mahoutSparkContext(masterUrl = master, appName = appName, sparkConf = sparkConf)

But I get the following runtime error

Connected to the target VM, address: '127.0.0.1:63145', transport: 'socket'
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/common/IOUtils
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:98)
	at com.fb.spark.SparkJobContext$class.setupSparkContext(SparkJobContext.scala:37)
	at com.fb.drivers.CooccurrenceDriver$.setupSparkContext(CooccurrenceDriver.scala:35)
	at com.fb.drivers.CooccurrenceDriver$delayedInit$body.apply(CooccurrenceDriver.scala:61)

IOUTils are used in the mahoutSparkContext call and I’ve tried using them in my own code, which works fine.

commons.oi is in the Mahout main pom and for good measure I put it in my own app’s build.sbt in an identical manner so I’m a bit baffled. It’s as if something in the mahoutSparkContext call itself is somehow removing the jar with common.io in it.

Any ideas?