You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Gokul Pillai <go...@gmail.com> on 2010/06/29 00:26:34 UTC

No class definition found for org.apache.mahout.math.Vector when invoking K-means example

I made a new installation of Hadoop 0.20 from Cloudera.
I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this morning
and ran mvn install on core, utils and examples.
I tried running the KMeans Clustering example by doing the following as
mentioned in the tutorial:
hadoop jar ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job

And I keep getting this error although I have explicitly set my Classpath to
"/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-math-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":

10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
attempt_201006280959_0027_r_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.mahout.clustering.kmeans.KMeansCombiner
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
    at
org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
    at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
    at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException:
org.apache.mahout.clustering.kmeans.KMeansCombiner
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
    ... 5 more


Any help would be appreciated.

Regards
Gokul

Re: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by Gokul Pillai <go...@gmail.com>.
To clarify, do the fixes relate to only Java code or is there any script in
the mahout/trunk/bin folder too?
If so, how does one get that from the repo?

On Fri, Jul 9, 2010 at 4:16 PM, Gokul Pillai <go...@gmail.com> wrote:

> Peter:
> I just noticed that there has been a new patch applied by Drew yesterday
> for this. (https://issues.apache.org/jira/browse/MAHOUT-426)
>
> The last comment on the thread says "Integrated in Mahout-Quality #128
> (See http://hudson.zones.apache.org/hudson/job/Mahout-Quality/128/)"
>
> Does it mean that if I did a "mvn-U" on the trunk, I should be able to get
> the fix and then not run into the issue ?
>
> Regards
> gokul
>
> On Thu, Jul 1, 2010 at 2:31 PM, Gokul Pillai <go...@gmail.com> wrote:
>
>> The version on my instance is 0.20.1+152.
>> I am guessing if the jar file name is
>> "hadoop-0.20.1+152-fairscheduler.jar" then that is the version.
>>
>>
>> On Thu, Jul 1, 2010 at 11:11 AM, Peter M. Goldstein <
>> peter_m_goldstein@yahoo.com> wrote:
>>
>>> Gokul,
>>>
>>> What version of Hadoop are you using?  I'm running with 0.20+320, and I
>>> don't run into this issue.
>>>
>>> Thanks.
>>>
>>> Regards,
>>>
>>> Peter
>>>
>>> -----Original Message-----
>>> From: Gokul Pillai [mailto:gokooltech@gmail.com]
>>> Sent: Thursday, July 01, 2010 10:29 AM
>>> To: user@mahout.apache.org
>>> Subject: Re: No class definition found for org.apache.mahout.math.Vector
>>> when invoking K-means example
>>>
>>> Hello Peter:
>>> When i run that command, I get the following although different error:
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/commons/cli2/OptionException
>>>    at java.lang.Class.forName0(Native Method)
>>>    at java.lang.Class.forName(Class.java:247)
>>>    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.commons.cli2.OptionException
>>>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>
>>> Digging into this, I find that hadoop 0.20 ships with
>>> "commons-cli-1.2.jar"
>>> but mahout 0.40 Snapshot version depends on "commons-cli-2.0-mahout.jar"
>>> that is there in the $MAHOUT_HOME/trunk/examples/target/ folder once you
>>> "mvn install" in the  "examples" folder.
>>>
>>> So i replaced the jar file in hadoop/lib with this one and then it goes
>>> back
>>> to the ClassNotFound error for Vector class.
>>>
>>> Regards
>>> Gokul
>>>
>>> On Thu, Jul 1, 2010 at 9:29 AM, Peter M. Goldstein <
>>> peter_m_goldstein@yahoo.com> wrote:
>>>
>>> > Hi Gokul,
>>> >
>>> > Using the Mahout on Amazon EC2 directions I'm able to get a little
>>> farther
>>> > than this.  Specifically, using the job file created when building the
>>> > examples I don't get any ClassNotFoundExceptions.  Instead I get an
>>> > InvalidInputException (presumably because I didn't prepare any input
>>> data
>>> > for the job).  If you provide some appropriately formatted candidate
>>> data
>>> > I'd be happy to try and run a complete test.
>>> >
>>> > In my test I'm using the command line:
>>> >
>>> > $HADOOP_HOME/bin/hadoop jar
>>> > $MAHOUT_HOME/examples/target/mahout-examples-0.4-SNAPSHOT.job
>>> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>>> >
>>> > Again, it's critical to build and use the examples Job file.  If I
>>> attempt
>>> > to use the core Job file instead key classes appear to be absent.
>>> >
>>> > Hope that helps.
>>> >
>>> > --Peter
>>> >
>>> > -----Original Message-----
>>> > From: Gokul Pillai [mailto:gokooltech@gmail.com]
>>> > Sent: Wednesday, June 30, 2010 2:35 PM
>>> > To: user@mahout.apache.org
>>> > Subject: Re: No class definition found for
>>> org.apache.mahout.math.Vector
>>> > when invoking K-means example
>>> >
>>> > Peter
>>> > Thank you for your response. However, after making your changes, it
>>> still
>>> > does not work for me.
>>> > In fact I tried another way which is:
>>> > 1. create a lib folder within my Job Jar-file and put the
>>> > "org.apache.mahout.math.Vector" and other dependent jars.
>>> > 2. Launch the command via "hadoop jar ./mahout-examples-with-libs.jar
>>> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job"
>>> >
>>> > However, I still get the error:
>>> > "Error: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
>>> >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)"
>>> >
>>> >
>>> > Another thing I tried is modify the $HADOOP_HOME/conf/hadoop-env.sh and
>>> > edit
>>> > the HADOOP_CLASSPATH to include the dependent libraries. Even that does
>>> not
>>> > work.
>>> >
>>> > Regards
>>> > Gokul
>>> >
>>> > On Mon, Jun 28, 2010 at 3:43 PM, Peter M. Goldstein <
>>> > peter_m_goldstein@yahoo.com> wrote:
>>> >
>>> > > Hi Gokul,
>>> > >
>>> > > You may want to see the following bug reports:
>>> > >
>>> > > https://issues.apache.org/jira/browse/MAHOUT-426
>>> > >
>>> > > https://issues.apache.org/jira/browse/MAHOUT-427
>>> > >
>>> > > https://issues.apache.org/jira/browse/MAHOUT-428
>>> > >
>>> > > I ran into similar issues as you and had to some minor patching to
>>> get
>>> > > everything to work.
>>> > >
>>> > > Also, make sure you run an explicit "mvn install" from within the
>>> > examples
>>> > > directory.  I had to do this to get the JOB files to build correctly.
>>> > > Getting the JOB files to build correctly solves (sort of) issues 426
>>> and
>>> > > 427, so you only need to apply the patch described in 428.
>>> > >
>>> > > Hope that helps.
>>> > >
>>> > > Regards,
>>> > >
>>> > > Peter
>>> > >
>>> > > -----Original Message-----
>>> > > From: Gokul Pillai [mailto:gokooltech@gmail.com]
>>> > > Sent: Monday, June 28, 2010 3:27 PM
>>> > > To: user@mahout.apache.org
>>> > > Subject: No class definition found for org.apache.mahout.math.Vector
>>> when
>>> > > invoking K-means example
>>> > >
>>> > > I made a new installation of Hadoop 0.20 from Cloudera.
>>> > > I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this
>>> morning
>>> > > and ran mvn install on core, utils and examples.
>>> > > I tried running the KMeans Clustering example by doing the following
>>> as
>>> > > mentioned in the tutorial:
>>> > > hadoop jar
>>> > ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
>>> > > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>>> > >
>>> > > And I keep getting this error although I have explicitly set my
>>> Classpath
>>> > > to
>>> > >
>>> > >
>>> >
>>> >
>>>
>>> "/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
>>> > >
>>> > >
>>> >
>>> >
>>>
>>> ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
>>> > > lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":
>>> > >
>>> > > 10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
>>> > > attempt_201006280959_0027_r_000000_1, Status : FAILED
>>> > > java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> > > org.apache.mahout.clustering.kmeans.KMeansCombiner
>>> > >    at
>>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
>>> > >    at
>>> > >
>>> > >
>>> >
>>> >
>>>
>>> org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
>>> > >    at
>>> org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
>>> > >    at
>>> > >
>>> > >
>>> >
>>> >
>>>
>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
>>> > > )
>>> > >    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
>>> > >    at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> > > Caused by: java.lang.ClassNotFoundException:
>>> > > org.apache.mahout.clustering.kmeans.KMeansCombiner
>>> > >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>> > >    at java.security.AccessController.doPrivileged(Native Method)
>>> > >    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>> > >    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> > >    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> > >    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>> > >    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>> > >    at java.lang.Class.forName0(Native Method)
>>> > >    at java.lang.Class.forName(Class.java:247)
>>> > >    at
>>> > >
>>> >
>>>
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
>>> > >    at
>>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
>>> > >    ... 5 more
>>> > >
>>> > >
>>> > > Any help would be appreciated.
>>> > >
>>> > > Regards
>>> > > Gokul
>>> > >
>>> > >
>>> > >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>
>

Re: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by Gokul Pillai <go...@gmail.com>.
Peter:
I just noticed that there has been a new patch applied by Drew yesterday for
this. (https://issues.apache.org/jira/browse/MAHOUT-426)

The last comment on the thread says "Integrated in Mahout-Quality #128 (See
http://hudson.zones.apache.org/hudson/job/Mahout-Quality/128/)"

Does it mean that if I did a "mvn-U" on the trunk, I should be able to get
the fix and then not run into the issue ?

Regards
gokul

On Thu, Jul 1, 2010 at 2:31 PM, Gokul Pillai <go...@gmail.com> wrote:

> The version on my instance is 0.20.1+152.
> I am guessing if the jar file name is "hadoop-0.20.1+152-fairscheduler.jar"
> then that is the version.
>
>
> On Thu, Jul 1, 2010 at 11:11 AM, Peter M. Goldstein <
> peter_m_goldstein@yahoo.com> wrote:
>
>> Gokul,
>>
>> What version of Hadoop are you using?  I'm running with 0.20+320, and I
>> don't run into this issue.
>>
>> Thanks.
>>
>> Regards,
>>
>> Peter
>>
>> -----Original Message-----
>> From: Gokul Pillai [mailto:gokooltech@gmail.com]
>> Sent: Thursday, July 01, 2010 10:29 AM
>> To: user@mahout.apache.org
>> Subject: Re: No class definition found for org.apache.mahout.math.Vector
>> when invoking K-means example
>>
>> Hello Peter:
>> When i run that command, I get the following although different error:
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/commons/cli2/OptionException
>>    at java.lang.Class.forName0(Native Method)
>>    at java.lang.Class.forName(Class.java:247)
>>    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.commons.cli2.OptionException
>>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>
>> Digging into this, I find that hadoop 0.20 ships with
>> "commons-cli-1.2.jar"
>> but mahout 0.40 Snapshot version depends on "commons-cli-2.0-mahout.jar"
>> that is there in the $MAHOUT_HOME/trunk/examples/target/ folder once you
>> "mvn install" in the  "examples" folder.
>>
>> So i replaced the jar file in hadoop/lib with this one and then it goes
>> back
>> to the ClassNotFound error for Vector class.
>>
>> Regards
>> Gokul
>>
>> On Thu, Jul 1, 2010 at 9:29 AM, Peter M. Goldstein <
>> peter_m_goldstein@yahoo.com> wrote:
>>
>> > Hi Gokul,
>> >
>> > Using the Mahout on Amazon EC2 directions I'm able to get a little
>> farther
>> > than this.  Specifically, using the job file created when building the
>> > examples I don't get any ClassNotFoundExceptions.  Instead I get an
>> > InvalidInputException (presumably because I didn't prepare any input
>> data
>> > for the job).  If you provide some appropriately formatted candidate
>> data
>> > I'd be happy to try and run a complete test.
>> >
>> > In my test I'm using the command line:
>> >
>> > $HADOOP_HOME/bin/hadoop jar
>> > $MAHOUT_HOME/examples/target/mahout-examples-0.4-SNAPSHOT.job
>> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>> >
>> > Again, it's critical to build and use the examples Job file.  If I
>> attempt
>> > to use the core Job file instead key classes appear to be absent.
>> >
>> > Hope that helps.
>> >
>> > --Peter
>> >
>> > -----Original Message-----
>> > From: Gokul Pillai [mailto:gokooltech@gmail.com]
>> > Sent: Wednesday, June 30, 2010 2:35 PM
>> > To: user@mahout.apache.org
>> > Subject: Re: No class definition found for org.apache.mahout.math.Vector
>> > when invoking K-means example
>> >
>> > Peter
>> > Thank you for your response. However, after making your changes, it
>> still
>> > does not work for me.
>> > In fact I tried another way which is:
>> > 1. create a lib folder within my Job Jar-file and put the
>> > "org.apache.mahout.math.Vector" and other dependent jars.
>> > 2. Launch the command via "hadoop jar ./mahout-examples-with-libs.jar
>> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job"
>> >
>> > However, I still get the error:
>> > "Error: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
>> >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)"
>> >
>> >
>> > Another thing I tried is modify the $HADOOP_HOME/conf/hadoop-env.sh and
>> > edit
>> > the HADOOP_CLASSPATH to include the dependent libraries. Even that does
>> not
>> > work.
>> >
>> > Regards
>> > Gokul
>> >
>> > On Mon, Jun 28, 2010 at 3:43 PM, Peter M. Goldstein <
>> > peter_m_goldstein@yahoo.com> wrote:
>> >
>> > > Hi Gokul,
>> > >
>> > > You may want to see the following bug reports:
>> > >
>> > > https://issues.apache.org/jira/browse/MAHOUT-426
>> > >
>> > > https://issues.apache.org/jira/browse/MAHOUT-427
>> > >
>> > > https://issues.apache.org/jira/browse/MAHOUT-428
>> > >
>> > > I ran into similar issues as you and had to some minor patching to get
>> > > everything to work.
>> > >
>> > > Also, make sure you run an explicit "mvn install" from within the
>> > examples
>> > > directory.  I had to do this to get the JOB files to build correctly.
>> > > Getting the JOB files to build correctly solves (sort of) issues 426
>> and
>> > > 427, so you only need to apply the patch described in 428.
>> > >
>> > > Hope that helps.
>> > >
>> > > Regards,
>> > >
>> > > Peter
>> > >
>> > > -----Original Message-----
>> > > From: Gokul Pillai [mailto:gokooltech@gmail.com]
>> > > Sent: Monday, June 28, 2010 3:27 PM
>> > > To: user@mahout.apache.org
>> > > Subject: No class definition found for org.apache.mahout.math.Vector
>> when
>> > > invoking K-means example
>> > >
>> > > I made a new installation of Hadoop 0.20 from Cloudera.
>> > > I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this
>> morning
>> > > and ran mvn install on core, utils and examples.
>> > > I tried running the KMeans Clustering example by doing the following
>> as
>> > > mentioned in the tutorial:
>> > > hadoop jar
>> > ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
>> > > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>> > >
>> > > And I keep getting this error although I have explicitly set my
>> Classpath
>> > > to
>> > >
>> > >
>> >
>> >
>>
>> "/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
>> > >
>> > >
>> >
>> >
>>
>> ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
>> > > lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":
>> > >
>> > > 10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
>> > > attempt_201006280959_0027_r_000000_1, Status : FAILED
>> > > java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> > > org.apache.mahout.clustering.kmeans.KMeansCombiner
>> > >    at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
>> > >    at
>> > >
>> > >
>> >
>> >
>>
>> org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
>> > >    at
>> org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
>> > >    at
>> > >
>> > >
>> >
>> >
>>
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
>> > > )
>> > >    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
>> > >    at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> > > Caused by: java.lang.ClassNotFoundException:
>> > > org.apache.mahout.clustering.kmeans.KMeansCombiner
>> > >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> > >    at java.security.AccessController.doPrivileged(Native Method)
>> > >    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> > >    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > >    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> > >    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> > >    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> > >    at java.lang.Class.forName0(Native Method)
>> > >    at java.lang.Class.forName(Class.java:247)
>> > >    at
>> > >
>> >
>>
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
>> > >    at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
>> > >    ... 5 more
>> > >
>> > >
>> > > Any help would be appreciated.
>> > >
>> > > Regards
>> > > Gokul
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>>
>

Re: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by Gokul Pillai <go...@gmail.com>.
The version on my instance is 0.20.1+152.
I am guessing if the jar file name is "hadoop-0.20.1+152-fairscheduler.jar"
then that is the version.

On Thu, Jul 1, 2010 at 11:11 AM, Peter M. Goldstein <
peter_m_goldstein@yahoo.com> wrote:

> Gokul,
>
> What version of Hadoop are you using?  I'm running with 0.20+320, and I
> don't run into this issue.
>
> Thanks.
>
> Regards,
>
> Peter
>
> -----Original Message-----
> From: Gokul Pillai [mailto:gokooltech@gmail.com]
> Sent: Thursday, July 01, 2010 10:29 AM
> To: user@mahout.apache.org
> Subject: Re: No class definition found for org.apache.mahout.math.Vector
> when invoking K-means example
>
> Hello Peter:
> When i run that command, I get the following although different error:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/cli2/OptionException
>    at java.lang.Class.forName0(Native Method)
>    at java.lang.Class.forName(Class.java:247)
>    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.cli2.OptionException
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>
> Digging into this, I find that hadoop 0.20 ships with "commons-cli-1.2.jar"
> but mahout 0.40 Snapshot version depends on "commons-cli-2.0-mahout.jar"
> that is there in the $MAHOUT_HOME/trunk/examples/target/ folder once you
> "mvn install" in the  "examples" folder.
>
> So i replaced the jar file in hadoop/lib with this one and then it goes
> back
> to the ClassNotFound error for Vector class.
>
> Regards
> Gokul
>
> On Thu, Jul 1, 2010 at 9:29 AM, Peter M. Goldstein <
> peter_m_goldstein@yahoo.com> wrote:
>
> > Hi Gokul,
> >
> > Using the Mahout on Amazon EC2 directions I'm able to get a little
> farther
> > than this.  Specifically, using the job file created when building the
> > examples I don't get any ClassNotFoundExceptions.  Instead I get an
> > InvalidInputException (presumably because I didn't prepare any input data
> > for the job).  If you provide some appropriately formatted candidate data
> > I'd be happy to try and run a complete test.
> >
> > In my test I'm using the command line:
> >
> > $HADOOP_HOME/bin/hadoop jar
> > $MAHOUT_HOME/examples/target/mahout-examples-0.4-SNAPSHOT.job
> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
> >
> > Again, it's critical to build and use the examples Job file.  If I
> attempt
> > to use the core Job file instead key classes appear to be absent.
> >
> > Hope that helps.
> >
> > --Peter
> >
> > -----Original Message-----
> > From: Gokul Pillai [mailto:gokooltech@gmail.com]
> > Sent: Wednesday, June 30, 2010 2:35 PM
> > To: user@mahout.apache.org
> > Subject: Re: No class definition found for org.apache.mahout.math.Vector
> > when invoking K-means example
> >
> > Peter
> > Thank you for your response. However, after making your changes, it still
> > does not work for me.
> > In fact I tried another way which is:
> > 1. create a lib folder within my Job Jar-file and put the
> > "org.apache.mahout.math.Vector" and other dependent jars.
> > 2. Launch the command via "hadoop jar ./mahout-examples-with-libs.jar
> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job"
> >
> > However, I still get the error:
> > "Error: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
> >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)"
> >
> >
> > Another thing I tried is modify the $HADOOP_HOME/conf/hadoop-env.sh and
> > edit
> > the HADOOP_CLASSPATH to include the dependent libraries. Even that does
> not
> > work.
> >
> > Regards
> > Gokul
> >
> > On Mon, Jun 28, 2010 at 3:43 PM, Peter M. Goldstein <
> > peter_m_goldstein@yahoo.com> wrote:
> >
> > > Hi Gokul,
> > >
> > > You may want to see the following bug reports:
> > >
> > > https://issues.apache.org/jira/browse/MAHOUT-426
> > >
> > > https://issues.apache.org/jira/browse/MAHOUT-427
> > >
> > > https://issues.apache.org/jira/browse/MAHOUT-428
> > >
> > > I ran into similar issues as you and had to some minor patching to get
> > > everything to work.
> > >
> > > Also, make sure you run an explicit "mvn install" from within the
> > examples
> > > directory.  I had to do this to get the JOB files to build correctly.
> > > Getting the JOB files to build correctly solves (sort of) issues 426
> and
> > > 427, so you only need to apply the patch described in 428.
> > >
> > > Hope that helps.
> > >
> > > Regards,
> > >
> > > Peter
> > >
> > > -----Original Message-----
> > > From: Gokul Pillai [mailto:gokooltech@gmail.com]
> > > Sent: Monday, June 28, 2010 3:27 PM
> > > To: user@mahout.apache.org
> > > Subject: No class definition found for org.apache.mahout.math.Vector
> when
> > > invoking K-means example
> > >
> > > I made a new installation of Hadoop 0.20 from Cloudera.
> > > I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this
> morning
> > > and ran mvn install on core, utils and examples.
> > > I tried running the KMeans Clustering example by doing the following as
> > > mentioned in the tutorial:
> > > hadoop jar
> > ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
> > > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
> > >
> > > And I keep getting this error although I have explicitly set my
> Classpath
> > > to
> > >
> > >
> >
> >
>
> "/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
> > >
> > >
> >
> >
>
> ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
> > > lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":
> > >
> > > 10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
> > > attempt_201006280959_0027_r_000000_1, Status : FAILED
> > > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > > org.apache.mahout.clustering.kmeans.KMeansCombiner
> > >    at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
> > >    at
> > >
> > >
> >
> >
>
> org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
> > >    at
> org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
> > >    at
> > >
> > >
> >
> >
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
> > > )
> > >    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
> > >    at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.mahout.clustering.kmeans.KMeansCombiner
> > >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > >    at java.security.AccessController.doPrivileged(Native Method)
> > >    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > >    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > >    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> > >    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> > >    at java.lang.Class.forName0(Native Method)
> > >    at java.lang.Class.forName(Class.java:247)
> > >    at
> > >
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
> > >    at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
> > >    ... 5 more
> > >
> > >
> > > Any help would be appreciated.
> > >
> > > Regards
> > > Gokul
> > >
> > >
> > >
> >
> >
> >
>
>
>

RE: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Gokul,

What version of Hadoop are you using?  I'm running with 0.20+320, and I
don't run into this issue.

Thanks.

Regards,

Peter

-----Original Message-----
From: Gokul Pillai [mailto:gokooltech@gmail.com] 
Sent: Thursday, July 01, 2010 10:29 AM
To: user@mahout.apache.org
Subject: Re: No class definition found for org.apache.mahout.math.Vector
when invoking K-means example

Hello Peter:
When i run that command, I get the following although different error:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/cli2/OptionException
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.cli2.OptionException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)

Digging into this, I find that hadoop 0.20 ships with "commons-cli-1.2.jar"
but mahout 0.40 Snapshot version depends on "commons-cli-2.0-mahout.jar"
that is there in the $MAHOUT_HOME/trunk/examples/target/ folder once you
"mvn install" in the  "examples" folder.

So i replaced the jar file in hadoop/lib with this one and then it goes back
to the ClassNotFound error for Vector class.

Regards
Gokul

On Thu, Jul 1, 2010 at 9:29 AM, Peter M. Goldstein <
peter_m_goldstein@yahoo.com> wrote:

> Hi Gokul,
>
> Using the Mahout on Amazon EC2 directions I'm able to get a little farther
> than this.  Specifically, using the job file created when building the
> examples I don't get any ClassNotFoundExceptions.  Instead I get an
> InvalidInputException (presumably because I didn't prepare any input data
> for the job).  If you provide some appropriately formatted candidate data
> I'd be happy to try and run a complete test.
>
> In my test I'm using the command line:
>
> $HADOOP_HOME/bin/hadoop jar
> $MAHOUT_HOME/examples/target/mahout-examples-0.4-SNAPSHOT.job
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>
> Again, it's critical to build and use the examples Job file.  If I attempt
> to use the core Job file instead key classes appear to be absent.
>
> Hope that helps.
>
> --Peter
>
> -----Original Message-----
> From: Gokul Pillai [mailto:gokooltech@gmail.com]
> Sent: Wednesday, June 30, 2010 2:35 PM
> To: user@mahout.apache.org
> Subject: Re: No class definition found for org.apache.mahout.math.Vector
> when invoking K-means example
>
> Peter
> Thank you for your response. However, after making your changes, it still
> does not work for me.
> In fact I tried another way which is:
> 1. create a lib folder within my Job Jar-file and put the
> "org.apache.mahout.math.Vector" and other dependent jars.
> 2. Launch the command via "hadoop jar ./mahout-examples-with-libs.jar
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job"
>
> However, I still get the error:
> "Error: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)"
>
>
> Another thing I tried is modify the $HADOOP_HOME/conf/hadoop-env.sh and
> edit
> the HADOOP_CLASSPATH to include the dependent libraries. Even that does
not
> work.
>
> Regards
> Gokul
>
> On Mon, Jun 28, 2010 at 3:43 PM, Peter M. Goldstein <
> peter_m_goldstein@yahoo.com> wrote:
>
> > Hi Gokul,
> >
> > You may want to see the following bug reports:
> >
> > https://issues.apache.org/jira/browse/MAHOUT-426
> >
> > https://issues.apache.org/jira/browse/MAHOUT-427
> >
> > https://issues.apache.org/jira/browse/MAHOUT-428
> >
> > I ran into similar issues as you and had to some minor patching to get
> > everything to work.
> >
> > Also, make sure you run an explicit "mvn install" from within the
> examples
> > directory.  I had to do this to get the JOB files to build correctly.
> > Getting the JOB files to build correctly solves (sort of) issues 426 and
> > 427, so you only need to apply the patch described in 428.
> >
> > Hope that helps.
> >
> > Regards,
> >
> > Peter
> >
> > -----Original Message-----
> > From: Gokul Pillai [mailto:gokooltech@gmail.com]
> > Sent: Monday, June 28, 2010 3:27 PM
> > To: user@mahout.apache.org
> > Subject: No class definition found for org.apache.mahout.math.Vector
when
> > invoking K-means example
> >
> > I made a new installation of Hadoop 0.20 from Cloudera.
> > I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this
morning
> > and ran mvn install on core, utils and examples.
> > I tried running the KMeans Clustering example by doing the following as
> > mentioned in the tutorial:
> > hadoop jar
> ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
> >
> > And I keep getting this error although I have explicitly set my
Classpath
> > to
> >
> >
>
>
"/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
> >
> >
>
>
ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
> > lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":
> >
> > 10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
> > attempt_201006280959_0027_r_000000_1, Status : FAILED
> > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > org.apache.mahout.clustering.kmeans.KMeansCombiner
> >    at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
> >    at
> >
> >
>
>
org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
> >    at
org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
> >    at
> >
> >
>
>
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
> > )
> >    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
> >    at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.mahout.clustering.kmeans.KMeansCombiner
> >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> >    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> >    at java.lang.Class.forName0(Native Method)
> >    at java.lang.Class.forName(Class.java:247)
> >    at
> >
>
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
> >    at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
> >    ... 5 more
> >
> >
> > Any help would be appreciated.
> >
> > Regards
> > Gokul
> >
> >
> >
>
>
>



Re: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by Gokul Pillai <go...@gmail.com>.
Hello Peter:
When i run that command, I get the following although different error:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/cli2/OptionException
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.cli2.OptionException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)

Digging into this, I find that hadoop 0.20 ships with "commons-cli-1.2.jar"
but mahout 0.40 Snapshot version depends on "commons-cli-2.0-mahout.jar"
that is there in the $MAHOUT_HOME/trunk/examples/target/ folder once you
"mvn install" in the  "examples" folder.

So i replaced the jar file in hadoop/lib with this one and then it goes back
to the ClassNotFound error for Vector class.

Regards
Gokul

On Thu, Jul 1, 2010 at 9:29 AM, Peter M. Goldstein <
peter_m_goldstein@yahoo.com> wrote:

> Hi Gokul,
>
> Using the Mahout on Amazon EC2 directions I'm able to get a little farther
> than this.  Specifically, using the job file created when building the
> examples I don't get any ClassNotFoundExceptions.  Instead I get an
> InvalidInputException (presumably because I didn't prepare any input data
> for the job).  If you provide some appropriately formatted candidate data
> I'd be happy to try and run a complete test.
>
> In my test I'm using the command line:
>
> $HADOOP_HOME/bin/hadoop jar
> $MAHOUT_HOME/examples/target/mahout-examples-0.4-SNAPSHOT.job
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>
> Again, it's critical to build and use the examples Job file.  If I attempt
> to use the core Job file instead key classes appear to be absent.
>
> Hope that helps.
>
> --Peter
>
> -----Original Message-----
> From: Gokul Pillai [mailto:gokooltech@gmail.com]
> Sent: Wednesday, June 30, 2010 2:35 PM
> To: user@mahout.apache.org
> Subject: Re: No class definition found for org.apache.mahout.math.Vector
> when invoking K-means example
>
> Peter
> Thank you for your response. However, after making your changes, it still
> does not work for me.
> In fact I tried another way which is:
> 1. create a lib folder within my Job Jar-file and put the
> "org.apache.mahout.math.Vector" and other dependent jars.
> 2. Launch the command via "hadoop jar ./mahout-examples-with-libs.jar
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job"
>
> However, I still get the error:
> "Error: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)"
>
>
> Another thing I tried is modify the $HADOOP_HOME/conf/hadoop-env.sh and
> edit
> the HADOOP_CLASSPATH to include the dependent libraries. Even that does not
> work.
>
> Regards
> Gokul
>
> On Mon, Jun 28, 2010 at 3:43 PM, Peter M. Goldstein <
> peter_m_goldstein@yahoo.com> wrote:
>
> > Hi Gokul,
> >
> > You may want to see the following bug reports:
> >
> > https://issues.apache.org/jira/browse/MAHOUT-426
> >
> > https://issues.apache.org/jira/browse/MAHOUT-427
> >
> > https://issues.apache.org/jira/browse/MAHOUT-428
> >
> > I ran into similar issues as you and had to some minor patching to get
> > everything to work.
> >
> > Also, make sure you run an explicit "mvn install" from within the
> examples
> > directory.  I had to do this to get the JOB files to build correctly.
> > Getting the JOB files to build correctly solves (sort of) issues 426 and
> > 427, so you only need to apply the patch described in 428.
> >
> > Hope that helps.
> >
> > Regards,
> >
> > Peter
> >
> > -----Original Message-----
> > From: Gokul Pillai [mailto:gokooltech@gmail.com]
> > Sent: Monday, June 28, 2010 3:27 PM
> > To: user@mahout.apache.org
> > Subject: No class definition found for org.apache.mahout.math.Vector when
> > invoking K-means example
> >
> > I made a new installation of Hadoop 0.20 from Cloudera.
> > I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this morning
> > and ran mvn install on core, utils and examples.
> > I tried running the KMeans Clustering example by doing the following as
> > mentioned in the tutorial:
> > hadoop jar
> ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
> > org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
> >
> > And I keep getting this error although I have explicitly set my Classpath
> > to
> >
> >
>
> "/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
> >
> >
>
> ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
> > lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":
> >
> > 10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
> > attempt_201006280959_0027_r_000000_1, Status : FAILED
> > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > org.apache.mahout.clustering.kmeans.KMeansCombiner
> >    at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
> >    at
> >
> >
>
> org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
> >    at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
> >    at
> >
> >
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
> > )
> >    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
> >    at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.mahout.clustering.kmeans.KMeansCombiner
> >    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> >    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> >    at java.lang.Class.forName0(Native Method)
> >    at java.lang.Class.forName(Class.java:247)
> >    at
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
> >    at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
> >    ... 5 more
> >
> >
> > Any help would be appreciated.
> >
> > Regards
> > Gokul
> >
> >
> >
>
>
>

RE: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Hi Gokul,

Using the Mahout on Amazon EC2 directions I'm able to get a little farther
than this.  Specifically, using the job file created when building the
examples I don't get any ClassNotFoundExceptions.  Instead I get an
InvalidInputException (presumably because I didn't prepare any input data
for the job).  If you provide some appropriately formatted candidate data
I'd be happy to try and run a complete test.

In my test I'm using the command line:

$HADOOP_HOME/bin/hadoop jar
$MAHOUT_HOME/examples/target/mahout-examples-0.4-SNAPSHOT.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job

Again, it's critical to build and use the examples Job file.  If I attempt
to use the core Job file instead key classes appear to be absent.

Hope that helps.

--Peter

-----Original Message-----
From: Gokul Pillai [mailto:gokooltech@gmail.com] 
Sent: Wednesday, June 30, 2010 2:35 PM
To: user@mahout.apache.org
Subject: Re: No class definition found for org.apache.mahout.math.Vector
when invoking K-means example

Peter
Thank you for your response. However, after making your changes, it still
does not work for me.
In fact I tried another way which is:
1. create a lib folder within my Job Jar-file and put the
"org.apache.mahout.math.Vector" and other dependent jars.
2. Launch the command via "hadoop jar ./mahout-examples-with-libs.jar
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job"

However, I still get the error:
"Error: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)"


Another thing I tried is modify the $HADOOP_HOME/conf/hadoop-env.sh and edit
the HADOOP_CLASSPATH to include the dependent libraries. Even that does not
work.

Regards
Gokul

On Mon, Jun 28, 2010 at 3:43 PM, Peter M. Goldstein <
peter_m_goldstein@yahoo.com> wrote:

> Hi Gokul,
>
> You may want to see the following bug reports:
>
> https://issues.apache.org/jira/browse/MAHOUT-426
>
> https://issues.apache.org/jira/browse/MAHOUT-427
>
> https://issues.apache.org/jira/browse/MAHOUT-428
>
> I ran into similar issues as you and had to some minor patching to get
> everything to work.
>
> Also, make sure you run an explicit "mvn install" from within the examples
> directory.  I had to do this to get the JOB files to build correctly.
> Getting the JOB files to build correctly solves (sort of) issues 426 and
> 427, so you only need to apply the patch described in 428.
>
> Hope that helps.
>
> Regards,
>
> Peter
>
> -----Original Message-----
> From: Gokul Pillai [mailto:gokooltech@gmail.com]
> Sent: Monday, June 28, 2010 3:27 PM
> To: user@mahout.apache.org
> Subject: No class definition found for org.apache.mahout.math.Vector when
> invoking K-means example
>
> I made a new installation of Hadoop 0.20 from Cloudera.
> I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this morning
> and ran mvn install on core, utils and examples.
> I tried running the KMeans Clustering example by doing the following as
> mentioned in the tutorial:
> hadoop jar ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>
> And I keep getting this error although I have explicitly set my Classpath
> to
>
>
"/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
>
>
ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
> lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":
>
> 10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
> attempt_201006280959_0027_r_000000_1, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.clustering.kmeans.KMeansCombiner
>    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
>    at
>
>
org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
>    at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
>    at
>
>
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
> )
>    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
>    at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.clustering.kmeans.KMeansCombiner
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>    at java.lang.Class.forName0(Native Method)
>    at java.lang.Class.forName(Class.java:247)
>    at
>
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
>    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
>    ... 5 more
>
>
> Any help would be appreciated.
>
> Regards
> Gokul
>
>
>



Re: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by Gokul Pillai <go...@gmail.com>.
Peter
Thank you for your response. However, after making your changes, it still
does not work for me.
In fact I tried another way which is:
1. create a lib folder within my Job Jar-file and put the
"org.apache.mahout.math.Vector" and other dependent jars.
2. Launch the command via "hadoop jar ./mahout-examples-with-libs.jar
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job"

However, I still get the error:
"Error: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)"


Another thing I tried is modify the $HADOOP_HOME/conf/hadoop-env.sh and edit
the HADOOP_CLASSPATH to include the dependent libraries. Even that does not
work.

Regards
Gokul

On Mon, Jun 28, 2010 at 3:43 PM, Peter M. Goldstein <
peter_m_goldstein@yahoo.com> wrote:

> Hi Gokul,
>
> You may want to see the following bug reports:
>
> https://issues.apache.org/jira/browse/MAHOUT-426
>
> https://issues.apache.org/jira/browse/MAHOUT-427
>
> https://issues.apache.org/jira/browse/MAHOUT-428
>
> I ran into similar issues as you and had to some minor patching to get
> everything to work.
>
> Also, make sure you run an explicit "mvn install" from within the examples
> directory.  I had to do this to get the JOB files to build correctly.
> Getting the JOB files to build correctly solves (sort of) issues 426 and
> 427, so you only need to apply the patch described in 428.
>
> Hope that helps.
>
> Regards,
>
> Peter
>
> -----Original Message-----
> From: Gokul Pillai [mailto:gokooltech@gmail.com]
> Sent: Monday, June 28, 2010 3:27 PM
> To: user@mahout.apache.org
> Subject: No class definition found for org.apache.mahout.math.Vector when
> invoking K-means example
>
> I made a new installation of Hadoop 0.20 from Cloudera.
> I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this morning
> and ran mvn install on core, utils and examples.
> I tried running the KMeans Clustering example by doing the following as
> mentioned in the tutorial:
> hadoop jar ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>
> And I keep getting this error although I have explicitly set my Classpath
> to
>
> "/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
>
> ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
> lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":
>
> 10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
> attempt_201006280959_0027_r_000000_1, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.clustering.kmeans.KMeansCombiner
>    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
>    at
>
> org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
>    at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
>    at
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
> )
>    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
>    at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.clustering.kmeans.KMeansCombiner
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>    at java.lang.Class.forName0(Native Method)
>    at java.lang.Class.forName(Class.java:247)
>    at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
>    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
>    ... 5 more
>
>
> Any help would be appreciated.
>
> Regards
> Gokul
>
>
>

RE: No class definition found for org.apache.mahout.math.Vector when invoking K-means example

Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Hi Gokul,

You may want to see the following bug reports:

https://issues.apache.org/jira/browse/MAHOUT-426

https://issues.apache.org/jira/browse/MAHOUT-427

https://issues.apache.org/jira/browse/MAHOUT-428

I ran into similar issues as you and had to some minor patching to get
everything to work.  

Also, make sure you run an explicit "mvn install" from within the examples
directory.  I had to do this to get the JOB files to build correctly.
Getting the JOB files to build correctly solves (sort of) issues 426 and
427, so you only need to apply the patch described in 428.

Hope that helps.

Regards,

Peter

-----Original Message-----
From: Gokul Pillai [mailto:gokooltech@gmail.com] 
Sent: Monday, June 28, 2010 3:27 PM
To: user@mahout.apache.org
Subject: No class definition found for org.apache.mahout.math.Vector when
invoking K-means example

I made a new installation of Hadoop 0.20 from Cloudera.
I then installed Mahout 0.4-SNAPSHOT by doing a SVN checkout this morning
and ran mvn install on core, utils and examples.
I tried running the KMeans Clustering example by doing the following as
mentioned in the tutorial:
hadoop jar ~/mahout/trunk/examples/target/mahout-examples-0.4-SNAPSHOT.jar
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job

And I keep getting this error although I have explicitly set my Classpath to
"/usr/lib/hadoop/lib/mahout-collections-1.0.jar:/usr/lib/hadoop/lib/mahout-m
ath-0.4-SNAPSHOT.jar:/usr/lib/hadoop/lib/mahout-utils-0.4-SNAPSHOT.jar:/usr/
lib/hadoop/lib/mahout-core-0.4-SNAPSHOT.jar":

10/06/28 15:19:21 INFO mapred.JobClient: Task Id :
attempt_201006280959_0027_r_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.mahout.clustering.kmeans.KMeansCombiner
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
    at
org.apache.hadoop.mapreduce.JobContext.getCombinerClass(JobContext.java:169)
    at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1107)
    at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:1780
)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:375)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException:
org.apache.mahout.clustering.kmeans.KMeansCombiner
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
    ... 5 more


Any help would be appreciated.

Regards
Gokul



improving SVD performance

Posted by Tamas Jambor <ja...@gmail.com>.
hi,

one more thing in terms of the SVD based recommender performance. There 
is one single reason why it is so slow. When it does training there is a 
phrase when it calculates the dot product of two vectors for every 
single entry. If I want to train on the Netflix dataset (100m entries) 
using 60 iterations and 64 features that comes down to 100m*64*60 
multiplications. But this could be improved since there is only one 
entry in each vector that is updated at each iteration. If the rest was 
cached that would reduce the whole thing to 100m*60 multiplications.

implemented this just now, if interested adding it.

Tamas