You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Janina <ma...@googlemail.com> on 2012/03/15 10:42:56 UTC
ClassNotFoundException while using RecommenderJob
Hi all,
I am trying to run a RecommenderJob from a Java program. I have added the
files users.txt and input.txt to a Hadoop VM and use the run-method of
RecommenderJob to start the calculation. But the the following error
message occurs while running the MapReducer:
Error: java.lang.ClassNotFoundException: com.google.common.primitives.Longs
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
at
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
at
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
I have added the required guava-r09.jar explicitly to my jar which also
lays on the hadoop cluster.
This may be a stupid question, but does anyone know where this error comes
from? This would help me a lot.
Thanks and greetings,
Janina
Re: ClassNotFoundException while using RecommenderJob
Posted by Lance Norskog <go...@gmail.com>.
Wait! I thought there was only one RecommenderJob?
On Thu, Mar 15, 2012 at 3:44 AM, Sean Owen <sr...@gmail.com> wrote:
> You would still need to use the 'job' file generated by the build to
> get an artifact with all the dependencies.
> You don't need to add Guava as a dependency; it already is one. It's
> the job file that you're missing.
>
> There are two RecommenderJobs. One is what I call pseudo-distributed,
> yes. The other is a fully distributed item-based recommender.
>
> Sean
>
> On Thu, Mar 15, 2012 at 10:01 AM, Janina <ma...@googlemail.com> wrote:
>> Thanks for your fast answer.
>>
>> I haven't added the jar manually, but by adding the dependency to the
>> pom.xml. I tried it with and without the dependendy and with different
>> versions of the dependency, but it remained the same error message.
>>
>> But the RecommenderJob is meant to be to run a pseudo distributed
>> recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
>> have another possibility to run recommendations on a Hadoop Cluster? I have
>> read that only the clustering and classification parts of mahout are really
>> able to be distributed on a hadoop cluster.
>>
>> 2012/3/15 Sean Owen <sr...@gmail.com>
>>
>>> You shouldn't have to add anything to your jar, if you use the
>>> supplied 'job' file which contains all transitive dependencies.
>>> If you do add your own jars, I think you need to unpack and repack
>>> them, not put them into the overall jar as a jar file, even with a
>>> MANIFEST.MF entry. I am not sure that works on Hadoop.
>>>
>>> On Thu, Mar 15, 2012 at 9:42 AM, Janina <ma...@googlemail.com>
>>> wrote:
>>> > Hi all,
>>> >
>>> > I am trying to run a RecommenderJob from a Java program. I have added the
>>> > files users.txt and input.txt to a Hadoop VM and use the run-method of
>>> > RecommenderJob to start the calculation. But the the following error
>>> > message occurs while running the MapReducer:
>>> >
>>> > Error: java.lang.ClassNotFoundException:
>>> com.google.common.primitives.Longs
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> > at
>>> >
>>> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
>>> > at
>>> >
>>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
>>> > at
>>> >
>>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
>>> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> >
>>> > I have added the required guava-r09.jar explicitly to my jar which also
>>> > lays on the hadoop cluster.
>>> > This may be a stupid question, but does anyone know where this error
>>> comes
>>> > from? This would help me a lot.
>>> >
>>> > Thanks and greetings,
>>> > Janina
>>>
--
Lance Norskog
goksron@gmail.com
Re: ClassNotFoundException while using RecommenderJob
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Thanks, [1] is a very good reference for ClassNotFoundException.
As you say in [2] the mahout script does not accept hadoop job
parameters in all cases although I hope it does in the future,
especially where a parameter to the job is a classname (seq2sparse for
instance).
What I had to do was copy my custom analyzer jar into $HADOOP_HOME/lib
on the master node. Evidently a symlink does not work, it appears you
have to copy each jar you want to the directory.
Then don't forget to stop and start hadoop because as [1] says it
packages the libs at startup.
On 3/16/12 7:12 AM, Ioan Eugen Stan wrote:
> Pe 15.03.2012 15:51, Pat Ferrel a scris:
>> Can you elaborate on how to set these classpaths?
>>
>> The job takes a classpath param, (from mahout --help)
>>
>> -libjars <paths> comma separated jar files to include in
>> the classpath.
>>
>> is the needed code bundled with the job or does the path need to be
>> available to hadoop/mahout on all nodes? Also is this the only thing
>> that affects the classpath to any code outside the jar file?
>>
>> On 3/15/12 5:33 AM, Ioan Eugen Stan wrote:
>>> Also, if you submit a job there are two class paths that you must be
>>> aware of:
>>> - the one that starts the job submitting app
>>> - the one that will be available to the job on each node
>>
>
> Hi Pat,
>
> To set up your main app class-path use HADOOP_CLASS_PATH env var. For
> the Job class-path it's best explained here [1], although, from my
> experience -libjars did not work as advertised. It's hadoop version
> dependant probably but I may be wrong. Test and see for yourself.
>
> Also see my post on common-user about this issue [2]. Search for
> ClassNotFoundException: libjars .
>
>
> [1]
> http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
> [2] http://hadoop.markmail.org/thread/st3cy7alcodlztng
>
> Hope this helps,
>
Re: ClassNotFoundException while using RecommenderJob
Posted by Ioan Eugen Stan <st...@gmail.com>.
Pe 15.03.2012 15:51, Pat Ferrel a scris:
> Can you elaborate on how to set these classpaths?
>
> The job takes a classpath param, (from mahout --help)
>
> -libjars <paths> comma separated jar files to include in
> the classpath.
>
> is the needed code bundled with the job or does the path need to be
> available to hadoop/mahout on all nodes? Also is this the only thing
> that affects the classpath to any code outside the jar file?
>
> On 3/15/12 5:33 AM, Ioan Eugen Stan wrote:
>> Also, if you submit a job there are two class paths that you must be
>> aware of:
>> - the one that starts the job submitting app
>> - the one that will be available to the job on each node
>
Hi Pat,
To set up your main app class-path use HADOOP_CLASS_PATH env var. For
the Job class-path it's best explained here [1], although, from my
experience -libjars did not work as advertised. It's hadoop version
dependant probably but I may be wrong. Test and see for yourself.
Also see my post on common-user about this issue [2]. Search for
ClassNotFoundException: libjars .
[1]
http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
[2] http://hadoop.markmail.org/thread/st3cy7alcodlztng
Hope this helps,
--
Ioan Eugen Stan
http://ieugen.blogspot.com
Re: ClassNotFoundException while using RecommenderJob
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Can you elaborate on how to set these classpaths?
The job takes a classpath param, (from mahout --help)
-libjars <paths> comma separated jar files to include in
the classpath.
is the needed code bundled with the job or does the path need to be
available to hadoop/mahout on all nodes? Also is this the only thing
that affects the classpath to any code outside the jar file?
On 3/15/12 5:33 AM, Ioan Eugen Stan wrote:
> Also, if you submit a job there are two class paths that you must be
> aware of:
> - the one that starts the job submitting app
> - the one that will be available to the job on each node
Re: ClassNotFoundException while using RecommenderJob
Posted by Ioan Eugen Stan <st...@gmail.com>.
Pe 15.03.2012 13:14, Janina a scris:
> Ahhh, ok...yes, I am using this generated jar file with Hadoop, but this
> does not change the error message about com.google.common.primitives.Longs
> .....
Hello Janina,
Could you please enable bash debugging [1] and show us the command line
that submits your job?
Also, if you submit a job there are two class paths that you must be
aware of:
- the one that starts the job submitting app
- the one that will be available to the job on each node
Another thing, I see you are using both org.apache.hadoop.mapreduce and
org.apache.hadoop.mapred packages there. From what I know, mapred is
deprecated in favor of the new mapreduce so try to use the new api in
your code.
[1] http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html
Cheers,
--
Ioan Eugen Stan
http://ieugen.blogspot.com
Re: ClassNotFoundException while using RecommenderJob
Posted by Janina <ma...@googlemail.com>.
Ahhh, ok...yes, I am using this generated jar file with Hadoop, but this
does not change the error message about com.google.common.primitives.Longs
.....
2012/3/15 Sean Owen <sr...@gmail.com>
> After 'mvn package' you should see a file ending in 'job.jar' under
> target/ This is the jar file to use with Hadoop.
>
> On Thu, Mar 15, 2012 at 10:56 AM, Janina <ma...@googlemail.com>
> wrote:
> > These are great news! I was not quite sure if the item based recommender
> is
> > fully distributes, but this helps! Thanks!
> >
> > I have removed the dependency now. But although it may again be a stupid
> > question: what do you exactly mean with "I am missing the job file"?
> Sorry,
> > but I am completely new to Mahout and Hadoop and I am very thankful for
> > your help :)
>
Re: ClassNotFoundException while using RecommenderJob
Posted by Sean Owen <sr...@gmail.com>.
After 'mvn package' you should see a file ending in 'job.jar' under
target/ This is the jar file to use with Hadoop.
On Thu, Mar 15, 2012 at 10:56 AM, Janina <ma...@googlemail.com> wrote:
> These are great news! I was not quite sure if the item based recommender is
> fully distributes, but this helps! Thanks!
>
> I have removed the dependency now. But although it may again be a stupid
> question: what do you exactly mean with "I am missing the job file"? Sorry,
> but I am completely new to Mahout and Hadoop and I am very thankful for
> your help :)
Re: ClassNotFoundException while using RecommenderJob
Posted by Janina <ma...@googlemail.com>.
These are great news! I was not quite sure if the item based recommender is
fully distributes, but this helps! Thanks!
I have removed the dependency now. But although it may again be a stupid
question: what do you exactly mean with "I am missing the job file"? Sorry,
but I am completely new to Mahout and Hadoop and I am very thankful for
your help :)
2012/3/15 Sean Owen <sr...@gmail.com>
> You would still need to use the 'job' file generated by the build to
> get an artifact with all the dependencies.
> You don't need to add Guava as a dependency; it already is one. It's
> the job file that you're missing.
>
> There are two RecommenderJobs. One is what I call pseudo-distributed,
> yes. The other is a fully distributed item-based recommender.
>
> Sean
>
> On Thu, Mar 15, 2012 at 10:01 AM, Janina <ma...@googlemail.com>
> wrote:
> > Thanks for your fast answer.
> >
> > I haven't added the jar manually, but by adding the dependency to the
> > pom.xml. I tried it with and without the dependendy and with different
> > versions of the dependency, but it remained the same error message.
> >
> > But the RecommenderJob is meant to be to run a pseudo distributed
> > recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
> > have another possibility to run recommendations on a Hadoop Cluster? I
> have
> > read that only the clustering and classification parts of mahout are
> really
> > able to be distributed on a hadoop cluster.
> >
> > 2012/3/15 Sean Owen <sr...@gmail.com>
> >
> >> You shouldn't have to add anything to your jar, if you use the
> >> supplied 'job' file which contains all transitive dependencies.
> >> If you do add your own jars, I think you need to unpack and repack
> >> them, not put them into the overall jar as a jar file, even with a
> >> MANIFEST.MF entry. I am not sure that works on Hadoop.
> >>
> >> On Thu, Mar 15, 2012 at 9:42 AM, Janina <ma...@googlemail.com>
> >> wrote:
> >> > Hi all,
> >> >
> >> > I am trying to run a RecommenderJob from a Java program. I have added
> the
> >> > files users.txt and input.txt to a Hadoop VM and use the run-method of
> >> > RecommenderJob to start the calculation. But the the following error
> >> > message occurs while running the MapReducer:
> >> >
> >> > Error: java.lang.ClassNotFoundException:
> >> com.google.common.primitives.Longs
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> > at
> >> >
> >>
> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
> >> > at
> >> >
> >>
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
> >> > at
> >> >
> >>
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
> >> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >> >
> >> > I have added the required guava-r09.jar explicitly to my jar which
> also
> >> > lays on the hadoop cluster.
> >> > This may be a stupid question, but does anyone know where this error
> >> comes
> >> > from? This would help me a lot.
> >> >
> >> > Thanks and greetings,
> >> > Janina
> >>
>
Re: ClassNotFoundException while using RecommenderJob
Posted by Sean Owen <sr...@gmail.com>.
You would still need to use the 'job' file generated by the build to
get an artifact with all the dependencies.
You don't need to add Guava as a dependency; it already is one. It's
the job file that you're missing.
There are two RecommenderJobs. One is what I call pseudo-distributed,
yes. The other is a fully distributed item-based recommender.
Sean
On Thu, Mar 15, 2012 at 10:01 AM, Janina <ma...@googlemail.com> wrote:
> Thanks for your fast answer.
>
> I haven't added the jar manually, but by adding the dependency to the
> pom.xml. I tried it with and without the dependendy and with different
> versions of the dependency, but it remained the same error message.
>
> But the RecommenderJob is meant to be to run a pseudo distributed
> recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
> have another possibility to run recommendations on a Hadoop Cluster? I have
> read that only the clustering and classification parts of mahout are really
> able to be distributed on a hadoop cluster.
>
> 2012/3/15 Sean Owen <sr...@gmail.com>
>
>> You shouldn't have to add anything to your jar, if you use the
>> supplied 'job' file which contains all transitive dependencies.
>> If you do add your own jars, I think you need to unpack and repack
>> them, not put them into the overall jar as a jar file, even with a
>> MANIFEST.MF entry. I am not sure that works on Hadoop.
>>
>> On Thu, Mar 15, 2012 at 9:42 AM, Janina <ma...@googlemail.com>
>> wrote:
>> > Hi all,
>> >
>> > I am trying to run a RecommenderJob from a Java program. I have added the
>> > files users.txt and input.txt to a Hadoop VM and use the run-method of
>> > RecommenderJob to start the calculation. But the the following error
>> > message occurs while running the MapReducer:
>> >
>> > Error: java.lang.ClassNotFoundException:
>> com.google.common.primitives.Longs
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> > at
>> >
>> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
>> > at
>> >
>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
>> > at
>> >
>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
>> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >
>> > I have added the required guava-r09.jar explicitly to my jar which also
>> > lays on the hadoop cluster.
>> > This may be a stupid question, but does anyone know where this error
>> comes
>> > from? This would help me a lot.
>> >
>> > Thanks and greetings,
>> > Janina
>>
Re: ClassNotFoundException while using RecommenderJob
Posted by Janina <ma...@googlemail.com>.
Thanks for your fast answer.
I haven't added the jar manually, but by adding the dependency to the
pom.xml. I tried it with and without the dependendy and with different
versions of the dependency, but it remained the same error message.
But the RecommenderJob is meant to be to run a pseudo distributed
recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
have another possibility to run recommendations on a Hadoop Cluster? I have
read that only the clustering and classification parts of mahout are really
able to be distributed on a hadoop cluster.
2012/3/15 Sean Owen <sr...@gmail.com>
> You shouldn't have to add anything to your jar, if you use the
> supplied 'job' file which contains all transitive dependencies.
> If you do add your own jars, I think you need to unpack and repack
> them, not put them into the overall jar as a jar file, even with a
> MANIFEST.MF entry. I am not sure that works on Hadoop.
>
> On Thu, Mar 15, 2012 at 9:42 AM, Janina <ma...@googlemail.com>
> wrote:
> > Hi all,
> >
> > I am trying to run a RecommenderJob from a Java program. I have added the
> > files users.txt and input.txt to a Hadoop VM and use the run-method of
> > RecommenderJob to start the calculation. But the the following error
> > message occurs while running the MapReducer:
> >
> > Error: java.lang.ClassNotFoundException:
> com.google.common.primitives.Longs
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at
> >
> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
> > at
> >
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
> > at
> >
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
> > I have added the required guava-r09.jar explicitly to my jar which also
> > lays on the hadoop cluster.
> > This may be a stupid question, but does anyone know where this error
> comes
> > from? This would help me a lot.
> >
> > Thanks and greetings,
> > Janina
>
Re: ClassNotFoundException while using RecommenderJob
Posted by Sean Owen <sr...@gmail.com>.
You shouldn't have to add anything to your jar, if you use the
supplied 'job' file which contains all transitive dependencies.
If you do add your own jars, I think you need to unpack and repack
them, not put them into the overall jar as a jar file, even with a
MANIFEST.MF entry. I am not sure that works on Hadoop.
On Thu, Mar 15, 2012 at 9:42 AM, Janina <ma...@googlemail.com> wrote:
> Hi all,
>
> I am trying to run a RecommenderJob from a Java program. I have added the
> files users.txt and input.txt to a Hadoop VM and use the run-method of
> RecommenderJob to start the calculation. But the the following error
> message occurs while running the MapReducer:
>
> Error: java.lang.ClassNotFoundException: com.google.common.primitives.Longs
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at
> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
> at
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
> at
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> I have added the required guava-r09.jar explicitly to my jar which also
> lays on the hadoop cluster.
> This may be a stupid question, but does anyone know where this error comes
> from? This would help me a lot.
>
> Thanks and greetings,
> Janina