You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Simon Reavely <si...@gmail.com> on 2010/08/09 17:16:24 UTC
Trouble running RecommenderJob with Mahout 0.3 - class not found issues
#!/bin/sh
# mapred.input.dir = file/dir containing the history to process
# usersFile = user ids to produce recommendations for
# This will run a co-occurrence algorithm on it
hadoop jar mahout-0.3/mahout-core-0.3.job
org.apache.mahout.cf.taste.hadoop.item.RecommenderJob --input
/user/sreavely/mahout-boolean-enduseraction-input.csv --output
/user/sreavely/mahout-output --usersFile
/user/sreavely/mahout-users-to-recommend-for.txt
This is what i get...
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
... 12 more
Caused by: java.lang.ClassNotFoundException:
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 13 more
I was assuming that the .job file (being a jar file) would have all this
stuff in it. Any thoughts what is going wrong?
Cheers,
Simon
--
Simon Reavely
simon.reavely@gmail.com
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found issues
Posted by Drew Farris <dr...@apache.org>.
On Mon, Aug 9, 2010 at 4:14 PM, Simon Reavely <si...@gmail.com> wrote:
> Please note, i suspect that this might be an issue with how I
> hacked together my package since I can't figure out how to create a proper
> binary release from src.
I'm not familiar with the taste code, but as far as creating a build
from sources:
svn co https://svn.apache.org/repos/asf/mahout/trunk
cd trunk
mvn clean install -Pfastinstall
Your can use the job files in
core/target/mahout-core-0.4-SNAPSHOT.job, or in
examples/target/mahout-examples-0.4-SNAPSHOT.job
(-Pfastinstall skips the unit tests, checkstyle and other reports)
hth,
Drew
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found
issues
Posted by Simon Reavely <si...@gmail.com>.
This same dataset works fine outside of hadoop, with the non-hadoop versions
of logliklihood, slopeone, etc
However, it could be that this code is more sensitive to issues.
When I get a few minutes I think I'm gonna need to look more closely at
these lines:
String[] tokens = TasteHadoopUtils.splitPrefTokens(value.toString());
long itemID = Long.parseLong(tokens[transpose ? 0 : 1]);
...and how data without preferences works
I'll also run on my data with preference values.
Stay tuned!!
S.
On Tue, Aug 10, 2010 at 10:48 AM, Sean Owen <sr...@gmail.com> wrote:
> You sure you don't have a blank line or something in there somewhere?
>
> On Tue, Aug 10, 2010 at 8:45 AM, Simon Reavely <si...@gmail.com>
> wrote:
> > My input is csv, of the form
> > userid, itemid
> >
>
--
Simon Reavely
simon.reavely@gmail.com
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found
issues
Posted by Sebastian Schelter <ss...@googlemail.com>.
Try removing the whitespace after the comma.
Am 10.08.2010 16:48, schrieb Sean Owen:
> You sure you don't have a blank line or something in there somewhere?
>
> On Tue, Aug 10, 2010 at 8:45 AM, Simon Reavely <si...@gmail.com> wrote:
>
>> My input is csv, of the form
>> userid, itemid
>>
>>
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found issues
Posted by Sean Owen <sr...@gmail.com>.
You sure you don't have a blank line or something in there somewhere?
On Tue, Aug 10, 2010 at 8:45 AM, Simon Reavely <si...@gmail.com> wrote:
> My input is csv, of the form
> userid, itemid
>
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found
issues
Posted by Simon Reavely <si...@gmail.com>.
My input is csv, of the form
userid, itemid
I set booleanData=true
So my script looks like this:
#!/bin/bash
# --input = hdfs file/dir containing the history to process
# --output = hdfs directory to put output into
# --usersFile = user ids to produce recommendations for
# This will run a co-occurrence algorithm on it
mahoutdir=/home/sreavely/mahout-0.4
mahoutver=0.4-SNAPSHOT
hadoop jar $mahoutdir/mahout-core-$mahoutver.job
org.apache.mahout.cf.taste.hadoop.item.RecommenderJob --input
/user/sreavely/mahout-boolean-enduseraction-input.csv --output
/user/sreavely/mahout-output --usersFile
/user/sreavely/mahout-users-to-recommend-for.txt --booleanData true
Cheers,
Simon
p.s. I also have a dataset with a preference column that I've not tested
with yet.
On Tue, Aug 10, 2010 at 1:38 AM, Sean Owen <sr...@gmail.com> wrote:
> I think your input is malformed, what does it look like?
> (But the error could be better.)
>
> On Mon, Aug 9, 2010 at 3:14 PM, Simon Reavely <si...@gmail.com>
> wrote:
> > I built and hacked together 0.4-snapshot from src
> >
> > It now finds the class files - hurrah!
> > However, I now get an ArrayIndexOutOfBoundsException
> >
> >
> > 10/08/09 16:07:14 INFO mapred.JobClient: Task Id :
> > attempt_201005101218_0012_m_000000_2, Status : FAILED
> > java.lang.ArrayIndexOutOfBoundsException: 1
> > at
> >
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:47)
> > at
> >
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
> >
> > Looking at the source code, the issue is on the array indexing on tokens
> > below, which seems to be an issue
> > with: TasteHadoopUtils.splitPrefTokens(value.toString());
> >
> > @Override
> > protected void map(LongWritable key,
> > Text value,
> > Context context) throws IOException,
> > InterruptedException {
> > String[] tokens = TasteHadoopUtils.splitPrefTokens(value.toString());
> > long itemID = Long.parseLong(tokens[transpose ? 0 : 1]);
> > int index = TasteHadoopUtils.idToIndex(itemID);
> > context.write(new VarIntWritable(index), new VarLongWritable(itemID));
> > }
> >
> > Any ideas? Please note, i suspect that this might be an issue with how I
> > hacked together my package since I can't figure out how to create a
> proper
> > binary release from src.
> >
> > If not, I'm off to the debugger!
> >
> > Cheers,
> > Simon
>
--
Simon Reavely
simon.reavely@gmail.com
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found
issues
Posted by Sean Owen <sr...@gmail.com>.
I think your input is malformed, what does it look like?
(But the error could be better.)
On Mon, Aug 9, 2010 at 3:14 PM, Simon Reavely <si...@gmail.com> wrote:
> I built and hacked together 0.4-snapshot from src
>
> It now finds the class files - hurrah!
> However, I now get an ArrayIndexOutOfBoundsException
>
>
> 10/08/09 16:07:14 INFO mapred.JobClient: Task Id :
> attempt_201005101218_0012_m_000000_2, Status : FAILED
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:47)
> at
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> Looking at the source code, the issue is on the array indexing on tokens
> below, which seems to be an issue
> with: TasteHadoopUtils.splitPrefTokens(value.toString());
>
> @Override
> protected void map(LongWritable key,
> Text value,
> Context context) throws IOException,
> InterruptedException {
> String[] tokens = TasteHadoopUtils.splitPrefTokens(value.toString());
> long itemID = Long.parseLong(tokens[transpose ? 0 : 1]);
> int index = TasteHadoopUtils.idToIndex(itemID);
> context.write(new VarIntWritable(index), new VarLongWritable(itemID));
> }
>
> Any ideas? Please note, i suspect that this might be an issue with how I
> hacked together my package since I can't figure out how to create a proper
> binary release from src.
>
> If not, I'm off to the debugger!
>
> Cheers,
> Simon
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found
issues
Posted by Simon Reavely <si...@gmail.com>.
I built and hacked together 0.4-snapshot from src
It now finds the class files - hurrah!
However, I now get an ArrayIndexOutOfBoundsException
10/08/09 16:07:14 INFO mapred.JobClient: Task Id :
attempt_201005101218_0012_m_000000_2, Status : FAILED
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:47)
at
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Looking at the source code, the issue is on the array indexing on tokens
below, which seems to be an issue
with: TasteHadoopUtils.splitPrefTokens(value.toString());
@Override
protected void map(LongWritable key,
Text value,
Context context) throws IOException,
InterruptedException {
String[] tokens = TasteHadoopUtils.splitPrefTokens(value.toString());
long itemID = Long.parseLong(tokens[transpose ? 0 : 1]);
int index = TasteHadoopUtils.idToIndex(itemID);
context.write(new VarIntWritable(index), new VarLongWritable(itemID));
}
Any ideas? Please note, i suspect that this might be an issue with how I
hacked together my package since I can't figure out how to create a proper
binary release from src.
If not, I'm off to the debugger!
Cheers,
Simon
On Mon, Aug 9, 2010 at 11:19 AM, Sean Owen <sr...@gmail.com> wrote:
> I don't really get it either -- the .job file ought to have
> everything. That class isn't even in a separate module.
> But I can tell you the next good step is to use the latest code from
> Subversion instead of 0.3, since I 99% know that works.
>
> On Mon, Aug 9, 2010 at 10:16 AM, Simon Reavely <si...@gmail.com>
> wrote:
> > #!/bin/sh
> > # mapred.input.dir = file/dir containing the history to process
> > # usersFile = user ids to produce recommendations for
> > # This will run a co-occurrence algorithm on it
> > hadoop jar mahout-0.3/mahout-core-0.3.job
> > org.apache.mahout.cf.taste.hadoop.item.RecommenderJob --input
> > /user/sreavely/mahout-boolean-enduseraction-input.csv --output
> > /user/sreavely/mahout-output --usersFile
> > /user/sreavely/mahout-users-to-recommend-for.txt
> >
> > This is what i get...
> >
> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
> > ... 12 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> > ... 13 more
> >
> > I was assuming that the .job file (being a jar file) would have all this
> > stuff in it. Any thoughts what is going wrong?
> >
> > Cheers,
> > Simon
> >
> > --
> > Simon Reavely
> > simon.reavely@gmail.com
> >
>
--
Simon Reavely
simon.reavely@gmail.com
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found
issues
Posted by Simon Reavely <si...@gmail.com>.
Where do I get mahout-0.4-snapshot from?
I can't see it in tags, branches, etc http://svn.apache.org/repos/asf/mahout
Do I just pull down trunk...seems a bit bleeding edge??
On Mon, Aug 9, 2010 at 11:19 AM, Sean Owen <sr...@gmail.com> wrote:
> I don't really get it either -- the .job file ought to have
> everything. That class isn't even in a separate module.
> But I can tell you the next good step is to use the latest code from
> Subversion instead of 0.3, since I 99% know that works.
>
> On Mon, Aug 9, 2010 at 10:16 AM, Simon Reavely <si...@gmail.com>
> wrote:
> > #!/bin/sh
> > # mapred.input.dir = file/dir containing the history to process
> > # usersFile = user ids to produce recommendations for
> > # This will run a co-occurrence algorithm on it
> > hadoop jar mahout-0.3/mahout-core-0.3.job
> > org.apache.mahout.cf.taste.hadoop.item.RecommenderJob --input
> > /user/sreavely/mahout-boolean-enduseraction-input.csv --output
> > /user/sreavely/mahout-output --usersFile
> > /user/sreavely/mahout-users-to-recommend-for.txt
> >
> > This is what i get...
> >
> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
> > ... 12 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> > ... 13 more
> >
> > I was assuming that the .job file (being a jar file) would have all this
> > stuff in it. Any thoughts what is going wrong?
> >
> > Cheers,
> > Simon
> >
> > --
> > Simon Reavely
> > simon.reavely@gmail.com
> >
>
--
Simon Reavely
simon.reavely@gmail.com
Re: Trouble running RecommenderJob with Mahout 0.3 - class not found
issues
Posted by Sean Owen <sr...@gmail.com>.
I don't really get it either -- the .job file ought to have
everything. That class isn't even in a separate module.
But I can tell you the next good step is to use the latest code from
Subversion instead of 0.3, since I 99% know that works.
On Mon, Aug 9, 2010 at 10:16 AM, Simon Reavely <si...@gmail.com> wrote:
> #!/bin/sh
> # mapred.input.dir = file/dir containing the history to process
> # usersFile = user ids to produce recommendations for
> # This will run a co-occurrence algorithm on it
> hadoop jar mahout-0.3/mahout-core-0.3.job
> org.apache.mahout.cf.taste.hadoop.item.RecommenderJob --input
> /user/sreavely/mahout-boolean-enduseraction-input.csv --output
> /user/sreavely/mahout-output --usersFile
> /user/sreavely/mahout-users-to-recommend-for.txt
>
> This is what i get...
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
> ... 12 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> ... 13 more
>
> I was assuming that the .job file (being a jar file) would have all this
> stuff in it. Any thoughts what is going wrong?
>
> Cheers,
> Simon
>
> --
> Simon Reavely
> simon.reavely@gmail.com
>