You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by jamborta <ja...@gmail.com> on 2009/10/15 17:11:49 UTC
Running mahout on hadoop
hi guys,
i've been using your library for a while for my phd project. we just decided
that we're going to run all the projects on hadoop, which is available in
your library. but I couldn't figure out how to run a simple recommender on
hadoop which is in org.apache.mahout.cf.taste.hadoop
I'd be glad if someone could point me towards the right direction with this.
thanks a lot
tamas
--
View this message in context: http://www.nabble.com/Running-mahout-on-hadoop-tp25910380p25910380.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Running mahout on hadoop
Posted by Ted Dunning <te...@gmail.com>.
+1 for separate but integrated.
The output of the off-line computations is usable for non-Taste applications
as well and Taste should be open enough to accept other kinds of off-line
input.
On Thu, Oct 15, 2009 at 1:21 PM, Sean Owen <sr...@gmail.com> wrote:
> I think it must be a separate implementation. Both goals are important but
> can't be achieved with one implementation.
>
--
Ted Dunning, CTO
DeepDyve
Re: Running mahout on hadoop
Posted by Sean Owen <sr...@gmail.com>.
I think it must be a separate implementation. Both goals are important but
can't be achieved with one implementation.
On Oct 15, 2009 11:38 AM, "Gökhan Çapan" <gk...@gmail.com> wrote:
Ok, I also wonder what you think about the type of that implementation.
-Will it be an extesion to Taste? I mean Taste's current implementation will
remain same, also that will be added as an extra option.
-Or do you plan to migrate Taste to that style?
I am asking because I am also working on implementing that model creation
with known similarity metrics, like log-likelihood similarity.
On Thu, Oct 15, 2009 at 9:29 PM, Sean Owen <sr...@gmail.com> wrote: >
Agree, and no that does not...
--
Gökhan Çapan
Re: Running mahout on hadoop
Posted by Gökhan Çapan <gk...@gmail.com>.
Ok, I also wonder what you think about the type of that implementation.
-Will it be an extesion to Taste? I mean Taste's current implementation will
remain same, also that will be added as an extra option.
-Or do you plan to migrate Taste to that style?
I am asking because I am also working on implementing that model creation
with known similarity metrics, like log-likelihood similarity.
On Thu, Oct 15, 2009 at 9:29 PM, Sean Owen <sr...@gmail.com> wrote:
> Agree, and no that does not exist yet. I suppose I have temporarily paused
> thinking about it until 0.20.x sorts out these issues.
>
> On Oct 15, 2009 11:21 AM, "Gökhan Çapan" <gk...@gmail.com> wrote:
>
> Sorry, i've sent last post accidently.
> The question was not about which version of hadoop it uses.
> Let me explain,
> I think we all agree that item based recommendation is a 2 step process.
> 1-compute item-item similarities
> 2-create recommendations to a specific user for an item
>
> And, 1st step can be computed offline.
> If we want to compute all item-item similarities, it would be very good if
> that computation had a map-reduce implementation.
>
> I remember we had a discussion about making this computation with a matrix
> based approach and its potential of being distributable. I think that will
> be a very good extension for taste.
>
> On Thu, Oct 15, 2009 at 9:12 PM, Gökhan Çapan <gk...@gmail.com> wrote: >
> I
> mean, > > On Thu, Oct...
> --
> Gökhan Çapan
>
--
Gökhan Çapan
Re: Running mahout on hadoop
Posted by Sean Owen <sr...@gmail.com>.
Agree, and no that does not exist yet. I suppose I have temporarily paused
thinking about it until 0.20.x sorts out these issues.
On Oct 15, 2009 11:21 AM, "Gökhan Çapan" <gk...@gmail.com> wrote:
Sorry, i've sent last post accidently.
The question was not about which version of hadoop it uses.
Let me explain,
I think we all agree that item based recommendation is a 2 step process.
1-compute item-item similarities
2-create recommendations to a specific user for an item
And, 1st step can be computed offline.
If we want to compute all item-item similarities, it would be very good if
that computation had a map-reduce implementation.
I remember we had a discussion about making this computation with a matrix
based approach and its potential of being distributable. I think that will
be a very good extension for taste.
On Thu, Oct 15, 2009 at 9:12 PM, Gökhan Çapan <gk...@gmail.com> wrote: > I
mean, > > On Thu, Oct...
--
Gökhan Çapan
Re: Running mahout on hadoop
Posted by Gökhan Çapan <gk...@gmail.com>.
Sorry, i've sent last post accidently.
The question was not about which version of hadoop it uses.
Let me explain,
I think we all agree that item based recommendation is a 2 step process.
1-compute item-item similarities
2-create recommendations to a specific user for an item
And, 1st step can be computed offline.
If we want to compute all item-item similarities, it would be very good if
that computation had a map-reduce implementation.
I remember we had a discussion about making this computation with a matrix
based approach and its potential of being distributable. I think that will
be a very good extension for taste.
On Thu, Oct 15, 2009 at 9:12 PM, Gökhan Çapan <gk...@gmail.com> wrote:
> I mean,
>
> On Thu, Oct 15, 2009 at 9:08 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> Meaning using the 0.19 API? It did and I updated it but have hit the
>> problem
>> I mention. I am reluctant to move backwards if a fix is coming though I am
>> increasingly concerned about this.
>>
>> On Oct 15, 2009 10:52 AM, "Gökhan Çapan" <gk...@gmail.com> wrote:
>>
>> I also want to ask something about topic. Does Taste have a mapred
>> implementation?
>>
>> On Thu, Oct 15, 2009 at 8:38 PM, Sean Owen <sr...@gmail.com> wrote: >
>> Yeah
>> you see the same probl...
>> --
>> Gökhan Çapan
>>
>
>
>
> --
> Gökhan Çapan
>
--
Gökhan Çapan
Re: Running mahout on hadoop
Posted by Gökhan Çapan <gk...@gmail.com>.
I mean,
On Thu, Oct 15, 2009 at 9:08 PM, Sean Owen <sr...@gmail.com> wrote:
> Meaning using the 0.19 API? It did and I updated it but have hit the
> problem
> I mention. I am reluctant to move backwards if a fix is coming though I am
> increasingly concerned about this.
>
> On Oct 15, 2009 10:52 AM, "Gökhan Çapan" <gk...@gmail.com> wrote:
>
> I also want to ask something about topic. Does Taste have a mapred
> implementation?
>
> On Thu, Oct 15, 2009 at 8:38 PM, Sean Owen <sr...@gmail.com> wrote: >
> Yeah
> you see the same probl...
> --
> Gökhan Çapan
>
--
Gökhan Çapan
Re: Running mahout on hadoop
Posted by Sean Owen <sr...@gmail.com>.
Meaning using the 0.19 API? It did and I updated it but have hit the problem
I mention. I am reluctant to move backwards if a fix is coming though I am
increasingly concerned about this.
On Oct 15, 2009 10:52 AM, "Gökhan Çapan" <gk...@gmail.com> wrote:
I also want to ask something about topic. Does Taste have a mapred
implementation?
On Thu, Oct 15, 2009 at 8:38 PM, Sean Owen <sr...@gmail.com> wrote: > Yeah
you see the same probl...
--
Gökhan Çapan
Re: Running mahout on hadoop
Posted by Gökhan Çapan <gk...@gmail.com>.
I also want to ask something about topic. Does Taste have a mapred
implementation?
On Thu, Oct 15, 2009 at 8:38 PM, Sean Owen <sr...@gmail.com> wrote:
> Yeah you see the same problem I do. If you use the Hadoop 0.20 API here it
> fails because something in Hadoop expects 0.19 APIs. Seems like a bug and I
> am waiting on a fix.
> The other error sound like a Hadoop version mismatch.
>
> It remains required to pass in a list of user ID to recommend for as input,
> yes.
>
> On Oct 15, 2009 10:27 AM, "jamborta" <ja...@gmail.com> wrote:
>
>
> thanks. I tried to run it. I have two places to run it. my desktop where I
> installed hadoop. it gives me this error:
>
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: class
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat not
> org.apache.hadoop.mapred.OutputFormat
>
> and we have a server running where it gives me another error (I guess this
> one is becuase this hadoop is an older version 0.18):
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.setBooleanIfUnset(Ljava/lang/String;Z)V
> at org.apache.hadoop.mapreduce.Job.setUseNewAPI(Job.java:388)
>
> this is what I used to run it:
>
> hadoop-0.20.1/bin/hadoop jar data/mahout-examples-0.2-SNAPSHOT.job
> org.apache.mahout.cf.taste.hadoop.RecommenderJob -r
> org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender -n
> 10 -m data/data/all_data.data -f data/data/all_data.data -o /output
>
> I'm not sure about what I should put for userIdFile because userids are
> already in the dataModelFile, at least that's how your input worked before.
>
> thanks a lot for your help,
>
> Tamas
>
>
>
> --
> View this message in context:
> http://www.nabble.com/Running-mahout-on-hadoop-tp25910380p25912732.html
>
> Sent from the Mahout User List mailing list archive at Nabble.com.
>
--
Gökhan Çapan
Re: Running mahout on hadoop
Posted by Sean Owen <sr...@gmail.com>.
Oh, gawd how did I not put that together... SVD got me thinking of
algorithms not infrastructure. No the thing I committed today does not
resolve the problem. It's a Hadoop issue, I'm sure.
On Tue, Oct 20, 2009 at 8:06 PM, Ted Dunning <te...@gmail.com> wrote:
> SVN?
>
> On Tue, Oct 20, 2009 at 11:57 AM, Sean Owen <sr...@gmail.com> wrote:
>
>> What do you mean by svd? not sure what you are referring to.
Re: Running mahout on hadoop
Posted by jamborta <ja...@gmail.com>.
yes. sorry. svn.
Ted Dunning wrote:
>
> SVN?
>
> On Tue, Oct 20, 2009 at 11:57 AM, Sean Owen <sr...@gmail.com> wrote:
>
>> What do you mean by svd? not sure what you are referring to.
>>
>> On Tue, Oct 20, 2009 at 7:47 PM, jamborta <ja...@gmail.com> wrote:
>> >
>> > thanks. I saw there is a new version on svd. did you manage to solve
>> the
>> > problem?
>> >
>> >
>> >
>> > srowen wrote:
>> >>
>> >> Yeah you see the same problem I do. If you use the Hadoop 0.20 API
>> here
>> it
>> >> fails because something in Hadoop expects 0.19 APIs. Seems like a bug
>> and
>> >> I
>> >> am waiting on a fix.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
>
--
View this message in context: http://www.nabble.com/Running-mahout-on-hadoop-tp25910380p25981692.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Running mahout on hadoop
Posted by Ted Dunning <te...@gmail.com>.
SVN?
On Tue, Oct 20, 2009 at 11:57 AM, Sean Owen <sr...@gmail.com> wrote:
> What do you mean by svd? not sure what you are referring to.
>
> On Tue, Oct 20, 2009 at 7:47 PM, jamborta <ja...@gmail.com> wrote:
> >
> > thanks. I saw there is a new version on svd. did you manage to solve the
> > problem?
> >
> >
> >
> > srowen wrote:
> >>
> >> Yeah you see the same problem I do. If you use the Hadoop 0.20 API here
> it
> >> fails because something in Hadoop expects 0.19 APIs. Seems like a bug
> and
> >> I
> >> am waiting on a fix.
>
--
Ted Dunning, CTO
DeepDyve
Re: Running mahout on hadoop
Posted by Sean Owen <sr...@gmail.com>.
What do you mean by svd? not sure what you are referring to.
On Tue, Oct 20, 2009 at 7:47 PM, jamborta <ja...@gmail.com> wrote:
>
> thanks. I saw there is a new version on svd. did you manage to solve the
> problem?
>
>
>
> srowen wrote:
>>
>> Yeah you see the same problem I do. If you use the Hadoop 0.20 API here it
>> fails because something in Hadoop expects 0.19 APIs. Seems like a bug and
>> I
>> am waiting on a fix.
Re: Running mahout on hadoop
Posted by jamborta <ja...@gmail.com>.
thanks. I saw there is a new version on svd. did you manage to solve the
problem?
srowen wrote:
>
> Yeah you see the same problem I do. If you use the Hadoop 0.20 API here it
> fails because something in Hadoop expects 0.19 APIs. Seems like a bug and
> I
> am waiting on a fix.
> The other error sound like a Hadoop version mismatch.
>
> It remains required to pass in a list of user ID to recommend for as
> input,
> yes.
>
> On Oct 15, 2009 10:27 AM, "jamborta" <ja...@gmail.com> wrote:
>
>
> thanks. I tried to run it. I have two places to run it. my desktop where I
> installed hadoop. it gives me this error:
>
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: class
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat not
> org.apache.hadoop.mapred.OutputFormat
>
> and we have a server running where it gives me another error (I guess this
> one is becuase this hadoop is an older version 0.18):
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.setBooleanIfUnset(Ljava/lang/String;Z)V
> at org.apache.hadoop.mapreduce.Job.setUseNewAPI(Job.java:388)
>
> this is what I used to run it:
>
> hadoop-0.20.1/bin/hadoop jar data/mahout-examples-0.2-SNAPSHOT.job
> org.apache.mahout.cf.taste.hadoop.RecommenderJob -r
> org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender
> -n
> 10 -m data/data/all_data.data -f data/data/all_data.data -o /output
>
> I'm not sure about what I should put for userIdFile because userids are
> already in the dataModelFile, at least that's how your input worked
> before.
>
> thanks a lot for your help,
>
> Tamas
>
>
>
> --
> View this message in context:
> http://www.nabble.com/Running-mahout-on-hadoop-tp25910380p25912732.html
>
> Sent from the Mahout User List mailing list archive at Nabble.com.
>
>
--
View this message in context: http://www.nabble.com/Running-mahout-on-hadoop-tp25910380p25980411.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Running mahout on hadoop
Posted by Sean Owen <sr...@gmail.com>.
Yeah you see the same problem I do. If you use the Hadoop 0.20 API here it
fails because something in Hadoop expects 0.19 APIs. Seems like a bug and I
am waiting on a fix.
The other error sound like a Hadoop version mismatch.
It remains required to pass in a list of user ID to recommend for as input,
yes.
On Oct 15, 2009 10:27 AM, "jamborta" <ja...@gmail.com> wrote:
thanks. I tried to run it. I have two places to run it. my desktop where I
installed hadoop. it gives me this error:
Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: class
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat not
org.apache.hadoop.mapred.OutputFormat
and we have a server running where it gives me another error (I guess this
one is becuase this hadoop is an older version 0.18):
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.setBooleanIfUnset(Ljava/lang/String;Z)V
at org.apache.hadoop.mapreduce.Job.setUseNewAPI(Job.java:388)
this is what I used to run it:
hadoop-0.20.1/bin/hadoop jar data/mahout-examples-0.2-SNAPSHOT.job
org.apache.mahout.cf.taste.hadoop.RecommenderJob -r
org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender -n
10 -m data/data/all_data.data -f data/data/all_data.data -o /output
I'm not sure about what I should put for userIdFile because userids are
already in the dataModelFile, at least that's how your input worked before.
thanks a lot for your help,
Tamas
--
View this message in context:
http://www.nabble.com/Running-mahout-on-hadoop-tp25910380p25912732.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Running mahout on hadoop
Posted by jamborta <ja...@gmail.com>.
thanks. I tried to run it. I have two places to run it. my desktop where I
installed hadoop. it gives me this error:
Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: class
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat not
org.apache.hadoop.mapred.OutputFormat
and we have a server running where it gives me another error (I guess this
one is becuase this hadoop is an older version 0.18):
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.setBooleanIfUnset(Ljava/lang/String;Z)V
at org.apache.hadoop.mapreduce.Job.setUseNewAPI(Job.java:388)
this is what I used to run it:
hadoop-0.20.1/bin/hadoop jar data/mahout-examples-0.2-SNAPSHOT.job
org.apache.mahout.cf.taste.hadoop.RecommenderJob -r
org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender -n
10 -m data/data/all_data.data -f data/data/all_data.data -o /output
I'm not sure about what I should put for userIdFile because userids are
already in the dataModelFile, at least that's how your input worked before.
thanks a lot for your help,
Tamas
--
View this message in context: http://www.nabble.com/Running-mahout-on-hadoop-tp25910380p25912732.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Running mahout on hadoop
Posted by Sean Owen <sr...@gmail.com>.
Yes could you be more specific at all? not sure what the issue you are
facing is.
On Thu, Oct 15, 2009 at 8:37 AM, Isabel Drost <is...@apache.org> wrote:
>
> Hello jamborta,
>
> On Thu jamborta <ja...@gmail.com> wrote:
>
>> org.apache.mahout.cf.taste.hadoop I'd be glad if someone could point
>> me towards the right direction with this.
>
> There just so happens to be some documentation on how to start the
> taste tooling on a hadoop cluster on the wiki:
>
> http://cwiki.apache.org/MAHOUT/tastecommandline.html
>
> I am sure, Sean can add far more information to that.
>
> Cheers,
> Isabel
>
Re: Running mahout on hadoop
Posted by Isabel Drost <is...@apache.org>.
Hello jamborta,
On Thu jamborta <ja...@gmail.com> wrote:
> org.apache.mahout.cf.taste.hadoop I'd be glad if someone could point
> me towards the right direction with this.
There just so happens to be some documentation on how to start the
taste tooling on a hadoop cluster on the wiki:
http://cwiki.apache.org/MAHOUT/tastecommandline.html
I am sure, Sean can add far more information to that.
Cheers,
Isabel