You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Sean Owen <sr...@gmail.com> on 2011/03/21 01:03:31 UTC

First results from (non-distributed) Apache Mahout / Yahoo KDD Cup

I've been test-driving a simple application of Mahout recommenders
(the non-distributed kind) on Amazon EC2 on the new Yahoo KDD Cup data
set (kddcup.yahoo.com).

In the spirit of open-source, like I mentioned, I'm committing the
extra code to mahout-examples that can be used to run a Recommender on
the input and output the right format. And, I'd like to publish the
rough timings too. Find all the source in
org.apache.mahout.cf.taste.example.kddcup


Track 1

m2.2xlarge instance, 34.2GB RAM / 4 cores
Steady state memory consumption: ~19GB
Computation time: 30 hours (wall clock-time)
CPU time per user: ~0.43 sec
Cost on EC2: $34.20 (!)

(Helpful hint on cost I realized after the fact: you can almost surely
get spot instances for cheaper. The maximum price this sort of
instance has gone for as a spot instance is about $0.60/hour, vs
"retail price" of $1.14/hour.)

Resulted in an RMSE of 29.5618 (the rating scale is 0-100), which is
only good enough for 29th place at the moment. Not terrible for "out
of the box" performance -- it's just using an item-based recommender
with uncentered cosine similarity. But not really good in absolute
terms. A winning solution is going to try to factor in time, and apply
more sophisticated techniques. The best RMSE so far is about 23.


Track 2

c1.xlarge instance: 7GB RAM / 8 cores
Steady state memory consumption: ~3.8GB
Computation time: 4.1 hours (wall clock-time)
CPU time per user: ~1.1 sec
Cost on EC2: $3.20

For this I bothered to write a simplistic item-item similarity metric
to take into account the additional info that is available: track,
artist, album, genre. The result was comparatively better: 17.92%
error rate, good enough for 4th place at the moment.


Of course, the next task is to put this through the actual distributed
processing -- that's really the appropriate solution.

This shows you can still tackle fairly impressive scale with a
non-distributed solution. These results suggest that the largest
instances available from EC2 would accomodate almost 1 billion ratings
in memory. However at that scale running a user's full recommendations
would easily be measured in seconds, not milliseconds.

Re: First results from (non-distributed) Apache Mahout / Yahoo KDD Cup

Posted by Sean Owen <sr...@gmail.com>.
The AMI is jus the standard Amazon AMI for 64-bit Linux. It's whatever
the default option is when you launch an instance via the console.

For this test I was trying a simple non-distributed version. The
non-distributed SVD won't quite fit in memory at this scale. You can
see the exact code in examples/, under
org.apache.mahout.cf.taste.example.kddcup.

Yes you need different recommenders. For example one track has ratings
and the other doesn't. More than that, neither track is about testing
recommendations per se. So most of it is about implementing the
particular test from each track and outputting the right format.

On Mon, Mar 21, 2011 at 12:38 AM, Danny Bickson <da...@gmail.com> wrote:
> Hi Sean!
> Thanks for sharing - this sounds very interesting!
> 1) Would you mind sharing an Amazon EC2 AMI image so I can try to reproduce
> your exact setup?
> I may be willing to help you setup the distributed code.
> 2) What recommender are you using - it is based on matrix factorization
> using SVD?
> 3) Are you using the same algorithm for track 1 and track 2? It seems that
> the task are slightly different.
>

Re: First results from (non-distributed) Apache Mahout / Yahoo KDD Cup

Posted by Sean Owen <sr...@gmail.com>.
I was using all 4 / 8 cores 100% of the time, of course. The internal
computations from one request are not multi-threaded, but, the
assumption is that the caller is taking advantage of parallelism by
parallelizing at the request level. This gives more throughput. That
is to say, it's up to the up-stream caller.

And internally, one such caller exists -- the evaluators. They use
multiple threads too. Though they are not used in the KDD Cup code.

I don't know of any case where the synchronization you mention causes
significant contention. The blocks being synchronized are tiny, just a
few math operations and field read/writes. The cost of acquiring locks
is less than microseconds. And they turn up in places where there
ought not be many threads at once (e.g. slope-one's diffs -- two
threads should rarely touch the same diff)

So, there was evidently no contention problem, and I can't imagine any
acquiring of locks was significant overhead. To the extent it was,
well, it's necessary for program correctness.

On Mon, Mar 21, 2011 at 5:50 AM, Lance Norskog <go...@gmail.com> wrote:
> About the costs:
>
> Many of the Recommenders use the RunningAverage classes. These are
> synchronous. Given that all of the single-machine jobs are
> single-threaded, doesn't this thread-checking burn more than a few
> pennies?
>
> On Sun, Mar 20, 2011 at 5:38 PM, Danny Bickson <da...@gmail.com> wrote:
>> Hi Sean!
>> Thanks for sharing - this sounds very interesting!
>> 1) Would you mind sharing an Amazon EC2 AMI image so I can try to reproduce
>> your exact setup?
>> I may be willing to help you setup the distributed code.
>> 2) What recommender are you using - it is based on matrix factorization
>> using SVD?
>> 3) Are you using the same algorithm for track 1 and track 2? It seems that
>> the task are slightly different.
>>
>> Thanks!
>>
>> - Danny Bickson
>>
>> On Sun, Mar 20, 2011 at 8:03 PM, Sean Owen <sr...@gmail.com> wrote:
>>
>>> I've been test-driving a simple application of Mahout recommenders
>>> (the non-distributed kind) on Amazon EC2 on the new Yahoo KDD Cup data
>>> set (kddcup.yahoo.com).
>>>
>>> In the spirit of open-source, like I mentioned, I'm committing the
>>> extra code to mahout-examples that can be used to run a Recommender on
>>> the input and output the right format. And, I'd like to publish the
>>> rough timings too. Find all the source in
>>> org.apache.mahout.cf.taste.example.kddcup
>>>
>>>
>>> Track 1
>>>
>>> m2.2xlarge instance, 34.2GB RAM / 4 cores
>>> Steady state memory consumption: ~19GB
>>> Computation time: 30 hours (wall clock-time)
>>> CPU time per user: ~0.43 sec
>>> Cost on EC2: $34.20 (!)
>>>
>>> (Helpful hint on cost I realized after the fact: you can almost surely
>>> get spot instances for cheaper. The maximum price this sort of
>>> instance has gone for as a spot instance is about $0.60/hour, vs
>>> "retail price" of $1.14/hour.)
>>>
>>> Resulted in an RMSE of 29.5618 (the rating scale is 0-100), which is
>>> only good enough for 29th place at the moment. Not terrible for "out
>>> of the box" performance -- it's just using an item-based recommender
>>> with uncentered cosine similarity. But not really good in absolute
>>> terms. A winning solution is going to try to factor in time, and apply
>>> more sophisticated techniques. The best RMSE so far is about 23.
>>>
>>>
>>> Track 2
>>>
>>> c1.xlarge instance: 7GB RAM / 8 cores
>>> Steady state memory consumption: ~3.8GB
>>> Computation time: 4.1 hours (wall clock-time)
>>> CPU time per user: ~1.1 sec
>>> Cost on EC2: $3.20
>>>
>>> For this I bothered to write a simplistic item-item similarity metric
>>> to take into account the additional info that is available: track,
>>> artist, album, genre. The result was comparatively better: 17.92%
>>> error rate, good enough for 4th place at the moment.
>>>
>>>
>>> Of course, the next task is to put this through the actual distributed
>>> processing -- that's really the appropriate solution.
>>>
>>> This shows you can still tackle fairly impressive scale with a
>>> non-distributed solution. These results suggest that the largest
>>> instances available from EC2 would accomodate almost 1 billion ratings
>>> in memory. However at that scale running a user's full recommendations
>>> would easily be measured in seconds, not milliseconds.
>>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>

Re: First results from (non-distributed) Apache Mahout / Yahoo KDD Cup

Posted by Lance Norskog <go...@gmail.com>.
About the costs:

Many of the Recommenders use the RunningAverage classes. These are
synchronous. Given that all of the single-machine jobs are
single-threaded, doesn't this thread-checking burn more than a few
pennies?

On Sun, Mar 20, 2011 at 5:38 PM, Danny Bickson <da...@gmail.com> wrote:
> Hi Sean!
> Thanks for sharing - this sounds very interesting!
> 1) Would you mind sharing an Amazon EC2 AMI image so I can try to reproduce
> your exact setup?
> I may be willing to help you setup the distributed code.
> 2) What recommender are you using - it is based on matrix factorization
> using SVD?
> 3) Are you using the same algorithm for track 1 and track 2? It seems that
> the task are slightly different.
>
> Thanks!
>
> - Danny Bickson
>
> On Sun, Mar 20, 2011 at 8:03 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> I've been test-driving a simple application of Mahout recommenders
>> (the non-distributed kind) on Amazon EC2 on the new Yahoo KDD Cup data
>> set (kddcup.yahoo.com).
>>
>> In the spirit of open-source, like I mentioned, I'm committing the
>> extra code to mahout-examples that can be used to run a Recommender on
>> the input and output the right format. And, I'd like to publish the
>> rough timings too. Find all the source in
>> org.apache.mahout.cf.taste.example.kddcup
>>
>>
>> Track 1
>>
>> m2.2xlarge instance, 34.2GB RAM / 4 cores
>> Steady state memory consumption: ~19GB
>> Computation time: 30 hours (wall clock-time)
>> CPU time per user: ~0.43 sec
>> Cost on EC2: $34.20 (!)
>>
>> (Helpful hint on cost I realized after the fact: you can almost surely
>> get spot instances for cheaper. The maximum price this sort of
>> instance has gone for as a spot instance is about $0.60/hour, vs
>> "retail price" of $1.14/hour.)
>>
>> Resulted in an RMSE of 29.5618 (the rating scale is 0-100), which is
>> only good enough for 29th place at the moment. Not terrible for "out
>> of the box" performance -- it's just using an item-based recommender
>> with uncentered cosine similarity. But not really good in absolute
>> terms. A winning solution is going to try to factor in time, and apply
>> more sophisticated techniques. The best RMSE so far is about 23.
>>
>>
>> Track 2
>>
>> c1.xlarge instance: 7GB RAM / 8 cores
>> Steady state memory consumption: ~3.8GB
>> Computation time: 4.1 hours (wall clock-time)
>> CPU time per user: ~1.1 sec
>> Cost on EC2: $3.20
>>
>> For this I bothered to write a simplistic item-item similarity metric
>> to take into account the additional info that is available: track,
>> artist, album, genre. The result was comparatively better: 17.92%
>> error rate, good enough for 4th place at the moment.
>>
>>
>> Of course, the next task is to put this through the actual distributed
>> processing -- that's really the appropriate solution.
>>
>> This shows you can still tackle fairly impressive scale with a
>> non-distributed solution. These results suggest that the largest
>> instances available from EC2 would accomodate almost 1 billion ratings
>> in memory. However at that scale running a user's full recommendations
>> would easily be measured in seconds, not milliseconds.
>>
>



-- 
Lance Norskog
goksron@gmail.com

Re: First results from (non-distributed) Apache Mahout / Yahoo KDD Cup

Posted by Danny Bickson <da...@gmail.com>.
Hi Sean!
Thanks for sharing - this sounds very interesting!
1) Would you mind sharing an Amazon EC2 AMI image so I can try to reproduce
your exact setup?
I may be willing to help you setup the distributed code.
2) What recommender are you using - it is based on matrix factorization
using SVD?
3) Are you using the same algorithm for track 1 and track 2? It seems that
the task are slightly different.

Thanks!

- Danny Bickson

On Sun, Mar 20, 2011 at 8:03 PM, Sean Owen <sr...@gmail.com> wrote:

> I've been test-driving a simple application of Mahout recommenders
> (the non-distributed kind) on Amazon EC2 on the new Yahoo KDD Cup data
> set (kddcup.yahoo.com).
>
> In the spirit of open-source, like I mentioned, I'm committing the
> extra code to mahout-examples that can be used to run a Recommender on
> the input and output the right format. And, I'd like to publish the
> rough timings too. Find all the source in
> org.apache.mahout.cf.taste.example.kddcup
>
>
> Track 1
>
> m2.2xlarge instance, 34.2GB RAM / 4 cores
> Steady state memory consumption: ~19GB
> Computation time: 30 hours (wall clock-time)
> CPU time per user: ~0.43 sec
> Cost on EC2: $34.20 (!)
>
> (Helpful hint on cost I realized after the fact: you can almost surely
> get spot instances for cheaper. The maximum price this sort of
> instance has gone for as a spot instance is about $0.60/hour, vs
> "retail price" of $1.14/hour.)
>
> Resulted in an RMSE of 29.5618 (the rating scale is 0-100), which is
> only good enough for 29th place at the moment. Not terrible for "out
> of the box" performance -- it's just using an item-based recommender
> with uncentered cosine similarity. But not really good in absolute
> terms. A winning solution is going to try to factor in time, and apply
> more sophisticated techniques. The best RMSE so far is about 23.
>
>
> Track 2
>
> c1.xlarge instance: 7GB RAM / 8 cores
> Steady state memory consumption: ~3.8GB
> Computation time: 4.1 hours (wall clock-time)
> CPU time per user: ~1.1 sec
> Cost on EC2: $3.20
>
> For this I bothered to write a simplistic item-item similarity metric
> to take into account the additional info that is available: track,
> artist, album, genre. The result was comparatively better: 17.92%
> error rate, good enough for 4th place at the moment.
>
>
> Of course, the next task is to put this through the actual distributed
> processing -- that's really the appropriate solution.
>
> This shows you can still tackle fairly impressive scale with a
> non-distributed solution. These results suggest that the largest
> instances available from EC2 would accomodate almost 1 billion ratings
> in memory. However at that scale running a user's full recommendations
> would easily be measured in seconds, not milliseconds.
>

Re: First results from (non-distributed) Apache Mahout / Yahoo KDD Cup

Posted by Danny Bickson <da...@gmail.com>.
I wrote a simple Matlab script to convert KDD  cup 2011  track1 text data
into Matlab sparse matrix format.
If anyone is interested it is found here:
http://bickson.blogspot.com/2011/03/large-scale-matrix-factorization-yahoo.html

Best,

Danny Bickson

On Mon, Mar 21, 2011 at 1:34 PM, Grant Ingersoll <gs...@apache.org>wrote:

> Nice, Sean!  I love that a near out of the box implementation puts us in
> 4th place!
>
> On Mar 20, 2011, at 8:03 PM, Sean Owen wrote:
>
> > I've been test-driving a simple application of Mahout recommenders
> > (the non-distributed kind) on Amazon EC2 on the new Yahoo KDD Cup data
> > set (kddcup.yahoo.com).
> >
> > In the spirit of open-source, like I mentioned, I'm committing the
> > extra code to mahout-examples that can be used to run a Recommender on
> > the input and output the right format. And, I'd like to publish the
> > rough timings too. Find all the source in
> > org.apache.mahout.cf.taste.example.kddcup
> >
> >
> > Track 1
> >
> > m2.2xlarge instance, 34.2GB RAM / 4 cores
> > Steady state memory consumption: ~19GB
> > Computation time: 30 hours (wall clock-time)
> > CPU time per user: ~0.43 sec
> > Cost on EC2: $34.20 (!)
> >
> > (Helpful hint on cost I realized after the fact: you can almost surely
> > get spot instances for cheaper. The maximum price this sort of
> > instance has gone for as a spot instance is about $0.60/hour, vs
> > "retail price" of $1.14/hour.)
> >
> > Resulted in an RMSE of 29.5618 (the rating scale is 0-100), which is
> > only good enough for 29th place at the moment. Not terrible for "out
> > of the box" performance -- it's just using an item-based recommender
> > with uncentered cosine similarity. But not really good in absolute
> > terms. A winning solution is going to try to factor in time, and apply
> > more sophisticated techniques. The best RMSE so far is about 23.
> >
> >
> > Track 2
> >
> > c1.xlarge instance: 7GB RAM / 8 cores
> > Steady state memory consumption: ~3.8GB
> > Computation time: 4.1 hours (wall clock-time)
> > CPU time per user: ~1.1 sec
> > Cost on EC2: $3.20
> >
> > For this I bothered to write a simplistic item-item similarity metric
> > to take into account the additional info that is available: track,
> > artist, album, genre. The result was comparatively better: 17.92%
> > error rate, good enough for 4th place at the moment.
> >
> >
> > Of course, the next task is to put this through the actual distributed
> > processing -- that's really the appropriate solution.
> >
> > This shows you can still tackle fairly impressive scale with a
> > non-distributed solution. These results suggest that the largest
> > instances available from EC2 would accomodate almost 1 billion ratings
> > in memory. However at that scale running a user's full recommendations
> > would easily be measured in seconds, not milliseconds.
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>

Re: First results from (non-distributed) Apache Mahout / Yahoo KDD Cup

Posted by Grant Ingersoll <gs...@apache.org>.
Nice, Sean!  I love that a near out of the box implementation puts us in 4th place!

On Mar 20, 2011, at 8:03 PM, Sean Owen wrote:

> I've been test-driving a simple application of Mahout recommenders
> (the non-distributed kind) on Amazon EC2 on the new Yahoo KDD Cup data
> set (kddcup.yahoo.com).
> 
> In the spirit of open-source, like I mentioned, I'm committing the
> extra code to mahout-examples that can be used to run a Recommender on
> the input and output the right format. And, I'd like to publish the
> rough timings too. Find all the source in
> org.apache.mahout.cf.taste.example.kddcup
> 
> 
> Track 1
> 
> m2.2xlarge instance, 34.2GB RAM / 4 cores
> Steady state memory consumption: ~19GB
> Computation time: 30 hours (wall clock-time)
> CPU time per user: ~0.43 sec
> Cost on EC2: $34.20 (!)
> 
> (Helpful hint on cost I realized after the fact: you can almost surely
> get spot instances for cheaper. The maximum price this sort of
> instance has gone for as a spot instance is about $0.60/hour, vs
> "retail price" of $1.14/hour.)
> 
> Resulted in an RMSE of 29.5618 (the rating scale is 0-100), which is
> only good enough for 29th place at the moment. Not terrible for "out
> of the box" performance -- it's just using an item-based recommender
> with uncentered cosine similarity. But not really good in absolute
> terms. A winning solution is going to try to factor in time, and apply
> more sophisticated techniques. The best RMSE so far is about 23.
> 
> 
> Track 2
> 
> c1.xlarge instance: 7GB RAM / 8 cores
> Steady state memory consumption: ~3.8GB
> Computation time: 4.1 hours (wall clock-time)
> CPU time per user: ~1.1 sec
> Cost on EC2: $3.20
> 
> For this I bothered to write a simplistic item-item similarity metric
> to take into account the additional info that is available: track,
> artist, album, genre. The result was comparatively better: 17.92%
> error rate, good enough for 4th place at the moment.
> 
> 
> Of course, the next task is to put this through the actual distributed
> processing -- that's really the appropriate solution.
> 
> This shows you can still tackle fairly impressive scale with a
> non-distributed solution. These results suggest that the largest
> instances available from EC2 would accomodate almost 1 billion ratings
> in memory. However at that scale running a user's full recommendations
> would easily be measured in seconds, not milliseconds.

--------------------------
Grant Ingersoll
http://www.lucidimagination.com