You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by udachny <a....@gmail.com> on 2011/09/28 10:38:46 UTC

Scalability concerns with concurrent recommendations

I am building a large scale user-based recommender which could have up to 1
billion preferences. I am planning to reduce it to the 100M chunks are
recommended by Sean:
http://lucene.472066.n3.nabble.com/Evaluating-Mahout-s-recommender-support-td2161876.html#a2167800

With this size, the application can be set up in a non-distributed mode. The
recommender will be set up as a web servlet whose service will be consumed
by web applications. That means there will be lots of concurrent
recommendation requests. 

Thus my main concern is how well do Mahout recommenders handle the volumes
of concurrent recommendations. I have done some benchmarking with JMeter
using out-of-the-box Mahout GenericBooleanPrefUserBasedRecommender examples
and am seeing the following trends:

Number of concurrent recommendations | time per recommendation
10 | 150 ms
100 | 2000 ms
1000 | 14000 ms

As you can see, even with 100 concurrent users the time-per-recommendation
is unacceptably slow. 

Has anyone done more benchmarks about concurrent recommendations?
Can you post any architectural ideas about setting up scalable distributed
recommenders that can handle high concurrency?








--
View this message in context: http://lucene.472066.n3.nabble.com/Scalability-concerns-with-concurrent-recommendations-tp3375424p3375424.html
Sent from the Mahout User List mailing list archive at Nabble.com.

Re: Scalability concerns with concurrent recommendations

Posted by Sebastian Schelter <ss...@apache.org>.
Maybe this article will help you:

http://ssc.io/deploying-a-massively-scalable-recommender-system-with-apache-mahout/

Can you give some more details about your recommender setup?

Some general hints that to scale that out:

* precompute item similarities, only keep the top k similar items per
item (use something like 50), experiment with this number

* make sure these item similarities are loaded into memory and the
candidate item strategy of your recommender accesses those directly (eg
use MySQLJDBCInMemoryItemSimilarity together with
AllSimilarItemsCandidateItemsStrategy)

* only use a max number of preferences per user for recommendation,
maybe the n latest interactions, experiment with this number, make sure
you either fetch those preferences from memory (if the data fits into
RAM) or you use a setup similar to that in my blogpost where you can
fetch all preferences for a user in a single database query

--sebastian




On 28.09.2011 10:38, udachny wrote:
> I am building a large scale user-based recommender which could have up to 1
> billion preferences. I am planning to reduce it to the 100M chunks are
> recommended by Sean:
> http://lucene.472066.n3.nabble.com/Evaluating-Mahout-s-recommender-support-td2161876.html#a2167800
> 
> With this size, the application can be set up in a non-distributed mode. The
> recommender will be set up as a web servlet whose service will be consumed
> by web applications. That means there will be lots of concurrent
> recommendation requests. 
> 
> Thus my main concern is how well do Mahout recommenders handle the volumes
> of concurrent recommendations. I have done some benchmarking with JMeter
> using out-of-the-box Mahout GenericBooleanPrefUserBasedRecommender examples
> and am seeing the following trends:
> 
> Number of concurrent recommendations | time per recommendation
> 10 | 150 ms
> 100 | 2000 ms
> 1000 | 14000 ms
> 
> As you can see, even with 100 concurrent users the time-per-recommendation
> is unacceptably slow. 
> 
> Has anyone done more benchmarks about concurrent recommendations?
> Can you post any architectural ideas about setting up scalable distributed
> recommenders that can handle high concurrency?
> 
> 
> 
> 
> 
> 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Scalability-concerns-with-concurrent-recommendations-tp3375424p3375424.html
> Sent from the Mahout User List mailing list archive at Nabble.com.


Re: Scalability concerns with concurrent recommendations

Posted by Ted Dunning <te...@gmail.com>.
Also, concurrent connections is not a very understandable metric.

It is much better to use queries per second as the independent variable and
then look at response time.  You can estimate queries per second from your
traffic much more easily than concurrency level.  Moreover, your audience is
driving queries per second and for the same rate, the concurrency level can
change dramatically due to secondary factors like machine loading.

On Wed, Sep 28, 2011 at 9:20 AM, Sean Owen <sr...@gmail.com> wrote:

> Those times sound about right to me. Past about 10 you're starting to be
> bottlenecked on CPU time, so the response time is scaling up nearly
> linearly.
> You're just going to need to throw more machines at it: for 10x the volume
> put up 10x servers. That's actually the good news.
>
> On Wed, Sep 28, 2011 at 9:38 AM, udachny <a....@gmail.com> wrote:
>
> > I am building a large scale user-based recommender which could have up to
> 1
> > billion preferences. I am planning to reduce it to the 100M chunks are
> > recommended by Sean:
> >
> >
> http://lucene.472066.n3.nabble.com/Evaluating-Mahout-s-recommender-support-td2161876.html#a2167800
> >
> > With this size, the application can be set up in a non-distributed mode.
> > The
> > recommender will be set up as a web servlet whose service will be
> consumed
> > by web applications. That means there will be lots of concurrent
> > recommendation requests.
> >
> > Thus my main concern is how well do Mahout recommenders handle the
> volumes
> > of concurrent recommendations. I have done some benchmarking with JMeter
> > using out-of-the-box Mahout GenericBooleanPrefUserBasedRecommender
> examples
> > and am seeing the following trends:
> >
> > Number of concurrent recommendations | time per recommendation
> > 10 | 150 ms
> > 100 | 2000 ms
> > 1000 | 14000 ms
> >
> > As you can see, even with 100 concurrent users the
> time-per-recommendation
> > is unacceptably slow.
> >
> > Has anyone done more benchmarks about concurrent recommendations?
> > Can you post any architectural ideas about setting up scalable
> distributed
> > recommenders that can handle high concurrency?
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Scalability-concerns-with-concurrent-recommendations-tp3375424p3375424.html
> > Sent from the Mahout User List mailing list archive at Nabble.com.
> >
>

Re: Scalability concerns with concurrent recommendations

Posted by Sean Owen <sr...@gmail.com>.
Those times sound about right to me. Past about 10 you're starting to be
bottlenecked on CPU time, so the response time is scaling up nearly
linearly.
You're just going to need to throw more machines at it: for 10x the volume
put up 10x servers. That's actually the good news.

On Wed, Sep 28, 2011 at 9:38 AM, udachny <a....@gmail.com> wrote:

> I am building a large scale user-based recommender which could have up to 1
> billion preferences. I am planning to reduce it to the 100M chunks are
> recommended by Sean:
>
> http://lucene.472066.n3.nabble.com/Evaluating-Mahout-s-recommender-support-td2161876.html#a2167800
>
> With this size, the application can be set up in a non-distributed mode.
> The
> recommender will be set up as a web servlet whose service will be consumed
> by web applications. That means there will be lots of concurrent
> recommendation requests.
>
> Thus my main concern is how well do Mahout recommenders handle the volumes
> of concurrent recommendations. I have done some benchmarking with JMeter
> using out-of-the-box Mahout GenericBooleanPrefUserBasedRecommender examples
> and am seeing the following trends:
>
> Number of concurrent recommendations | time per recommendation
> 10 | 150 ms
> 100 | 2000 ms
> 1000 | 14000 ms
>
> As you can see, even with 100 concurrent users the time-per-recommendation
> is unacceptably slow.
>
> Has anyone done more benchmarks about concurrent recommendations?
> Can you post any architectural ideas about setting up scalable distributed
> recommenders that can handle high concurrency?
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Scalability-concerns-with-concurrent-recommendations-tp3375424p3375424.html
> Sent from the Mahout User List mailing list archive at Nabble.com.
>