save Mahout result in the memcached

[Hank Li <ha...@gmail.com>]

when the result is saved in the memory, mahout is running  very fast.
memcached can expand the available memory, does anybody try to use memcached
in mahout? what the pros and cons?

Re: save Mahout result in the memcached

[Ted Dunning <te...@gmail.com>]

I haven't heard of anybody doing this, but it should work pretty well.  It
would be
noticeably slower than in-memory, but much faster than something like JDBC.

It should be relatively easy to implement this.

On Mon, Oct 25, 2010 at 2:26 PM, Hank Li <ha...@gmail.com> wrote:

> when the result is saved in the memory, mahout is running  very fast.
> memcached can expand the available memory, does anybody try to use
> memcached
> in mahout? what the pros and cons?
>

Re: save Mahout result in the memcached

[Hank Li <ha...@gmail.com>]

Ted is correct, I am trying to use a cluster of machines to handle large
amount of recommendation requests. One design is to use hadoop to process
the raw data, then copy the result into the memcached or memcachedb. Another
design is to use memcached/b directly to process the data since it provides
unlimited memory for it to use. The key part is to let the recommendation
services to use memcached/b to do recommendation with fast response time.

Both memcached and memcachedb are good options, memcachedb is a little
complicated to setup than memcached though. failover issue can be solved by
other solutions too.

If anybody had done it or is going to test it, please share your experience.

Hank



On Mon, Oct 25, 2010 at 3:23 PM, Ted Dunning <te...@gmail.com> wrote:

> Stuffing more memory and using a larger heap is definitely better on a
> single machine.
>
> Memcached would allow you to centralize that memory if you have a cluster
> of
> recommenders
> and with the right clever settings would probably let you have a limited
> kind of failure tolerance.
>
> I think that the confusion here might be that memcached is memory backed
> while memcachedb
> is the equivalent (almost) with persistence.  That single letter difference
> is very hard to see.
>
> Again, this is only interesting if you have a cluster of recommendation
> machines.  If you don't,
> Sean is exactly right that keeping the data on a single large machine is
> better.
>
> On Mon, Oct 25, 2010 at 3:00 PM, Sean Owen <sr...@gmail.com> wrote:
>
> > Perhaps I'm naive but wouldn't virtual memory be a better way to expand
> > memory by using storage? Or, I think you'd have to know there were
> > particular access patterns, and design your use of memcached to exploit
> > those well, to do better than swap.
> >
> > On Mon, Oct 25, 2010 at 10:26 PM, Hank Li <ha...@gmail.com> wrote:
> >
> > > when the result is saved in the memory, mahout is running  very fast.
> > > memcached can expand the available memory, does anybody try to use
> > > memcached
> > > in mahout? what the pros and cons?
> > >
> >
>

Re: save Mahout result in the memcached

[Ted Dunning <te...@gmail.com>]

Stuffing more memory and using a larger heap is definitely better on a
single machine.

Memcached would allow you to centralize that memory if you have a cluster of
recommenders
and with the right clever settings would probably let you have a limited
kind of failure tolerance.

I think that the confusion here might be that memcached is memory backed
while memcachedb
is the equivalent (almost) with persistence.  That single letter difference
is very hard to see.

Again, this is only interesting if you have a cluster of recommendation
machines.  If you don't,
Sean is exactly right that keeping the data on a single large machine is
better.

On Mon, Oct 25, 2010 at 3:00 PM, Sean Owen <sr...@gmail.com> wrote:

> Perhaps I'm naive but wouldn't virtual memory be a better way to expand
> memory by using storage? Or, I think you'd have to know there were
> particular access patterns, and design your use of memcached to exploit
> those well, to do better than swap.
>
> On Mon, Oct 25, 2010 at 10:26 PM, Hank Li <ha...@gmail.com> wrote:
>
> > when the result is saved in the memory, mahout is running  very fast.
> > memcached can expand the available memory, does anybody try to use
> > memcached
> > in mahout? what the pros and cons?
> >
>

Re: save Mahout result in the memcached

[Sean Owen <sr...@gmail.com>]

Perhaps I'm naive but wouldn't virtual memory be a better way to expand
memory by using storage? Or, I think you'd have to know there were
particular access patterns, and design your use of memcached to exploit
those well, to do better than swap.

On Mon, Oct 25, 2010 at 10:26 PM, Hank Li <ha...@gmail.com> wrote:

> when the result is saved in the memory, mahout is running  very fast.
> memcached can expand the available memory, does anybody try to use
> memcached
> in mahout? what the pros and cons?
>