Re: Can Mahout make recommendations while the recommender is being refreshed?

[Sebastian Schelter <ss...@apache.org>]

+dev

Good thing to talk about that issue, it's really not an ideal situation
to have FileDataModel need double memory at refresh time as it
effectively cuts in a half the amount of preferences that can be loaded
into per memory per machine.

Only two issues come to my mind if we would do a simple update of the data:

 a) existing preferences that are not included in the refreshed data
will not be removed
 b) recommenders reading the data at update time see an inconsistent
snapshot of the data

Let's assume that you only update and add preferences, then a) cannot
occur. I'm not sure how much of a problem b) would be in that case, you
could miss some cooccurrences when a preference is not added yet and
have some similarity value wrong if an update has not yet occurred. I'd
guess that is not a big problem as long as you don't cache the
inconsistent recommendations resulting from that.

So an idea could be to have a FileDataModel that never deletes
preferences and have that directly update the data. Any thoughts on this?

--sebastian

Am 19.11.2010 21:49, schrieb Sean Owen:
> ut the old dat

Re: Can Mahout make recommendations while the recommender is being refreshed?

[Sean Owen <sr...@gmail.com>]

Agree with Ted's solution in practice.
Stuff should already be using floats, or have the option to use floats, for
this reason. The precision is usually not helpful in this context.

On Sun, Nov 21, 2010 at 4:44 AM, Lance Norskog <go...@gmail.com> wrote:

> Another option is to change the Double arrays to Floats.
>
> I tried this on one of the Recommenders (can't remember which one) and
> the speed and result value were exactly the same.
>
> On Fri, Nov 19, 2010 at 1:46 PM, Ted Dunning <te...@gmail.com>
> wrote:
> > If you are running that close to full, then running more than one
> > recommender might be a preferable solution.  You can take one off-line,
> > update it, bring it back, update the other.  This decreases redundancy
> for a
> > small period, but that is usually an acceptable risk.
> >
> > On Fri, Nov 19, 2010 at 1:44 PM, Sebastian Schelter <ss...@apache.org>
> wrote:
> >
> >> Good thing to talk about that issue, it's really not an ideal situation
> >> to have FileDataModel need double memory at refresh time as it
> >> effectively cuts in a half the amount of preferences that can be loaded
> >> into per memory per machine.
> >>
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>

Re: Can Mahout make recommendations while the recommender is being refreshed?

[Lance Norskog <go...@gmail.com>]

Another option is to change the Double arrays to Floats.

I tried this on one of the Recommenders (can't remember which one) and
the speed and result value were exactly the same.

On Fri, Nov 19, 2010 at 1:46 PM, Ted Dunning <te...@gmail.com> wrote:
> If you are running that close to full, then running more than one
> recommender might be a preferable solution.  You can take one off-line,
> update it, bring it back, update the other.  This decreases redundancy for a
> small period, but that is usually an acceptable risk.
>
> On Fri, Nov 19, 2010 at 1:44 PM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> Good thing to talk about that issue, it's really not an ideal situation
>> to have FileDataModel need double memory at refresh time as it
>> effectively cuts in a half the amount of preferences that can be loaded
>> into per memory per machine.
>>
>



-- 
Lance Norskog
goksron@gmail.com

Re: Can Mahout make recommendations while the recommender is being refreshed?

[Ted Dunning <te...@gmail.com>]

If you are running that close to full, then running more than one
recommender might be a preferable solution.  You can take one off-line,
update it, bring it back, update the other.  This decreases redundancy for a
small period, but that is usually an acceptable risk.

On Fri, Nov 19, 2010 at 1:44 PM, Sebastian Schelter <ss...@apache.org> wrote:

> Good thing to talk about that issue, it's really not an ideal situation
> to have FileDataModel need double memory at refresh time as it
> effectively cuts in a half the amount of preferences that can be loaded
> into per memory per machine.
>

Re: Can Mahout make recommendations while the recommender is being refreshed?

[Ted Dunning <te...@gmail.com>]

If you are running that close to full, then running more than one
recommender might be a preferable solution.  You can take one off-line,
update it, bring it back, update the other.  This decreases redundancy for a
small period, but that is usually an acceptable risk.

On Fri, Nov 19, 2010 at 1:44 PM, Sebastian Schelter <ss...@apache.org> wrote:

> Good thing to talk about that issue, it's really not an ideal situation
> to have FileDataModel need double memory at refresh time as it
> effectively cuts in a half the amount of preferences that can be loaded
> into per memory per machine.
>