Changing in-memory DataModel to a DB dependent only DataModel after building recommender

[Ceyhun Can ÜLKER <ce...@gmail.com>]

Hello,

In our application we are using ReloadFromJDBCDataModel for its speed
advantage of in-memory representation and being able to update periodically
to pull in new data from a database source.

However, once the recommender is build we do not want to keep the ratings
data in memory (we would like to query the database when rating data is
needed). We want to replace the ReloadFromJDBCDataModel with a
MySqlJDBCDataModel after build. But there is no setter method for it,
furthermore, the field that keeps the DataModel is in AbstractRecommender
(superclass of SVDRecommender) and it is declared final.

We thought we could write a new class that derives from DataModel, which
initial keeps a Reload model instance (let's call this delegateModel), has
a setter method for it, and delegates all DataModel methods, so that we
could set this delegateModel field to another instance, say
MySqlJDBCDataModel instance. Is this a good method for removing in-memory
representation dependency after the build process?

How can we achieve this change? Or is there an alternative and better way
to achieve this?

Thanks
Ceyhun Can Ulker

Re: Changing in-memory DataModel to a DB dependent only DataModel after building recommender

[Ceyhun Can ÜLKER <ce...@gmail.com>]

Hello Sean,

Thank you for your reply. What I understand from your reply is we are fine
with the proposed solution in the first mail.

Thanks again.
Regards.
Ceyhun Can Ulker


On Mon, Jan 21, 2013 at 10:59 PM, Sean Owen <sr...@gmail.com> wrote:

> That's all right, but it's not something implemented in this project,
> except maybe in SVDRecommender to some extent.
>
> On Mon, Jan 21, 2013 at 8:57 PM, Osman Başkaya
> <os...@computer.org> wrote:
> > Hey Sean,
> >
> > Once you compute user and item factors, you don't need to create a dense
> > matrix in which all the user's predicted ratings contains. Of course, one
> > dependency remains: you may want to know which user rates which movies.
> > Otherwise you recommend movies which are rated already. However,
> computing
> > all predictions and storing them does not provides this, neither.
> >
> > On Mon, Jan 21, 2013 at 10:40 PM, Sean Owen <sr...@gmail.com> wrote:
> >
> >> If you don't have the data in memory you can't compute anything. The
> >> recommender itself doesn't do anything without data. That's why it
> >> seemed like you really just wanted to compute everything offline
> >> first, in which case the simplest solution is to store it however you
> >> like and fetch that result however you like.
> >>
> >> On Mon, Jan 21, 2013 at 8:22 PM, Ceyhun Can ÜLKER <ceyhuncanu@gmail.com
> >
> >> wrote:
> >> > Hi again,
> >> >
> >> > Thank you for your quick reply, Sean. I couldn't understand one point.
> >> What
> >> > do you mean by pre-compute and store recommendations? Doesn't it mean
> >> > having a dense (rather filled?) rating matrix? So it would make memory
> >> > usage much worse, even if it is possible. Wouldn't it better to keep
> the
> >> > model and compute whenever necessary?
> >> >
> >> > Thanks
> >> > Ceyhun Can Ulker
> >> >
> >> >
> >> > On Mon, Jan 21, 2013 at 9:58 PM, Sean Owen <sr...@gmail.com> wrote:
> >> >
> >> >> You would have to write this yourself, yes.
> >> >> If you're not keeping the data in memory, you're not updating the
> >> >> results in real-time. So there's no real need to keep any DataModel
> >> >> around at all. Just pre-compute and store recommendations and update
> >> >> them periodically. Nothing has to be on-line then.
> >> >>
> >> >> On Mon, Jan 21, 2013 at 7:54 PM, Ceyhun Can ÜLKER <
> ceyhuncanu@gmail.com
> >> >
> >> >> wrote:
> >> >> > Hello,
> >> >> >
> >> >> > In our application we are using ReloadFromJDBCDataModel for its
> speed
> >> >> > advantage of in-memory representation and being able to update
> >> >> periodically
> >> >> > to pull in new data from a database source.
> >> >> >
> >> >> > However, once the recommender is build we do not want to keep the
> >> ratings
> >> >> > data in memory (we would like to query the database when rating
> data
> >> is
> >> >> > needed). We want to replace the ReloadFromJDBCDataModel with a
> >> >> > MySqlJDBCDataModel after build. But there is no setter method for
> it,
> >> >> > furthermore, the field that keeps the DataModel is in
> >> AbstractRecommender
> >> >> > (superclass of SVDRecommender) and it is declared final.
> >> >> >
> >> >> > We thought we could write a new class that derives from DataModel,
> >> which
> >> >> > initial keeps a Reload model instance (let's call this
> delegateModel),
> >> >> has
> >> >> > a setter method for it, and delegates all DataModel methods, so
> that
> >> we
> >> >> > could set this delegateModel field to another instance, say
> >> >> > MySqlJDBCDataModel instance. Is this a good method for removing
> >> in-memory
> >> >> > representation dependency after the build process?
> >> >> >
> >> >> > How can we achieve this change? Or is there an alternative and
> better
> >> way
> >> >> > to achieve this?
> >> >> >
> >> >> > Thanks
> >> >> > Ceyhun Can Ulker
> >> >>
> >>
> >
> >
> >
> > --
> > Osman Başkaya
> > Koc University
> > MS Student | Computer Science and Engineering
>

Re: Changing in-memory DataModel to a DB dependent only DataModel after building recommender

[Sean Owen <sr...@gmail.com>]

That's all right, but it's not something implemented in this project,
except maybe in SVDRecommender to some extent.

On Mon, Jan 21, 2013 at 8:57 PM, Osman Başkaya
<os...@computer.org> wrote:
> Hey Sean,
>
> Once you compute user and item factors, you don't need to create a dense
> matrix in which all the user's predicted ratings contains. Of course, one
> dependency remains: you may want to know which user rates which movies.
> Otherwise you recommend movies which are rated already. However, computing
> all predictions and storing them does not provides this, neither.
>
> On Mon, Jan 21, 2013 at 10:40 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> If you don't have the data in memory you can't compute anything. The
>> recommender itself doesn't do anything without data. That's why it
>> seemed like you really just wanted to compute everything offline
>> first, in which case the simplest solution is to store it however you
>> like and fetch that result however you like.
>>
>> On Mon, Jan 21, 2013 at 8:22 PM, Ceyhun Can ÜLKER <ce...@gmail.com>
>> wrote:
>> > Hi again,
>> >
>> > Thank you for your quick reply, Sean. I couldn't understand one point.
>> What
>> > do you mean by pre-compute and store recommendations? Doesn't it mean
>> > having a dense (rather filled?) rating matrix? So it would make memory
>> > usage much worse, even if it is possible. Wouldn't it better to keep the
>> > model and compute whenever necessary?
>> >
>> > Thanks
>> > Ceyhun Can Ulker
>> >
>> >
>> > On Mon, Jan 21, 2013 at 9:58 PM, Sean Owen <sr...@gmail.com> wrote:
>> >
>> >> You would have to write this yourself, yes.
>> >> If you're not keeping the data in memory, you're not updating the
>> >> results in real-time. So there's no real need to keep any DataModel
>> >> around at all. Just pre-compute and store recommendations and update
>> >> them periodically. Nothing has to be on-line then.
>> >>
>> >> On Mon, Jan 21, 2013 at 7:54 PM, Ceyhun Can ÜLKER <ceyhuncanu@gmail.com
>> >
>> >> wrote:
>> >> > Hello,
>> >> >
>> >> > In our application we are using ReloadFromJDBCDataModel for its speed
>> >> > advantage of in-memory representation and being able to update
>> >> periodically
>> >> > to pull in new data from a database source.
>> >> >
>> >> > However, once the recommender is build we do not want to keep the
>> ratings
>> >> > data in memory (we would like to query the database when rating data
>> is
>> >> > needed). We want to replace the ReloadFromJDBCDataModel with a
>> >> > MySqlJDBCDataModel after build. But there is no setter method for it,
>> >> > furthermore, the field that keeps the DataModel is in
>> AbstractRecommender
>> >> > (superclass of SVDRecommender) and it is declared final.
>> >> >
>> >> > We thought we could write a new class that derives from DataModel,
>> which
>> >> > initial keeps a Reload model instance (let's call this delegateModel),
>> >> has
>> >> > a setter method for it, and delegates all DataModel methods, so that
>> we
>> >> > could set this delegateModel field to another instance, say
>> >> > MySqlJDBCDataModel instance. Is this a good method for removing
>> in-memory
>> >> > representation dependency after the build process?
>> >> >
>> >> > How can we achieve this change? Or is there an alternative and better
>> way
>> >> > to achieve this?
>> >> >
>> >> > Thanks
>> >> > Ceyhun Can Ulker
>> >>
>>
>
>
>
> --
> Osman Başkaya
> Koc University
> MS Student | Computer Science and Engineering

Re: Changing in-memory DataModel to a DB dependent only DataModel after building recommender

[Osman Başkaya <os...@computer.org>]

Hey Sean,

Once you compute user and item factors, you don't need to create a dense
matrix in which all the user's predicted ratings contains. Of course, one
dependency remains: you may want to know which user rates which movies.
Otherwise you recommend movies which are rated already. However, computing
all predictions and storing them does not provides this, neither.

On Mon, Jan 21, 2013 at 10:40 PM, Sean Owen <sr...@gmail.com> wrote:

> If you don't have the data in memory you can't compute anything. The
> recommender itself doesn't do anything without data. That's why it
> seemed like you really just wanted to compute everything offline
> first, in which case the simplest solution is to store it however you
> like and fetch that result however you like.
>
> On Mon, Jan 21, 2013 at 8:22 PM, Ceyhun Can ÜLKER <ce...@gmail.com>
> wrote:
> > Hi again,
> >
> > Thank you for your quick reply, Sean. I couldn't understand one point.
> What
> > do you mean by pre-compute and store recommendations? Doesn't it mean
> > having a dense (rather filled?) rating matrix? So it would make memory
> > usage much worse, even if it is possible. Wouldn't it better to keep the
> > model and compute whenever necessary?
> >
> > Thanks
> > Ceyhun Can Ulker
> >
> >
> > On Mon, Jan 21, 2013 at 9:58 PM, Sean Owen <sr...@gmail.com> wrote:
> >
> >> You would have to write this yourself, yes.
> >> If you're not keeping the data in memory, you're not updating the
> >> results in real-time. So there's no real need to keep any DataModel
> >> around at all. Just pre-compute and store recommendations and update
> >> them periodically. Nothing has to be on-line then.
> >>
> >> On Mon, Jan 21, 2013 at 7:54 PM, Ceyhun Can ÜLKER <ceyhuncanu@gmail.com
> >
> >> wrote:
> >> > Hello,
> >> >
> >> > In our application we are using ReloadFromJDBCDataModel for its speed
> >> > advantage of in-memory representation and being able to update
> >> periodically
> >> > to pull in new data from a database source.
> >> >
> >> > However, once the recommender is build we do not want to keep the
> ratings
> >> > data in memory (we would like to query the database when rating data
> is
> >> > needed). We want to replace the ReloadFromJDBCDataModel with a
> >> > MySqlJDBCDataModel after build. But there is no setter method for it,
> >> > furthermore, the field that keeps the DataModel is in
> AbstractRecommender
> >> > (superclass of SVDRecommender) and it is declared final.
> >> >
> >> > We thought we could write a new class that derives from DataModel,
> which
> >> > initial keeps a Reload model instance (let's call this delegateModel),
> >> has
> >> > a setter method for it, and delegates all DataModel methods, so that
> we
> >> > could set this delegateModel field to another instance, say
> >> > MySqlJDBCDataModel instance. Is this a good method for removing
> in-memory
> >> > representation dependency after the build process?
> >> >
> >> > How can we achieve this change? Or is there an alternative and better
> way
> >> > to achieve this?
> >> >
> >> > Thanks
> >> > Ceyhun Can Ulker
> >>
>



-- 
Osman Başkaya
Koc University
MS Student | Computer Science and Engineering

Re: Changing in-memory DataModel to a DB dependent only DataModel after building recommender

[Sean Owen <sr...@gmail.com>]

If you don't have the data in memory you can't compute anything. The
recommender itself doesn't do anything without data. That's why it
seemed like you really just wanted to compute everything offline
first, in which case the simplest solution is to store it however you
like and fetch that result however you like.

On Mon, Jan 21, 2013 at 8:22 PM, Ceyhun Can ÜLKER <ce...@gmail.com> wrote:
> Hi again,
>
> Thank you for your quick reply, Sean. I couldn't understand one point. What
> do you mean by pre-compute and store recommendations? Doesn't it mean
> having a dense (rather filled?) rating matrix? So it would make memory
> usage much worse, even if it is possible. Wouldn't it better to keep the
> model and compute whenever necessary?
>
> Thanks
> Ceyhun Can Ulker
>
>
> On Mon, Jan 21, 2013 at 9:58 PM, Sean Owen <sr...@gmail.com> wrote:
>
>> You would have to write this yourself, yes.
>> If you're not keeping the data in memory, you're not updating the
>> results in real-time. So there's no real need to keep any DataModel
>> around at all. Just pre-compute and store recommendations and update
>> them periodically. Nothing has to be on-line then.
>>
>> On Mon, Jan 21, 2013 at 7:54 PM, Ceyhun Can ÜLKER <ce...@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > In our application we are using ReloadFromJDBCDataModel for its speed
>> > advantage of in-memory representation and being able to update
>> periodically
>> > to pull in new data from a database source.
>> >
>> > However, once the recommender is build we do not want to keep the ratings
>> > data in memory (we would like to query the database when rating data is
>> > needed). We want to replace the ReloadFromJDBCDataModel with a
>> > MySqlJDBCDataModel after build. But there is no setter method for it,
>> > furthermore, the field that keeps the DataModel is in AbstractRecommender
>> > (superclass of SVDRecommender) and it is declared final.
>> >
>> > We thought we could write a new class that derives from DataModel, which
>> > initial keeps a Reload model instance (let's call this delegateModel),
>> has
>> > a setter method for it, and delegates all DataModel methods, so that we
>> > could set this delegateModel field to another instance, say
>> > MySqlJDBCDataModel instance. Is this a good method for removing in-memory
>> > representation dependency after the build process?
>> >
>> > How can we achieve this change? Or is there an alternative and better way
>> > to achieve this?
>> >
>> > Thanks
>> > Ceyhun Can Ulker
>>

Re: Changing in-memory DataModel to a DB dependent only DataModel after building recommender

[Ceyhun Can ÜLKER <ce...@gmail.com>]

Hi again,

Thank you for your quick reply, Sean. I couldn't understand one point. What
do you mean by pre-compute and store recommendations? Doesn't it mean
having a dense (rather filled?) rating matrix? So it would make memory
usage much worse, even if it is possible. Wouldn't it better to keep the
model and compute whenever necessary?

Thanks
Ceyhun Can Ulker


On Mon, Jan 21, 2013 at 9:58 PM, Sean Owen <sr...@gmail.com> wrote:

> You would have to write this yourself, yes.
> If you're not keeping the data in memory, you're not updating the
> results in real-time. So there's no real need to keep any DataModel
> around at all. Just pre-compute and store recommendations and update
> them periodically. Nothing has to be on-line then.
>
> On Mon, Jan 21, 2013 at 7:54 PM, Ceyhun Can ÜLKER <ce...@gmail.com>
> wrote:
> > Hello,
> >
> > In our application we are using ReloadFromJDBCDataModel for its speed
> > advantage of in-memory representation and being able to update
> periodically
> > to pull in new data from a database source.
> >
> > However, once the recommender is build we do not want to keep the ratings
> > data in memory (we would like to query the database when rating data is
> > needed). We want to replace the ReloadFromJDBCDataModel with a
> > MySqlJDBCDataModel after build. But there is no setter method for it,
> > furthermore, the field that keeps the DataModel is in AbstractRecommender
> > (superclass of SVDRecommender) and it is declared final.
> >
> > We thought we could write a new class that derives from DataModel, which
> > initial keeps a Reload model instance (let's call this delegateModel),
> has
> > a setter method for it, and delegates all DataModel methods, so that we
> > could set this delegateModel field to another instance, say
> > MySqlJDBCDataModel instance. Is this a good method for removing in-memory
> > representation dependency after the build process?
> >
> > How can we achieve this change? Or is there an alternative and better way
> > to achieve this?
> >
> > Thanks
> > Ceyhun Can Ulker
>

Re: Changing in-memory DataModel to a DB dependent only DataModel after building recommender

[Sean Owen <sr...@gmail.com>]

You would have to write this yourself, yes.
If you're not keeping the data in memory, you're not updating the
results in real-time. So there's no real need to keep any DataModel
around at all. Just pre-compute and store recommendations and update
them periodically. Nothing has to be on-line then.

On Mon, Jan 21, 2013 at 7:54 PM, Ceyhun Can ÜLKER <ce...@gmail.com> wrote:
> Hello,
>
> In our application we are using ReloadFromJDBCDataModel for its speed
> advantage of in-memory representation and being able to update periodically
> to pull in new data from a database source.
>
> However, once the recommender is build we do not want to keep the ratings
> data in memory (we would like to query the database when rating data is
> needed). We want to replace the ReloadFromJDBCDataModel with a
> MySqlJDBCDataModel after build. But there is no setter method for it,
> furthermore, the field that keeps the DataModel is in AbstractRecommender
> (superclass of SVDRecommender) and it is declared final.
>
> We thought we could write a new class that derives from DataModel, which
> initial keeps a Reload model instance (let's call this delegateModel), has
> a setter method for it, and delegates all DataModel methods, so that we
> could set this delegateModel field to another instance, say
> MySqlJDBCDataModel instance. Is this a good method for removing in-memory
> representation dependency after the build process?
>
> How can we achieve this change? Or is there an alternative and better way
> to achieve this?
>
> Thanks
> Ceyhun Can Ulker