FileDataModel vs ReloadFromJDBCDataModel

[Onur Kuru <ku...@gmail.com>]

Hi all,

If I use FileDataModel, it takes about 5 secs to build the data model
with 1m movielens data but it takes about 25 secs if I use
ReloadFromJDBCDataModel.
I know the former uses file and the latter uses a database but is this
performance difference normal?

Thank you,
Onur

Re: FileDataModel vs ReloadFromJDBCDataModel

[Sean Owen <sr...@gmail.com>]

It doesn't sound too crazy since it is  loading the entire contents of the
table in memory. It does so in one query. You can dig out that query and
run it separately to see how long it takes otherwise and make sure it's not
wildly different. That's just the time taken to read it, once, and then
it's in memory.


On Sun, Nov 11, 2012 at 10:53 AM, Onur Kuru <ku...@gmail.com> wrote:

> Hi all,
>
> If I use FileDataModel, it takes about 5 secs to build the data model
> with 1m movielens data but it takes about 25 secs if I use
> ReloadFromJDBCDataModel.
> I know the former uses file and the latter uses a database but is this
> performance difference normal?
>
> Thank you,
> Onur
>

Re: FileDataModel vs ReloadFromJDBCDataModel

[Sebastian Schelter <ss...@apache.org>]

Hi Onur,

I'd say these number looks somewhat reasonable. Reading a file
sequentially from disk does not incur the overhead of going through the
database.

However the time to load a data model from the database is subject to
the configuration settings of the database. I'm sure you can reduce the
loading time by tuning those.

Best,
Sebastian

On 11.11.2012 11:53, Onur Kuru wrote:
> Hi all,
> 
> If I use FileDataModel, it takes about 5 secs to build the data model
> with 1m movielens data but it takes about 25 secs if I use
> ReloadFromJDBCDataModel.
> I know the former uses file and the latter uses a database but is this
> performance difference normal?
> 
> Thank you,
> Onur
>