You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metamodel.apache.org by "tesmai4@gmail.com" <te...@gmail.com> on 2016/08/24 18:15:29 UTC
retain date while processing
Hi,
New to the list and beginner with Apache Metamodel.
I need to load data from multiple data sources using Apache Metamodel. The
application will allow user to query the the data loaded from the sources.
What is the best option to retain the loaded data in memory while user is
processing queries on the data ( We don't want to re-load the data over and
over again)?
Regards,
Re: retain date while processing
Posted by Kasper Sørensen <i....@gmail.com>.
There are no limits on the maximum result set size, but if it is an
in-memory implementation (PojoDataContext) then the data is kept ... in
memory. So you need to be sure it's not huge or else Java will run out of
memory. In addition I would say that if the data is big, then typically the
indexing and other query optimizations that (even a lightweight) a database
provides is going to offer superior performance on most non-trivial queries.
2016-08-25 2:19 GMT-07:00 tesmai4@gmail.com <te...@gmail.com>:
> Thanks for your reply Kasper. It is helpful.
>
> Two question on the same thread:
>
> 1) What is the default/maximum query result set that Apache MetaModel
> would return.
>
> 2) What is the maximum query result set size that Apache MetaModel can
> handle without using any additional components in Java.
>
>
> Regards,
>
>
>
> On 24 Aug 2016 20:42, "Kasper Sørensen" <i....@gmail.com>
> wrote:
>
>> Do you want to offer the users to run any type of query (as in - a
>> MetaModel Query) or is it specific things like "lookup by some attribute".
>> In the first case, you could put it into e.g. a PojoDataContext and just
>> serve that as a in-memory datastore. Or you could use something like H2 or
>> Derby and offer that via a JdbcDataContext. In the latter case I would
>> suggest simply to load it into a HashMap or something like that.
>>
>> 2016-08-24 11:15 GMT-07:00 tesmai4@gmail.com <te...@gmail.com>:
>>
>>> Hi,
>>>
>>> New to the list and beginner with Apache Metamodel.
>>>
>>> I need to load data from multiple data sources using Apache Metamodel.
>>> The application will allow user to query the the data loaded from the
>>> sources.
>>>
>>> What is the best option to retain the loaded data in memory while user
>>> is processing queries on the data ( We don't want to re-load the data over
>>> and over again)?
>>>
>>>
>>> Regards,
>>>
>>
>>
Re: retain date while processing
Posted by "tesmai4@gmail.com" <te...@gmail.com>.
Thanks for your reply Kasper. It is helpful.
Two question on the same thread:
1) What is the default/maximum query result set that Apache MetaModel would
return.
2) What is the maximum query result set size that Apache MetaModel can
handle without using any additional components in Java.
Regards,
On 24 Aug 2016 20:42, "Kasper Sørensen" <i....@gmail.com>
wrote:
> Do you want to offer the users to run any type of query (as in - a
> MetaModel Query) or is it specific things like "lookup by some attribute".
> In the first case, you could put it into e.g. a PojoDataContext and just
> serve that as a in-memory datastore. Or you could use something like H2 or
> Derby and offer that via a JdbcDataContext. In the latter case I would
> suggest simply to load it into a HashMap or something like that.
>
> 2016-08-24 11:15 GMT-07:00 tesmai4@gmail.com <te...@gmail.com>:
>
>> Hi,
>>
>> New to the list and beginner with Apache Metamodel.
>>
>> I need to load data from multiple data sources using Apache Metamodel.
>> The application will allow user to query the the data loaded from the
>> sources.
>>
>> What is the best option to retain the loaded data in memory while user is
>> processing queries on the data ( We don't want to re-load the data over and
>> over again)?
>>
>>
>> Regards,
>>
>
>
Re: retain date while processing
Posted by Kasper Sørensen <i....@gmail.com>.
That looks correct to me :) Maybe as a step 0, if you wish to maximize your
use of MetaModel, you can also use JdbcDataContext to populate (with
executeUpdate) the H2 or Derby database.
2016-08-30 2:20 GMT-07:00 tesmai4@gmail.com <te...@gmail.com>:
> Dear Kasper,
>
> Tying to understanding the flow for H2 or Derby. I'll do the following
> steps in the listed order:
>
> 1) Load data from a data source to H2/Derby for temporarily storing it in
> the memory
> 2) Use H2/Derby as data source for Apache MetaModel
> 3) Connect to H2/Derby data via jbdcDataContect
> 4) Query data using Apache MetaMoodel
>
> Is the above sequence correct or am I missing any step?
>
>
>
> Regards,
>
>
> On Wed, Aug 24, 2016 at 8:42 PM, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
>
>> Do you want to offer the users to run any type of query (as in - a
>> MetaModel Query) or is it specific things like "lookup by some attribute".
>> In the first case, you could put it into e.g. a PojoDataContext and just
>> serve that as a in-memory datastore. Or you could use something like H2 or
>> Derby and offer that via a JdbcDataContext. In the latter case I would
>> suggest simply to load it into a HashMap or something like that.
>>
>> 2016-08-24 11:15 GMT-07:00 tesmai4@gmail.com <te...@gmail.com>:
>>
>>> Hi,
>>>
>>> New to the list and beginner with Apache Metamodel.
>>>
>>> I need to load data from multiple data sources using Apache Metamodel.
>>> The application will allow user to query the the data loaded from the
>>> sources.
>>>
>>> What is the best option to retain the loaded data in memory while user
>>> is processing queries on the data ( We don't want to re-load the data over
>>> and over again)?
>>>
>>>
>>> Regards,
>>>
>>
>>
>
Re: retain date while processing
Posted by "tesmai4@gmail.com" <te...@gmail.com>.
Dear Kasper,
Tying to understanding the flow for H2 or Derby. I'll do the following
steps in the listed order:
1) Load data from a data source to H2/Derby for temporarily storing it in
the memory
2) Use H2/Derby as data source for Apache MetaModel
3) Connect to H2/Derby data via jbdcDataContect
4) Query data using Apache MetaMoodel
Is the above sequence correct or am I missing any step?
Regards,
On Wed, Aug 24, 2016 at 8:42 PM, Kasper Sørensen <
i.am.kasper.sorensen@gmail.com> wrote:
> Do you want to offer the users to run any type of query (as in - a
> MetaModel Query) or is it specific things like "lookup by some attribute".
> In the first case, you could put it into e.g. a PojoDataContext and just
> serve that as a in-memory datastore. Or you could use something like H2 or
> Derby and offer that via a JdbcDataContext. In the latter case I would
> suggest simply to load it into a HashMap or something like that.
>
> 2016-08-24 11:15 GMT-07:00 tesmai4@gmail.com <te...@gmail.com>:
>
>> Hi,
>>
>> New to the list and beginner with Apache Metamodel.
>>
>> I need to load data from multiple data sources using Apache Metamodel.
>> The application will allow user to query the the data loaded from the
>> sources.
>>
>> What is the best option to retain the loaded data in memory while user is
>> processing queries on the data ( We don't want to re-load the data over and
>> over again)?
>>
>>
>> Regards,
>>
>
>
Re: retain date while processing
Posted by Kasper Sørensen <i....@gmail.com>.
Do you want to offer the users to run any type of query (as in - a
MetaModel Query) or is it specific things like "lookup by some attribute".
In the first case, you could put it into e.g. a PojoDataContext and just
serve that as a in-memory datastore. Or you could use something like H2 or
Derby and offer that via a JdbcDataContext. In the latter case I would
suggest simply to load it into a HashMap or something like that.
2016-08-24 11:15 GMT-07:00 tesmai4@gmail.com <te...@gmail.com>:
> Hi,
>
> New to the list and beginner with Apache Metamodel.
>
> I need to load data from multiple data sources using Apache Metamodel. The
> application will allow user to query the the data loaded from the sources.
>
> What is the best option to retain the loaded data in memory while user is
> processing queries on the data ( We don't want to re-load the data over and
> over again)?
>
>
> Regards,
>