You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Øyvind Harboe <oy...@zylin.com> on 2008/06/30 15:06:26 UTC
Loading small tables into memory for the application lifetime
In the apps I'm working on I can generally sort tables into two
categories:
1. tables that change as part of normal operation of the database
and/or where the number of records is too large to fit into memory
2. various and sundry small tables. These have a small number
of records, change infrequently and can easily fit into memory. Loading
these tables take an instant(lest I'd categorize them as #1 tables).
#1 has relationships to #2 entries. These are constantly resolved via
sql queries.
At a guess I'd say that 95% of all queries are to resolve relationships from
#1 to #2. Ideally I should be able to mark a table as #2 via a configuration
option in my application as part of hardcoded tweaking of the app.
Is there a way to load #2 permanently into memory so as to avoid
SQL statements to resolve relationships from #1 to #2?
Since loading all #2 tables take an instant, they could be dumped if
more than N seconds has passed since they were last used.
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
RE: Loading small tables into memory for the application lifetime
Posted by Scott Anderson <sa...@airvana.com>.
Within a given context, unique database rows exist as unique objects. You could cache these objects with strong references, which would prevent them from being GC'd.
-----Original Message-----
From: oyvindharboe@gmail.com [mailto:oyvindharboe@gmail.com] On Behalf Of Øyvind Harboe
Sent: Monday, June 30, 2008 9:06 AM
To: cayenne-user@incubator.apache.org
Subject: Loading small tables into memory for the application lifetime
In the apps I'm working on I can generally sort tables into two
categories:
1. tables that change as part of normal operation of the database
and/or where the number of records is too large to fit into memory
2. various and sundry small tables. These have a small number
of records, change infrequently and can easily fit into memory. Loading
these tables take an instant(lest I'd categorize them as #1 tables).
#1 has relationships to #2 entries. These are constantly resolved via
sql queries.
At a guess I'd say that 95% of all queries are to resolve relationships from
#1 to #2. Ideally I should be able to mark a table as #2 via a configuration
option in my application as part of hardcoded tweaking of the app.
Is there a way to load #2 permanently into memory so as to avoid
SQL statements to resolve relationships from #1 to #2?
Since loading all #2 tables take an instant, they could be dumped if
more than N seconds has passed since they were last used.
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
Re: Loading small tables into memory for the application lifetime
Posted by Øyvind Harboe <oy...@zylin.com>.
I guess what I'm looking for is to have a writeback cache where entire
tables are loaded into memory upon startup of the application.
The tables and their content don't actually take up that much memory +
the memory is allocated upon startup so there are no surprises later
on by memory peak usage. Actual memory usage doesn't matter as much
as being able to know precisely what to set maximum heap to & knowing
how much physical memory to install.
Of course that would require Cayenne to be able to run queries against
tables in memory...
Cayenne is capable of resolving to-one relationships for objects that
are in memory, but to-many require a query to the database even
if all the resulting records are cached in memory already.
Or... I could be barking up the wrong tree... again.... I've been
pondering whether to modify the logger not to show queries that
take less than n ms. I never measured how long it actually
takes to log a query.... This sort of caching is what I would have
expected databases to have done to death...
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
Re: Loading small tables into memory for the application lifetime
Posted by Andrus Adamchik <an...@objectstyle.org>.
On Jun 30, 2008, at 6:33 PM, Mike Kienenberger wrote:
> But what about putting the constant tables into a parent DataContext,
> then doing all of your work in the child DataContext? Does that
> remove the need for using localObject, but still give you the
> flexibility to toss the child context objects?
That won't work directly, you still have to get a local object (it
won't be fetched from DB, and rather resolved from memory, but still).
But here is another way to work with lookup tables in 3.0 - use shared
caching... So locally you'd do this:
// you don't have to make it static; just wanted to point out
// that the query object itself is reusable (as long as there is no
parameters)
// This query can be mapped in the Modeler just as well.
static final SelectQuery countriesQuery = new
SelectQuery(Country.class);
static {
countriesQuery.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
}
public void myNonStaticMethod() {
...
List localCountries = context.performQuery(countriesQuery);
}
There is some overhead with the shared cache access, but still with
this approach you don't have to mess with 'localObject'. I'd say this
is an ideal separation of code from configuration.
Andrus
Re: Loading small tables into memory for the application lifetime
Posted by Mike Kienenberger <mk...@gmail.com>.
I don't remember the exact details, so this might be wrong.
But what about putting the constant tables into a parent DataContext,
then doing all of your work in the child DataContext? Does that
remove the need for using localObject, but still give you the
flexibility to toss the child context objects?
On Mon, Jun 30, 2008 at 11:11 AM, Michael Gentry <bl...@gmail.com> wrote:
> OK, thanks. I wasn't certain of that by any means and am too buried
> in other stuff to look it up.
>
> The pattern of using a dedicated DataContext for the small/cached
> tables and using localObject to make joins to them would still apply,
> though.
>
> mrg
>
>
> On Mon, Jun 30, 2008 at 10:49 AM, Andrus Adamchik
> <an...@objectstyle.org> wrote:
>>
>> On Jun 30, 2008, at 5:45 PM, Michael Gentry wrote:
>>
>>> I think in Cayenne 3 that it'll automatically pull an object
>>> into the DataContext if it exists in another DataContext.
>>
>> Not true. (well only true for transient unregistered objects). Generally you
>> still need to use 'localObject'.
>>
>> Andrus
>>
>>
>
Re: Loading small tables into memory for the application lifetime
Posted by Michael Gentry <bl...@gmail.com>.
OK, thanks. I wasn't certain of that by any means and am too buried
in other stuff to look it up.
The pattern of using a dedicated DataContext for the small/cached
tables and using localObject to make joins to them would still apply,
though.
mrg
On Mon, Jun 30, 2008 at 10:49 AM, Andrus Adamchik
<an...@objectstyle.org> wrote:
>
> On Jun 30, 2008, at 5:45 PM, Michael Gentry wrote:
>
>> I think in Cayenne 3 that it'll automatically pull an object
>> into the DataContext if it exists in another DataContext.
>
> Not true. (well only true for transient unregistered objects). Generally you
> still need to use 'localObject'.
>
> Andrus
>
>
Re: Loading small tables into memory for the application lifetime
Posted by Andrus Adamchik <an...@objectstyle.org>.
On Jun 30, 2008, at 5:45 PM, Michael Gentry wrote:
> I think in Cayenne 3 that it'll automatically pull an object
> into the DataContext if it exists in another DataContext.
Not true. (well only true for transient unregistered objects).
Generally you still need to use 'localObject'.
Andrus
Re: Loading small tables into memory for the application lifetime
Posted by Michael Gentry <bl...@gmail.com>.
Actually, I think in Cayenne 3 that it'll automatically pull an object
into the DataContext if it exists in another DataContext. My memory
is a bit rusty, though, but I'm sure someone will correct me if I am
wrong. But in a nutshell, make a separate/global DataContext for your
small tables and cache the data in it.
On Mon, Jun 30, 2008 at 10:28 AM, Michael Gentry <bl...@gmail.com> wrote:
> http://cayenne.apache.org/doc/moving-objects-between-contexts.html
>
> On Mon, Jun 30, 2008 at 9:06 AM, Øyvind Harboe <oy...@zylin.com> wrote:
>> In the apps I'm working on I can generally sort tables into two
>> categories:
>>
>> 1. tables that change as part of normal operation of the database
>> and/or where the number of records is too large to fit into memory
>>
>> 2. various and sundry small tables. These have a small number
>> of records, change infrequently and can easily fit into memory. Loading
>> these tables take an instant(lest I'd categorize them as #1 tables).
>>
>> #1 has relationships to #2 entries. These are constantly resolved via
>> sql queries.
>> At a guess I'd say that 95% of all queries are to resolve relationships from
>> #1 to #2. Ideally I should be able to mark a table as #2 via a configuration
>> option in my application as part of hardcoded tweaking of the app.
>>
>> Is there a way to load #2 permanently into memory so as to avoid
>> SQL statements to resolve relationships from #1 to #2?
>>
>>
>> Since loading all #2 tables take an instant, they could be dumped if
>> more than N seconds has passed since they were last used.
>>
>> --
>> Øyvind Harboe
>> http://www.zylin.com/zy1000.html
>> ARM7 ARM9 XScale Cortex
>> JTAG debugger and flash programmer
>>
>
Re: Loading small tables into memory for the application lifetime
Posted by Michael Gentry <bl...@gmail.com>.
http://cayenne.apache.org/doc/moving-objects-between-contexts.html
On Mon, Jun 30, 2008 at 9:06 AM, Øyvind Harboe <oy...@zylin.com> wrote:
> In the apps I'm working on I can generally sort tables into two
> categories:
>
> 1. tables that change as part of normal operation of the database
> and/or where the number of records is too large to fit into memory
>
> 2. various and sundry small tables. These have a small number
> of records, change infrequently and can easily fit into memory. Loading
> these tables take an instant(lest I'd categorize them as #1 tables).
>
> #1 has relationships to #2 entries. These are constantly resolved via
> sql queries.
> At a guess I'd say that 95% of all queries are to resolve relationships from
> #1 to #2. Ideally I should be able to mark a table as #2 via a configuration
> option in my application as part of hardcoded tweaking of the app.
>
> Is there a way to load #2 permanently into memory so as to avoid
> SQL statements to resolve relationships from #1 to #2?
>
>
> Since loading all #2 tables take an instant, they could be dumped if
> more than N seconds has passed since they were last used.
>
> --
> Øyvind Harboe
> http://www.zylin.com/zy1000.html
> ARM7 ARM9 XScale Cortex
> JTAG debugger and flash programmer
>