You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Øyvind Harboe <oy...@zylin.com> on 2008/06/30 10:54:59 UTC
Memory usage and select statements
An SQL statement includes a list of which columns to fetch for
the query.
Consider a table "book" with three columns, id(pk), title(<100 chars)
and content(the entire text of the book).
SELECT id, title FROM book
If one of the columns are big and rarely used, then it makes little
sense to transfer it to the client from the database for every
query. E.g. above the entire text of the book is left out.
The difference in memory usage can be quite dramatic.
Can I do the same w/Cayenne?
Can I have the columns fetched on-demand?
I'd like to avoid hardcoded SQL templates obviously.
SelectTranslator.createSqlString() unconditionally adds all
columns...
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
Re: Memory usage and select statements
Posted by Øyvind Harboe <oy...@zylin.com>.
I discovered that for some of the cases it is the relationships that
actually cause the massive memory consumption, not the
unused columns.
Once I realized this, I modified my app to do the following:
- perform a normal query. Everything into memory using a normal query.
- do some simple checks on each object to see if it requires processing
- if the object requires processing (evaluation of relationships), then first
create a datacontext for every object to be processed and then use
DataContext.localObject() & process the object within the newly
created DataContext.
- everything is now nicely garbage collected while at the same time
using a sensible amount of memory to speeed up the Java code
that does the selection of which records to process
Being able to select which columns to load on demand will be
important elsewhere though.
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
Re: Memory usage and select statements
Posted by Andrus Adamchik <an...@objectstyle.org>.
On Jun 30, 2008, at 3:23 PM, Scott Anderson wrote:
> If I'm not mistaken, using an incomplete SQLTemplate statement will
> allow fetching of incomplete DataObjects, and Cayenne will not go
> back to the database again until you try to access unfetched data.
In this case Cayenne will create completely hollow objects and fault
the entire object when you try to access any of the properties (even
if they where the part of the earlier query).
Andrus
RE: Memory usage and select statements
Posted by Scott Anderson <sa...@airvana.com>.
If I'm not mistaken, using an incomplete SQLTemplate statement will allow fetching of incomplete DataObjects, and Cayenne will not go back to the database again until you try to access unfetched data.
At least, that's what I remember the behavior being; SELECTs occurred when I stopped at a breakpoint where cayenne was persuaded to resolve DataObjects for display in the debugger's trace window.
-----Original Message-----
From: Andrus Adamchik [mailto:andrus@objectstyle.org]
Sent: Monday, June 30, 2008 5:01 AM
To: user@cayenne.apache.org
Subject: Re: Memory usage and select statements
There is no lazy fetching of attributes (only relationships). We will
be adding it at some point. It is a bit more involved than simply
suppressing a given column in the SQL translator. Still you have a few
of options:
1. Place large columns in a separate table with 1..1 relationship to
the master table .
2. Do not map large columns in the ObjEntity and resolve them manually
on demand (either with SQLTemplate, or by mapping a second entity to
the same table).
...
Andrus
On Jun 30, 2008, at 11:54 AM, Øyvind Harboe wrote:
> An SQL statement includes a list of which columns to fetch for
> the query.
>
> Consider a table "book" with three columns, id(pk), title(<100 chars)
> and content(the entire text of the book).
>
> SELECT id, title FROM book
>
> If one of the columns are big and rarely used, then it makes little
> sense to transfer it to the client from the database for every
> query. E.g. above the entire text of the book is left out.
>
> The difference in memory usage can be quite dramatic.
>
> Can I do the same w/Cayenne?
>
> Can I have the columns fetched on-demand?
>
> I'd like to avoid hardcoded SQL templates obviously.
>
> SelectTranslator.createSqlString() unconditionally adds all
> columns...
>
> --
> Øyvind Harboe
> http://www.zylin.com/zy1000.html
> ARM7 ARM9 XScale Cortex
> JTAG debugger and flash programmer
>
Re: Memory usage and select statements
Posted by Mike Kienenberger <mk...@gmail.com>.
On Mon, Jun 30, 2008 at 5:30 AM, Øyvind Harboe <oy...@zylin.com> wrote:
> On Mon, Jun 30, 2008 at 11:01 AM, Andrus Adamchik
> <an...@objectstyle.org> wrote:
>> There is no lazy fetching of attributes (only relationships). We will be
>> adding it at some point. It is a bit more involved than simply suppressing a
>> given column in the SQL translator. Still you have a few of options:
>>
>> 1. Place large columns in a separate table with 1..1 relationship to the
>> master table .
>
> Hmm... will this actually work?
Yes, a separate dependent table is how I've been dong it. I hadn't
realized that #2 with a second entity to the same table was an option.
I'll have to keep this in mind for future work. Thanks!
> 2. Do not map large columns in the ObjEntity and resolve them manually on
> demand (either with SQLTemplate, or by mapping a second entity to the same
> table).
Re: Memory usage and select statements
Posted by Andrey Razumovsky <ra...@gmail.com>.
>
>
> Ah. I can see how the app uses more and more memory then.
>
> I assume the only way the relationship is broken is to gc the top level
> object.
Alternatively, you can use ValueHolder.invalidate() on top-level (reverse)
property. However, I'm not 100% sure that'll work.
BTW, I tried to map a second DbEntity to the same table, but as the
> name of the table is used as a key to the DbEntity that didn't work. This
> would have to be in a seperate DataMap then?
This is about multiple ObjEntities with sole DbEntity. This is possible
Re: Memory usage and select statements
Posted by Øyvind Harboe <oy...@zylin.com>.
On Mon, Jun 30, 2008 at 11:32 AM, Andrus Adamchik
<an...@objectstyle.org> wrote:
> No weak reference, but all relationships are faulted lazily.
Ah. I can see how the app uses more and more memory then.
I assume the only way the relationship is broken is to gc the top level
object.
BTW, I tried to map a second DbEntity to the same table, but as the
name of the table is used as a key to the DbEntity that didn't work. This
would have to be in a seperate DataMap then?
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
Re: Memory usage and select statements
Posted by Andrus Adamchik <an...@objectstyle.org>.
No weak reference, but all relationships are faulted lazily.
Andrus
On Jun 30, 2008, at 12:30 PM, Øyvind Harboe wrote:
> On Mon, Jun 30, 2008 at 11:01 AM, Andrus Adamchik
> <an...@objectstyle.org> wrote:
>> There is no lazy fetching of attributes (only relationships). We
>> will be
>> adding it at some point. It is a bit more involved than simply
>> suppressing a
>> given column in the SQL translator. Still you have a few of options:
>>
>> 1. Place large columns in a separate table with 1..1 relationship
>> to the
>> master table .
>
> Hmm... will this actually work?
>
> Is there a weak reference between the two?
>
>
>
> --
> Øyvind Harboe
> http://www.zylin.com/zy1000.html
> ARM7 ARM9 XScale Cortex
> JTAG debugger and flash programmer
>
Re: Memory usage and select statements
Posted by Øyvind Harboe <oy...@zylin.com>.
On Mon, Jun 30, 2008 at 11:01 AM, Andrus Adamchik
<an...@objectstyle.org> wrote:
> There is no lazy fetching of attributes (only relationships). We will be
> adding it at some point. It is a bit more involved than simply suppressing a
> given column in the SQL translator. Still you have a few of options:
>
> 1. Place large columns in a separate table with 1..1 relationship to the
> master table .
Hmm... will this actually work?
Is there a weak reference between the two?
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
Re: Memory usage and select statements
Posted by Øyvind Harboe <oy...@zylin.com>.
On Mon, Jun 30, 2008 at 11:01 AM, Andrus Adamchik
<an...@objectstyle.org> wrote:
> There is no lazy fetching of attributes (only relationships). We will be
> adding it at some point. It is a bit more involved than simply suppressing a
> given column in the SQL translator. Still you have a few of options:
>
> 1. Place large columns in a separate table with 1..1 relationship to the
> master table .
> 2. Do not map large columns in the ObjEntity and resolve them manually on
> demand (either with SQLTemplate, or by mapping a second entity to the same
> table).
> ...
Thanks!
Since I, generally, have no influence over the database table format, then #2
is the way to go. In my case create a "FastFoo" mapping for "Foo" and
otherwise leave "Foo"
unchanged in the code where performance is not an issue + extract an interface
which is common between FastFoo and Foo.
It didn't occur to me that it was supported to have multiple mappings
to the same database
table.
--
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer
Re: Memory usage and select statements
Posted by Andrus Adamchik <an...@objectstyle.org>.
There is no lazy fetching of attributes (only relationships). We will
be adding it at some point. It is a bit more involved than simply
suppressing a given column in the SQL translator. Still you have a few
of options:
1. Place large columns in a separate table with 1..1 relationship to
the master table .
2. Do not map large columns in the ObjEntity and resolve them manually
on demand (either with SQLTemplate, or by mapping a second entity to
the same table).
...
Andrus
On Jun 30, 2008, at 11:54 AM, Øyvind Harboe wrote:
> An SQL statement includes a list of which columns to fetch for
> the query.
>
> Consider a table "book" with three columns, id(pk), title(<100 chars)
> and content(the entire text of the book).
>
> SELECT id, title FROM book
>
> If one of the columns are big and rarely used, then it makes little
> sense to transfer it to the client from the database for every
> query. E.g. above the entire text of the book is left out.
>
> The difference in memory usage can be quite dramatic.
>
> Can I do the same w/Cayenne?
>
> Can I have the columns fetched on-demand?
>
> I'd like to avoid hardcoded SQL templates obviously.
>
> SelectTranslator.createSqlString() unconditionally adds all
> columns...
>
> --
> Øyvind Harboe
> http://www.zylin.com/zy1000.html
> ARM7 ARM9 XScale Cortex
> JTAG debugger and flash programmer
>