You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Jean-Paul Le Fèvre <je...@cea.fr> on 2007/07/23 15:29:35 UTC
Out of memory exception when creating a large number of objects
Hi,
I'm trying to import a pretty big amount of data into my database.
The input is a xml formatted file. It describes more than 10 millions
of objects each having tens of attributes. The application parses the input
file, creates the cayenne objects and commits the changes if requested.
As you can imagining I'm facing difficulties trying to avoid out of memory
errors. Unfortunately, at this point, I'm still unable to load my big
input file.
To figure out what it's happening I'm monitoring the application behavior
with jconsole. My tactic is the following : every 10000 objects (this number
is a parameter) I call rollbackChanges() or commitChanges().
When I run the program in rollback mode It turns out that the memory used
oscillates between a min and a max value as expected : after each rollback
the garbage collector feels free to cleanup the memory.
But in commit mode the amount of memory keeps on increasing and the
application fails eventually.
I've tried to call unregisterNewObjects() and startTrackingNewObjects() after
the commit :
ctxt.commitChanges();
ctxt.getObjectStore().unregisterNewObjects();
ctxt.getEntityResolver().clearCache();
ctxt.getObjectStore().startTrackingNewObjects();
but it didn't help. It seems that cayenne keeps reference of newly created
objects somewhere preventing the gc from working.
Would you have an idea how to fix the problem ?
Thanks,
--
___________________________________________________________________
Jean-Paul Le Fèvre * Mail : LeFevre@fonteny.org
Re: Out of memory exception when creating a large number of objects
Posted by Andrus Adamchik <an...@objectstyle.org>.
On a side note, Cayenne 3.0 has a leak-free DataContext context:
http://cayenne.apache.org/doc/objectcontext-memory-management.html
Andrus
On Jul 23, 2007, at 4:44 PM, Aristedes Maniatis wrote:
>
> On 23/07/2007, at 11:29 PM, Jean-Paul Le Fèvre wrote:
>
>> Hi,
>>
>> I'm trying to import a pretty big amount of data into my database.
>> The input is a xml formatted file. It describes more than 10 millions
>> of objects each having tens of attributes. The application parses
>> the input
>> file, creates the cayenne objects and commits the changes if
>> requested.
>
> We've just written something quite similar. We tested it with
> something like 500,000 objects across several tables.
>
>> As you can imagining I'm facing difficulties trying to avoid out
>> of memory
>> errors. Unfortunately, at this point, I'm still unable to load my big
>> input file.
>
> As a starting point I hope you are using a SAX parser for your XML.
>
>>
>> To figure out what it's happening I'm monitoring the application
>> behavior
>> with jconsole. My tactic is the following : every 10000 objects
>> (this number
>> is a parameter) I call rollbackChanges() or commitChanges().
>
> We committed each object individually (or sometimes a logical group
> of objects). That way we could have very fine control and
> validation errors only caused the loss of that particular record
> (or small group). I don't know that it helps to commit in batches
> like this.
>
>>
>> When I run the program in rollback mode It turns out that the
>> memory used
>> oscillates between a min and a max value as expected : after each
>> rollback
>> the garbage collector feels free to cleanup the memory.
>>
>> But in commit mode the amount of memory keeps on increasing and the
>> application fails eventually.
>
> Probably because the context continues to fill up with the objects
> you are committing. They aren't discarded. Try creating a new
> context (and discarding the old for the gc to clean up) after every
> couple thousand records.
>
> In our situation we were able to get away with 256Mb RAM on the
> client (we are running this as ROP) and 512Mb RAM on the server
> (most of which appears to be used by Derby).
>
> Ari
>
>
>
>
>
> -------------------------->
> Aristedes Maniatis
> phone +61 2 9660 9700
> PGP fingerprint 08 57 20 4B 80 69 59 E2 A9 BF 2D 48 C2 20 0C C8
>
>
Re: Out of memory exception when creating a large number of objects
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 23/07/2007, at 11:29 PM, Jean-Paul Le Fèvre wrote:
> Hi,
>
> I'm trying to import a pretty big amount of data into my database.
> The input is a xml formatted file. It describes more than 10 millions
> of objects each having tens of attributes. The application parses
> the input
> file, creates the cayenne objects and commits the changes if
> requested.
We've just written something quite similar. We tested it with
something like 500,000 objects across several tables.
> As you can imagining I'm facing difficulties trying to avoid out of
> memory
> errors. Unfortunately, at this point, I'm still unable to load my big
> input file.
As a starting point I hope you are using a SAX parser for your XML.
>
> To figure out what it's happening I'm monitoring the application
> behavior
> with jconsole. My tactic is the following : every 10000 objects
> (this number
> is a parameter) I call rollbackChanges() or commitChanges().
We committed each object individually (or sometimes a logical group
of objects). That way we could have very fine control and validation
errors only caused the loss of that particular record (or small
group). I don't know that it helps to commit in batches like this.
>
> When I run the program in rollback mode It turns out that the
> memory used
> oscillates between a min and a max value as expected : after each
> rollback
> the garbage collector feels free to cleanup the memory.
>
> But in commit mode the amount of memory keeps on increasing and the
> application fails eventually.
Probably because the context continues to fill up with the objects
you are committing. They aren't discarded. Try creating a new context
(and discarding the old for the gc to clean up) after every couple
thousand records.
In our situation we were able to get away with 256Mb RAM on the
client (we are running this as ROP) and 512Mb RAM on the server (most
of which appears to be used by Derby).
Ari
-------------------------->
Aristedes Maniatis
phone +61 2 9660 9700
PGP fingerprint 08 57 20 4B 80 69 59 E2 A9 BF 2D 48 C2 20 0C C8
Re: AW: AW: Out of memory exception when creating a large number of objects
Posted by Andrus Adamchik <an...@objectstyle.org>.
>> The following methods :
>> ctxt.getObjectStore().unregisterNewObjects();
>> ctxt.getObjectStore().startTrackingNewObjects();
>> are meant to avoid the problem but it turns out that they likely
>> forget
>> references in some unknown places.
Not that it matters much with new 3.0 stuff, but the API above used
to work just fine. Anyways, as was mentioned elsewhere, it is much
easier to just create a new DataContext instance after a few thousand
iterations.
Andrus
Re: AW: AW: Out of memory exception when creating a large number of objects
Posted by Andrus Adamchik <an...@objectstyle.org>.
This wasn't a bug, rather the way DataContext was originally designed
(it was assumed that it won't have a life-span longer than say a web
session). As I mentioned in the other message, we stopped making this
assumption and object retaining policy was redesigned in 3.0.
A bit OT for this thread, among other things this new behavior means
that a single DataContext can be shared by multiple threads for read-
only database access. Great for many high-performance session-less apps.
Andrus
On Jul 23, 2007, at 5:16 PM, Peter Schröder wrote:
> afaik that this was a bug related to weak-references
>
> -----Ursprüngliche Nachricht-----
> Von: Jean-Paul Le Fèvre [mailto:jean-paul.lefevre@cea.fr]
> Gesendet: Montag, 23. Juli 2007 16:05
> An: user@cayenne.apache.org
> Betreff: Re: AW: Out of memory exception when creating a large
> number of objects
>
> On Monday 23 July 2007 15:53, Peter Schröder wrote:
>> i think that this is a dataContext-issue. every commited object
>> stays in
>> the context, so you probably should create a new dataContext at
>> some point.
>
> Sure ! but is it a bug or a feature ?
>
> The following methods :
> ctxt.getObjectStore().unregisterNewObjects();
> ctxt.getObjectStore().startTrackingNewObjects();
> are meant to avoid the problem but it turns out that they likely
> forget
> references in some unknown places.
>
> --
> ___________________________________________________________________
>
> Jean-Paul Le Fèvre * Mail : LeFevre@fonteny.org
>
>
AW: AW: Out of memory exception when creating a large number of objects
Posted by Peter Schröder <Pe...@freenet-ag.de>.
afaik that this was a bug related to weak-references
-----Ursprüngliche Nachricht-----
Von: Jean-Paul Le Fèvre [mailto:jean-paul.lefevre@cea.fr]
Gesendet: Montag, 23. Juli 2007 16:05
An: user@cayenne.apache.org
Betreff: Re: AW: Out of memory exception when creating a large number of objects
On Monday 23 July 2007 15:53, Peter Schröder wrote:
> i think that this is a dataContext-issue. every commited object stays in
> the context, so you probably should create a new dataContext at some point.
Sure ! but is it a bug or a feature ?
The following methods :
ctxt.getObjectStore().unregisterNewObjects();
ctxt.getObjectStore().startTrackingNewObjects();
are meant to avoid the problem but it turns out that they likely forget
references in some unknown places.
--
___________________________________________________________________
Jean-Paul Le Fèvre * Mail : LeFevre@fonteny.org
Re: AW: Out of memory exception when creating a large number of objects
Posted by Jean-Paul Le Fèvre <je...@cea.fr>.
On Monday 23 July 2007 15:53, Peter Schröder wrote:
> i think that this is a dataContext-issue. every commited object stays in
> the context, so you probably should create a new dataContext at some point.
Sure ! but is it a bug or a feature ?
The following methods :
ctxt.getObjectStore().unregisterNewObjects();
ctxt.getObjectStore().startTrackingNewObjects();
are meant to avoid the problem but it turns out that they likely forget
references in some unknown places.
--
___________________________________________________________________
Jean-Paul Le Fèvre * Mail : LeFevre@fonteny.org
AW: Out of memory exception when creating a large number of objects
Posted by Peter Schröder <Pe...@freenet-ag.de>.
i think that this is a dataContext-issue. every commited object stays in the context, so you probably should create a new dataContext at some point.
-----Ursprüngliche Nachricht-----
Von: Jean-Paul Le Fèvre [mailto:jean-paul.lefevre@cea.fr]
Gesendet: Montag, 23. Juli 2007 15:30
An: user@cayenne.apache.org
Betreff: Out of memory exception when creating a large number of objects
Hi,
I'm trying to import a pretty big amount of data into my database.
The input is a xml formatted file. It describes more than 10 millions
of objects each having tens of attributes. The application parses the input
file, creates the cayenne objects and commits the changes if requested.
As you can imagining I'm facing difficulties trying to avoid out of memory
errors. Unfortunately, at this point, I'm still unable to load my big
input file.
To figure out what it's happening I'm monitoring the application behavior
with jconsole. My tactic is the following : every 10000 objects (this number
is a parameter) I call rollbackChanges() or commitChanges().
When I run the program in rollback mode It turns out that the memory used
oscillates between a min and a max value as expected : after each rollback
the garbage collector feels free to cleanup the memory.
But in commit mode the amount of memory keeps on increasing and the
application fails eventually.
I've tried to call unregisterNewObjects() and startTrackingNewObjects() after
the commit :
ctxt.commitChanges();
ctxt.getObjectStore().unregisterNewObjects();
ctxt.getEntityResolver().clearCache();
ctxt.getObjectStore().startTrackingNewObjects();
but it didn't help. It seems that cayenne keeps reference of newly created
objects somewhere preventing the gc from working.
Would you have an idea how to fix the problem ?
Thanks,
--
___________________________________________________________________
Jean-Paul Le Fèvre * Mail : LeFevre@fonteny.org