Memory and performance many new objects

[Andreas Hartmann <an...@apache.org>]

Hi everyone,

I'm facing the following situation:

I'm importing arbitrary spreadsheets with quite large numbers of rows. A 
row represents a recipient of a mailing. The spreadsheet can contain 
arbitrary columns, so I chose the following entity model:

* RecipientSet
  -> m Fields (i.e. spreadsheet column titles)
  -> n Recipients
   -> m Values (one for each Field)

Given a spreadsheet with 20 columns and 25.000 rows, this results in 
about 25.000 x 21 DataObjects in the context. The application consumes 
quite a lot of money, a profiler run showed that a large chunk of the 
memory is needed for Cayenne's difference objects. And committing the 
object context changes can take a couple of minutes.

Is there a best practise to approach this situation?

I considered to use a different entity model, e.g. use a generic table 
with columns "col_1" to "col_100" so that I wouldn't need extra objects 
for the values. But this feels like a bad workaround.

Of course I could commit the transaction each 1000 rows or so, but I'd 
rather commit the whole spreadsheet to the DB in a single transaction.

Thanks a lot for any hints!

-- Andreas


-- 
Andreas Hartmann, CTO
BeCompany GmbH
http://www.becompany.ch
Tel.: +41 (0) 43 818 57 01

Re: Memory and performance many new objects

[Andreas Hartmann <an...@apache.org>]

Hi Andrus,

Andrus Adamchik schrieb:
> 
> On Aug 13, 2009, at 12:36 PM, Andreas Hartmann wrote:
> 
>>
>> Of course I could commit the transaction each 1000 rows or so, but I'd 
>> rather commit the whole spreadsheet to the DB in a single transaction.
> 
> You can use user-defined transaction scope, then committing every 1000 
> rows will allow Java side garbage collection on them, while the entire 
> spreadsheet will load atomically (unless of course your DB does not 
> support ACID transactions, i.e. MySQL MyISAM and such) :
> 
> http://cayenne.apache.org/doc/understanding-transactions.html

thanks a lot, that works like a charm!

> Also I should note that if you are not on Cayenne 3.0 yet, that has 
> self-cleaning DataContexts, you will also need to replace a DataContext 
> after each commit.

I'm using 3.0M6, but thanks for mentioning this.

-- Andreas


-- 
Andreas Hartmann, CTO
BeCompany GmbH
http://www.becompany.ch
Tel.: +41 (0) 43 818 57 01

Re: Memory and performance many new objects

[Andrus Adamchik <an...@objectstyle.org>]

On Aug 13, 2009, at 12:36 PM, Andreas Hartmann wrote:

>
> Of course I could commit the transaction each 1000 rows or so, but  
> I'd rather commit the whole spreadsheet to the DB in a single  
> transaction.

You can use user-defined transaction scope, then committing every 1000  
rows will allow Java side garbage collection on them, while the entire  
spreadsheet will load atomically (unless of course your DB does not  
support ACID transactions, i.e. MySQL MyISAM and such) :

http://cayenne.apache.org/doc/understanding-transactions.html

Also I should note that if you are not on Cayenne 3.0 yet, that has  
self-cleaning DataContexts, you will also need to replace a  
DataContext after each commit.

Andrus