You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Steve Wells <we...@gmail.com> on 2010/05/24 08:58:45 UTC
OutOfMemory with large number of inserts
Hi All,
I am importing from a lot of csv files (2200+) and each csv has between 0
and 3000 rows, each row will create a new DataObject in a DataContext etc :
The code is simple enough and distilled here is:
for each csv file
RelatedObj rel = getRelatedObj()
for each line in csv file
Line l = new Line()
l.setxxx for each column of csv // 5 cols are set
l.setToRelatedObject(rel)
dataContext.registerNewObject(l)
end for each line in csv file
dataContext.commitChanges()
end
This goes well for a while but no matter how high I set the heap size I
eventually get OutOfMemoryException....I can see the speed of each commit
slowdown over time as the heap size is diminshed.
I have tried a few different ways around this but all end up the same.
Initially tuning the SQL server instance, but everything points to objects
in memory not being de-allocated.
Has anyone had experience with Cayenne (3.0RC3) and this number of inserts
they could advise with?
Cheers,
Steve
Re: OutOfMemory with large number of inserts
Posted by Andrus Adamchik <an...@objectstyle.org>.
Actually you can do batch processing with Cayenne, as DataContext uses
weak references to committed objects. So doing commit after each X
inserted objects should release the memory. However "rel" object
reference may prevent that from happening if it is the same object for
all inserted objects. For one thing make sure there's no reverse
relationship from rel to l's.
Andrus
On May 24, 2010, at 10:09 AM, Andrey Razumovsky wrote:
> ORM technologies require additional memory than just plain JDBC.
> Million
> records you read will lead to creating million data objects and this
> is not
> a case for ORM, because huge size of memory is needed. Moreover,
> you'll see
> major slowdowns during the commit. So the actual advice is to use
> plain SQL
> (or say Cayenne's SQLTemplate) for such great inserts
>
> 2010/5/24 Steve Wells <we...@gmail.com>
>
>> Hi All,
>>
>> I am importing from a lot of csv files (2200+) and each csv has
>> between 0
>> and 3000 rows, each row will create a new DataObject in a
>> DataContext etc :
>>
>> The code is simple enough and distilled here is:
>>
>> for each csv file
>> RelatedObj rel = getRelatedObj()
>> for each line in csv file
>> Line l = new Line()
>> l.setxxx for each column of csv // 5 cols are set
>> l.setToRelatedObject(rel)
>> dataContext.registerNewObject(l)
>> end for each line in csv file
>> dataContext.commitChanges()
>> end
>>
>> This goes well for a while but no matter how high I set the heap
>> size I
>> eventually get OutOfMemoryException....I can see the speed of each
>> commit
>> slowdown over time as the heap size is diminshed.
>>
>> I have tried a few different ways around this but all end up the
>> same.
>> Initially tuning the SQL server instance, but everything points to
>> objects
>> in memory not being de-allocated.
>>
>> Has anyone had experience with Cayenne (3.0RC3) and this number of
>> inserts
>> they could advise with?
>>
>> Cheers,
>>
>> Steve
>>
>
>
>
> --
> Andrey
Re: OutOfMemory with large number of inserts
Posted by Andrey Razumovsky <ra...@gmail.com>.
ORM technologies require additional memory than just plain JDBC. Million
records you read will lead to creating million data objects and this is not
a case for ORM, because huge size of memory is needed. Moreover, you'll see
major slowdowns during the commit. So the actual advice is to use plain SQL
(or say Cayenne's SQLTemplate) for such great inserts
2010/5/24 Steve Wells <we...@gmail.com>
> Hi All,
>
> I am importing from a lot of csv files (2200+) and each csv has between 0
> and 3000 rows, each row will create a new DataObject in a DataContext etc :
>
> The code is simple enough and distilled here is:
>
> for each csv file
> RelatedObj rel = getRelatedObj()
> for each line in csv file
> Line l = new Line()
> l.setxxx for each column of csv // 5 cols are set
> l.setToRelatedObject(rel)
> dataContext.registerNewObject(l)
> end for each line in csv file
> dataContext.commitChanges()
> end
>
> This goes well for a while but no matter how high I set the heap size I
> eventually get OutOfMemoryException....I can see the speed of each commit
> slowdown over time as the heap size is diminshed.
>
> I have tried a few different ways around this but all end up the same.
> Initially tuning the SQL server instance, but everything points to objects
> in memory not being de-allocated.
>
> Has anyone had experience with Cayenne (3.0RC3) and this number of inserts
> they could advise with?
>
> Cheers,
>
> Steve
>
--
Andrey
Re: OutOfMemory with large number of inserts
Posted by Michael Gentry <mg...@masslight.net>.
Oh, that'll do it ... :-)
On Tue, May 25, 2010 at 11:58 PM, Steve Wells <we...@gmail.com> wrote:
> The mistake I made was to create a new nested context for each CSV...not
> a new parent context.
>
>
> On 24 May 2010 20:32, Steve Wells <we...@gmail.com> wrote:
>
>> Typical Cayenne users exceedingly fast helpful answers! I've tried a few
>> more things based on your replies but nothing positive so far.
>>
>> Andrey, I am only creating *most* new objects in datacontext. But I am
>> selecting the related (master a.k.a 'rel') data-objects and "connecting"
>> them to the detail objects in the order of 1-2000 (could be 1-1 or 1-3000).
>> one thing I did notice was that I was selecting the 'rel' object using
>> shared cache. I set cache to NONE for that select and all the
>> create/inserts ran approx twice as fast (I do have some basic timings in
>> place). Perhaps this is an interesting separate thread of discussion.
>>
>> Juergen, yes I already had a new ObjectContext for each CSV and I was
>> commiting each 400 (and 200 and 1000 etc). I also created a new
>> ObjectContext at the same point, though I expect moving the 'related' object
>> to the new Context may negate the point of creating the new context...so
>> same result.
>>
>> Andrus, I've deleted the reverse relationship and the same issue
>> persists. What I did do was remove the forward relationship
>> 'l.setToRelatedObject(rel)' and it all worked fine...no OutOfMemory. But
>> I do need both relationship directions. At least I have some direction in
>> which to try.
>>
>> Thanks all for your help.
>>
>> On 24 May 2010 17:21, Juergen Saar <ju...@jsaar.org> wrote:
>>
>>> First try: use a new ObjectContext for each csv-File ...
>>>
>>> If this doesn't solve the memory-problem you should try to make a commit
>>> after let's say 200 inserts and then create a new ObjectContext ...
>>>
>>> 2010/5/24 Steve Wells <we...@gmail.com>
>>>
>>> > Hi All,
>>> >
>>> > I am importing from a lot of csv files (2200+) and each csv has between
>>> 0
>>> > and 3000 rows, each row will create a new DataObject in a DataContext
>>> etc :
>>> >
>>> > The code is simple enough and distilled here is:
>>> >
>>> > for each csv file
>>> > RelatedObj rel = getRelatedObj()
>>> > for each line in csv file
>>> > Line l = new Line()
>>> > l.setxxx for each column of csv // 5 cols are set
>>> > l.setToRelatedObject(rel)
>>> > dataContext.registerNewObject(l)
>>> > end for each line in csv file
>>> > dataContext.commitChanges()
>>> > end
>>> >
>>> > This goes well for a while but no matter how high I set the heap size I
>>> > eventually get OutOfMemoryException....I can see the speed of each
>>> commit
>>> > slowdown over time as the heap size is diminshed.
>>> >
>>> > I have tried a few different ways around this but all end up the same.
>>> > Initially tuning the SQL server instance, but everything points to
>>> objects
>>> > in memory not being de-allocated.
>>> >
>>> > Has anyone had experience with Cayenne (3.0RC3) and this number of
>>> inserts
>>> > they could advise with?
>>> >
>>> > Cheers,
>>> >
>>> > Steve
>>> >
>>>
>>
>>
>
Re: OutOfMemory with large number of inserts
Posted by Steve Wells <we...@gmail.com>.
The mistake I made was to create a new nested context for each CSV...not
a new parent context.
On 24 May 2010 20:32, Steve Wells <we...@gmail.com> wrote:
> Typical Cayenne users exceedingly fast helpful answers! I've tried a few
> more things based on your replies but nothing positive so far.
>
> Andrey, I am only creating *most* new objects in datacontext. But I am
> selecting the related (master a.k.a 'rel') data-objects and "connecting"
> them to the detail objects in the order of 1-2000 (could be 1-1 or 1-3000).
> one thing I did notice was that I was selecting the 'rel' object using
> shared cache. I set cache to NONE for that select and all the
> create/inserts ran approx twice as fast (I do have some basic timings in
> place). Perhaps this is an interesting separate thread of discussion.
>
> Juergen, yes I already had a new ObjectContext for each CSV and I was
> commiting each 400 (and 200 and 1000 etc). I also created a new
> ObjectContext at the same point, though I expect moving the 'related' object
> to the new Context may negate the point of creating the new context...so
> same result.
>
> Andrus, I've deleted the reverse relationship and the same issue
> persists. What I did do was remove the forward relationship
> 'l.setToRelatedObject(rel)' and it all worked fine...no OutOfMemory. But
> I do need both relationship directions. At least I have some direction in
> which to try.
>
> Thanks all for your help.
>
> On 24 May 2010 17:21, Juergen Saar <ju...@jsaar.org> wrote:
>
>> First try: use a new ObjectContext for each csv-File ...
>>
>> If this doesn't solve the memory-problem you should try to make a commit
>> after let's say 200 inserts and then create a new ObjectContext ...
>>
>> 2010/5/24 Steve Wells <we...@gmail.com>
>>
>> > Hi All,
>> >
>> > I am importing from a lot of csv files (2200+) and each csv has between
>> 0
>> > and 3000 rows, each row will create a new DataObject in a DataContext
>> etc :
>> >
>> > The code is simple enough and distilled here is:
>> >
>> > for each csv file
>> > RelatedObj rel = getRelatedObj()
>> > for each line in csv file
>> > Line l = new Line()
>> > l.setxxx for each column of csv // 5 cols are set
>> > l.setToRelatedObject(rel)
>> > dataContext.registerNewObject(l)
>> > end for each line in csv file
>> > dataContext.commitChanges()
>> > end
>> >
>> > This goes well for a while but no matter how high I set the heap size I
>> > eventually get OutOfMemoryException....I can see the speed of each
>> commit
>> > slowdown over time as the heap size is diminshed.
>> >
>> > I have tried a few different ways around this but all end up the same.
>> > Initially tuning the SQL server instance, but everything points to
>> objects
>> > in memory not being de-allocated.
>> >
>> > Has anyone had experience with Cayenne (3.0RC3) and this number of
>> inserts
>> > they could advise with?
>> >
>> > Cheers,
>> >
>> > Steve
>> >
>>
>
>
Re: OutOfMemory with large number of inserts
Posted by Steve Wells <we...@gmail.com>.
Typical Cayenne users exceedingly fast helpful answers! I've tried a few
more things based on your replies but nothing positive so far.
Andrey, I am only creating *most* new objects in datacontext. But I am
selecting the related (master a.k.a 'rel') data-objects and "connecting"
them to the detail objects in the order of 1-2000 (could be 1-1 or 1-3000).
one thing I did notice was that I was selecting the 'rel' object using
shared cache. I set cache to NONE for that select and all the
create/inserts ran approx twice as fast (I do have some basic timings in
place). Perhaps this is an interesting separate thread of discussion.
Juergen, yes I already had a new ObjectContext for each CSV and I was
commiting each 400 (and 200 and 1000 etc). I also created a new
ObjectContext at the same point, though I expect moving the 'related' object
to the new Context may negate the point of creating the new context...so
same result.
Andrus, I've deleted the reverse relationship and the same issue persists.
What I did do was remove the forward relationship
'l.setToRelatedObject(rel)' and it all worked fine...no OutOfMemory. But
I do need both relationship directions. At least I have some direction in
which to try.
Thanks all for your help.
On 24 May 2010 17:21, Juergen Saar <ju...@jsaar.org> wrote:
> First try: use a new ObjectContext for each csv-File ...
>
> If this doesn't solve the memory-problem you should try to make a commit
> after let's say 200 inserts and then create a new ObjectContext ...
>
> 2010/5/24 Steve Wells <we...@gmail.com>
>
> > Hi All,
> >
> > I am importing from a lot of csv files (2200+) and each csv has between 0
> > and 3000 rows, each row will create a new DataObject in a DataContext etc
> :
> >
> > The code is simple enough and distilled here is:
> >
> > for each csv file
> > RelatedObj rel = getRelatedObj()
> > for each line in csv file
> > Line l = new Line()
> > l.setxxx for each column of csv // 5 cols are set
> > l.setToRelatedObject(rel)
> > dataContext.registerNewObject(l)
> > end for each line in csv file
> > dataContext.commitChanges()
> > end
> >
> > This goes well for a while but no matter how high I set the heap size I
> > eventually get OutOfMemoryException....I can see the speed of each commit
> > slowdown over time as the heap size is diminshed.
> >
> > I have tried a few different ways around this but all end up the same.
> > Initially tuning the SQL server instance, but everything points to
> objects
> > in memory not being de-allocated.
> >
> > Has anyone had experience with Cayenne (3.0RC3) and this number of
> inserts
> > they could advise with?
> >
> > Cheers,
> >
> > Steve
> >
>
Re: OutOfMemory with large number of inserts
Posted by Juergen Saar <ju...@jsaar.org>.
First try: use a new ObjectContext for each csv-File ...
If this doesn't solve the memory-problem you should try to make a commit
after let's say 200 inserts and then create a new ObjectContext ...
2010/5/24 Steve Wells <we...@gmail.com>
> Hi All,
>
> I am importing from a lot of csv files (2200+) and each csv has between 0
> and 3000 rows, each row will create a new DataObject in a DataContext etc :
>
> The code is simple enough and distilled here is:
>
> for each csv file
> RelatedObj rel = getRelatedObj()
> for each line in csv file
> Line l = new Line()
> l.setxxx for each column of csv // 5 cols are set
> l.setToRelatedObject(rel)
> dataContext.registerNewObject(l)
> end for each line in csv file
> dataContext.commitChanges()
> end
>
> This goes well for a while but no matter how high I set the heap size I
> eventually get OutOfMemoryException....I can see the speed of each commit
> slowdown over time as the heap size is diminshed.
>
> I have tried a few different ways around this but all end up the same.
> Initially tuning the SQL server instance, but everything points to objects
> in memory not being de-allocated.
>
> Has anyone had experience with Cayenne (3.0RC3) and this number of inserts
> they could advise with?
>
> Cheers,
>
> Steve
>