You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by "mr.abanjo" <mr...@gmail.com> on 2009/03/18 12:14:33 UTC
Extract large data from database
Hi,i'm developing an application that must load a large ammount of data
(100.000 records) and stream them out from a servlet.
I created a "for" cycle that for every step load 200 record from the
database (using datacontext.performquery method), and then flush the data in
the serlvet output stream.
In this way i suppose that the max ammount of memory allocated by cayenne is
:
object size * 200
but seems that this is not true. When i try to get the data from the
servlet, after few seconds, i receive an "out of memory" error. (my 500Mb
allocated for the Heap are reached in a few time... :-(....)
Which is the best practice in this case?
I can increase the heap, but i want to find a workaround that allow me to
have a small heap allocated during this operation.
I found this article:
http://cayenne.apache.org/doc20/iterating-through-data-rows.html
Is this the right way?
The servlet i'm implementig will be called by a different webapp
(backoffice.. in a server 2 server way ) with a low traffic but i'm warried
about the warning reported in the article:
"In web applications, programmers must ensure that no open ResultIterators
are kept between HTTP requests. Failure to do so may result in too many
database connections being locked, thus quickly exhausting connection pool.
In general, an application with Web GUI is NOT a good candidate for
implementation using ResultIterators."
Thanks
Davide
Re: Extract large data from database
Posted by Michael Gentry <mg...@masslight.net>.
Are you doing the queries in the same DataContext? If so, try
discarding the old DC and create a new one. (DC creation is cheap.)
You could also try calling DC's invalidateObjects(Collection objects),
but discarding the DC is probably easier and more efficient.
mrg
On Wed, Mar 18, 2009 at 7:14 AM, mr.abanjo <mr...@gmail.com> wrote:
> Hi,i'm developing an application that must load a large ammount of data
> (100.000 records) and stream them out from a servlet.
>
> I created a "for" cycle that for every step load 200 record from the
> database (using datacontext.performquery method), and then flush the data in
> the serlvet output stream.
> In this way i suppose that the max ammount of memory allocated by cayenne is
> :
>
> object size * 200
>
> but seems that this is not true. When i try to get the data from the
> servlet, after few seconds, i receive an "out of memory" error. (my 500Mb
> allocated for the Heap are reached in a few time... :-(....)
>
> Which is the best practice in this case?
> I can increase the heap, but i want to find a workaround that allow me to
> have a small heap allocated during this operation.
>
> I found this article:
> http://cayenne.apache.org/doc20/iterating-through-data-rows.html
> Is this the right way?
>
> The servlet i'm implementig will be called by a different webapp
> (backoffice.. in a server 2 server way ) with a low traffic but i'm warried
> about the warning reported in the article:
>
> "In web applications, programmers must ensure that no open ResultIterators
> are kept between HTTP requests. Failure to do so may result in too many
> database connections being locked, thus quickly exhausting connection pool.
> In general, an application with Web GUI is NOT a good candidate for
> implementation using ResultIterators."
>
> Thanks
> Davide
>
Re: Extract large data from database
Posted by "mr.abanjo" <mr...@gmail.com>.
HI all,thanks for your answers!
Actually i'm using Cayenne 2.0. Resultiterator works well! The memory
doesn't grow, and the data are retrived in a resonable time (100.000 record,
60Mb in 50/60 seconds... this is the worst case).
So i think that this is a good solution that also allow me to maintain the
same logic i actually use for manage datacontext.
I was sure that Cayenne can solve my problem. Thanks for this great product!
;-)
Davide
On Wed, Mar 18, 2009 at 3:52 PM, Andrus Adamchik <an...@objectstyle.org>wrote:
> Actually using a ResultIterator should be entirely safe in 3.0 in respect
> to memory management. If a user doesn't do something bad (like storing
> obtained objects in a list himself), and only one object is processed at a
> time, there won't be any memory issues.
>
> The servlet i'm implementig will be called by a different webapp
>>> (backoffice.. in a server 2 server way ) with a low traffic but i'm
>>> warried
>>> about the warning reported in the article:
>>>
>>> "In web applications, programmers must ensure that no open
>>> ResultIterators
>>> are kept between HTTP requests. Failure to do so may result in too many
>>> database connections being locked, thus quickly exhausting connection
>>> pool.
>>>
>>
> This one is still valid. Keep ResultIterator request-scoped, and close it
> at the end.
>
> In general, an application with Web GUI is NOT a good candidate for
>>> implementation using ResultIterators."
>>>
>>
> Well, there are always exceptions... The general warning here is based on
> the assumption that large results are usually taking lots of time to fetch
> and process. So users may not see the response fast enough. If the response
> time is reasonable in your case, you can use ResultIterator.
>
> Andrus
>
>
> On Mar 18, 2009, at 4:17 PM, Robert Zeigler wrote:
>
> Problem is that the data context will cache the objects (up to the max
>> object limit); this is alleviated somewhat in 3.0 with the use of weak
>> references
>> in the caching, but there's still potential for running out of memory.
>>
>> You can certainly use ResultIterator, with the caveat that you'll be
>> getting DataRows instead of DataObjects.
>> Alternatively, periodically discard the context that you're using and
>> replace it with a fresh context, something like:
>>
>> Query query = createMyQuery();//SQLTemplate, or SelectQuery, or
>> EJBQLQuery, or whatever.
>> while(notDone) {
>> ObjectContext context = ...;//create a new object context here.
>> //do things like set the query fetch limit and fetch offset.
>> List objs = context.performQuery(query);
>> for(Obj o : objs) {
>> //stream out the obj...
>> }
>> }
>>
>> Robert
>>
>> On Mar 18, 2009, at 3/186:14 AM , mr.abanjo wrote:
>>
>> Hi,i'm developing an application that must load a large ammount of data
>>> (100.000 records) and stream them out from a servlet.
>>>
>>> I created a "for" cycle that for every step load 200 record from the
>>> database (using datacontext.performquery method), and then flush the data
>>> in
>>> the serlvet output stream.
>>> In this way i suppose that the max ammount of memory allocated by cayenne
>>> is
>>> :
>>>
>>> object size * 200
>>>
>>> but seems that this is not true. When i try to get the data from the
>>> servlet, after few seconds, i receive an "out of memory" error. (my 500Mb
>>> allocated for the Heap are reached in a few time... :-(....)
>>>
>>> Which is the best practice in this case?
>>> I can increase the heap, but i want to find a workaround that allow me to
>>> have a small heap allocated during this operation.
>>>
>>> I found this article:
>>> http://cayenne.apache.org/doc20/iterating-through-data-rows.html
>>> Is this the right way?
>>>
>>> The servlet i'm implementig will be called by a different webapp
>>> (backoffice.. in a server 2 server way ) with a low traffic but i'm
>>> warried
>>> about the warning reported in the article:
>>>
>>> "In web applications, programmers must ensure that no open
>>> ResultIterators
>>> are kept between HTTP requests. Failure to do so may result in too many
>>> database connections being locked, thus quickly exhausting connection
>>> pool.
>>> In general, an application with Web GUI is NOT a good candidate for
>>> implementation using ResultIterators."
>>>
>>> Thanks
>>> Davide
>>>
>>
>>
>>
>
Re: Extract large data from database
Posted by Andrus Adamchik <an...@objectstyle.org>.
Actually using a ResultIterator should be entirely safe in 3.0 in
respect to memory management. If a user doesn't do something bad (like
storing obtained objects in a list himself), and only one object is
processed at a time, there won't be any memory issues.
>> The servlet i'm implementig will be called by a different webapp
>> (backoffice.. in a server 2 server way ) with a low traffic but i'm
>> warried
>> about the warning reported in the article:
>>
>> "In web applications, programmers must ensure that no open
>> ResultIterators
>> are kept between HTTP requests. Failure to do so may result in too
>> many
>> database connections being locked, thus quickly exhausting
>> connection pool.
This one is still valid. Keep ResultIterator request-scoped, and close
it at the end.
>> In general, an application with Web GUI is NOT a good candidate for
>> implementation using ResultIterators."
Well, there are always exceptions... The general warning here is based
on the assumption that large results are usually taking lots of time
to fetch and process. So users may not see the response fast enough.
If the response time is reasonable in your case, you can use
ResultIterator.
Andrus
On Mar 18, 2009, at 4:17 PM, Robert Zeigler wrote:
> Problem is that the data context will cache the objects (up to the
> max object limit); this is alleviated somewhat in 3.0 with the use
> of weak references
> in the caching, but there's still potential for running out of memory.
>
> You can certainly use ResultIterator, with the caveat that you'll be
> getting DataRows instead of DataObjects.
> Alternatively, periodically discard the context that you're using
> and replace it with a fresh context, something like:
>
> Query query = createMyQuery();//SQLTemplate, or SelectQuery, or
> EJBQLQuery, or whatever.
> while(notDone) {
> ObjectContext context = ...;//create a new object context here.
> //do things like set the query fetch limit and fetch offset.
> List objs = context.performQuery(query);
> for(Obj o : objs) {
> //stream out the obj...
> }
> }
>
> Robert
>
> On Mar 18, 2009, at 3/186:14 AM , mr.abanjo wrote:
>
>> Hi,i'm developing an application that must load a large ammount of
>> data
>> (100.000 records) and stream them out from a servlet.
>>
>> I created a "for" cycle that for every step load 200 record from the
>> database (using datacontext.performquery method), and then flush
>> the data in
>> the serlvet output stream.
>> In this way i suppose that the max ammount of memory allocated by
>> cayenne is
>> :
>>
>> object size * 200
>>
>> but seems that this is not true. When i try to get the data from the
>> servlet, after few seconds, i receive an "out of memory" error. (my
>> 500Mb
>> allocated for the Heap are reached in a few time... :-(....)
>>
>> Which is the best practice in this case?
>> I can increase the heap, but i want to find a workaround that allow
>> me to
>> have a small heap allocated during this operation.
>>
>> I found this article:
>> http://cayenne.apache.org/doc20/iterating-through-data-rows.html
>> Is this the right way?
>>
>> The servlet i'm implementig will be called by a different webapp
>> (backoffice.. in a server 2 server way ) with a low traffic but i'm
>> warried
>> about the warning reported in the article:
>>
>> "In web applications, programmers must ensure that no open
>> ResultIterators
>> are kept between HTTP requests. Failure to do so may result in too
>> many
>> database connections being locked, thus quickly exhausting
>> connection pool.
>> In general, an application with Web GUI is NOT a good candidate for
>> implementation using ResultIterators."
>>
>> Thanks
>> Davide
>
>
Re: Extract large data from database
Posted by Robert Zeigler <ro...@gmail.com>.
Problem is that the data context will cache the objects (up to the max
object limit); this is alleviated somewhat in 3.0 with the use of weak
references
in the caching, but there's still potential for running out of memory.
You can certainly use ResultIterator, with the caveat that you'll be
getting DataRows instead of DataObjects.
Alternatively, periodically discard the context that you're using and
replace it with a fresh context, something like:
Query query = createMyQuery();//SQLTemplate, or SelectQuery, or
EJBQLQuery, or whatever.
while(notDone) {
ObjectContext context = ...;//create a new object context here.
//do things like set the query fetch limit and fetch offset.
List objs = context.performQuery(query);
for(Obj o : objs) {
//stream out the obj...
}
}
Robert
On Mar 18, 2009, at 3/186:14 AM , mr.abanjo wrote:
> Hi,i'm developing an application that must load a large ammount of
> data
> (100.000 records) and stream them out from a servlet.
>
> I created a "for" cycle that for every step load 200 record from the
> database (using datacontext.performquery method), and then flush the
> data in
> the serlvet output stream.
> In this way i suppose that the max ammount of memory allocated by
> cayenne is
> :
>
> object size * 200
>
> but seems that this is not true. When i try to get the data from the
> servlet, after few seconds, i receive an "out of memory" error. (my
> 500Mb
> allocated for the Heap are reached in a few time... :-(....)
>
> Which is the best practice in this case?
> I can increase the heap, but i want to find a workaround that allow
> me to
> have a small heap allocated during this operation.
>
> I found this article:
> http://cayenne.apache.org/doc20/iterating-through-data-rows.html
> Is this the right way?
>
> The servlet i'm implementig will be called by a different webapp
> (backoffice.. in a server 2 server way ) with a low traffic but i'm
> warried
> about the warning reported in the article:
>
> "In web applications, programmers must ensure that no open
> ResultIterators
> are kept between HTTP requests. Failure to do so may result in too
> many
> database connections being locked, thus quickly exhausting
> connection pool.
> In general, an application with Web GUI is NOT a good candidate for
> implementation using ResultIterators."
>
> Thanks
> Davide