You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by Paul Joseph <pj...@gmail.com> on 2011/10/24 17:15:58 UTC
thought I had fixe it
Hi there,
I thought I had fixed this memory issue but...
I am using Tomcat 6 in a Windows 32 bit environment (Windows 2003) with
Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
Postgres 9.1, JDBC 4 driver.
The behavior is this:
The user fires of a LARGE query that returns 25,000 large objects.
The user repeats this 20 times within 5 minutes to show me he can freeze
the app.
On the 20th time, it says out of memory (heap space).
The JVM indicates that it is maxed to the limit specified in Xms and
that there is only about 2MB of memory free.
I then ask her to log off.
The session time out is set to 20 minutes.
But even after an hour, the memory is not reclaimed by the JVM--it still
reports that only about 2MB is still free.
Is the fact that it is not reclaiming memory an indication of a memory leak?
I am using the following settings in my repository.database:
<jdbc-connection-descriptor
jcd-alias="WebApp"
default-connection="true"
platform="PostgreSQL"
jdbc-level="4.0"
driver="org.postgresql.Driver"
protocol="jdbc"
subprotocol="postgresql"
dbalias="//localhost:5432/WebApp"
username="******"
password="******"
eager-release="false"
batch-mode="false">
<connection-pool maxActive="200" validationQuery="" />
<sequence-manager
className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
/>
</jdbc-connection-descriptor>
Thanks much!
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: thought I had fixe it
Posted by Robby Pelssers <Ro...@nxp.com>.
Hi Paul,
You would definitely come to that conclusion. But maybe it's worth taking a look at your own code that gets executed to render that single page. Do you see anything fishy? If not did you try any java profiler to find out which class is responsible for the memory consumption? It would certainly be a step towards tracing the leak.
Robby
-----Original Message-----
From: Paul Joseph [mailto:pjoseph@gmail.com]
Sent: Monday, October 24, 2011 5:16 PM
To: users@cocoon.apache.org
Subject: thought I had fixe it
Hi there,
I thought I had fixed this memory issue but...
I am using Tomcat 6 in a Windows 32 bit environment (Windows 2003) with
Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
Postgres 9.1, JDBC 4 driver.
The behavior is this:
The user fires of a LARGE query that returns 25,000 large objects.
The user repeats this 20 times within 5 minutes to show me he can freeze
the app.
On the 20th time, it says out of memory (heap space).
The JVM indicates that it is maxed to the limit specified in Xms and
that there is only about 2MB of memory free.
I then ask her to log off.
The session time out is set to 20 minutes.
But even after an hour, the memory is not reclaimed by the JVM--it still
reports that only about 2MB is still free.
Is the fact that it is not reclaiming memory an indication of a memory leak?
I am using the following settings in my repository.database:
<jdbc-connection-descriptor
jcd-alias="WebApp"
default-connection="true"
platform="PostgreSQL"
jdbc-level="4.0"
driver="org.postgresql.Driver"
protocol="jdbc"
subprotocol="postgresql"
dbalias="//localhost:5432/WebApp"
username="******"
password="******"
eager-release="false"
batch-mode="false">
<connection-pool maxActive="200" validationQuery="" />
<sequence-manager
className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
/>
</jdbc-connection-descriptor>
Thanks much!
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
Hi there,
In connection with this issue where the user does a query that returns a
large number of results...as Jasha points out below the correct thing is
to implement paging logic that returns results only as demanded.
Is there anything already done in Cocoon OJB that uses this approach and
which I can leverage?
Paul
On 10/24/2011 12:22 PM, Jasha Joachimsthal wrote:
> The first question that comes to my mind: why would you need 25000
> objects as query result? Can't you do 1 query that only returns the
> total possible amount of objects and another query that returns e.g.
> the first 10 results and build in some paging logic for the next 10.
>
> Jasha Joachimsthal
>
> Europe - Amsterdam - Oosteinde 11, 1017 WT Amsterdam - +31(0)20 522 4466
> US - Boston - 1 Broadway, Cambridge, MA 02142 - +1 877 414 4776 (toll
> free)
>
> www.onehippo.com <http://www.onehippo.com/>
>
>
> On 24 October 2011 18:13, Paul Joseph <pjoseph@gmail.com
> <ma...@gmail.com>> wrote:
>
> I think you hit the nail on the head.
>
> I assumed that the memory is freed at session time out. I did not
> realize this was not the case.
>
> I know what the culprit is in this case--the array that I use to
> store the objects returned from the query.
>
> I do this in two places--once in the business logic (java code)
> and once in the flow script.
>
> The code in the java business logic is as follows--the item of
> interest is a "Task". How would I optimize it in terms of memory
> collection?
>
> public Object[] getSearchResults() throws Exception {
>
> Object[] results = getSearchResultBeans(sql_query);
>
> Vector secureResults = new Vector();
>
> for (int n=0; n < results.length; n++) {
>
> Task t = (Task)results[n];
>
> if (securityManager.canIViewTask(t)) secureResults.add(t);
>
> }
>
> return secureResults.toArray();
>
> }
>
> On 10/24/2011 11:50 AM, Nathaniel, Alfred wrote:
>
> Hi Paul,
>
> I don't think that it is a database issue.
> It is rather the question where the application places the
> large amounts of data and how it is cleaned up.
>
> I interprete your statements that it is placed in the session
> object assuming that the memory is freed at session timeout.
> Unfortunately that is not the case.
>
> The session timeout is a security feature to force a new login
> if the same user comes back after a longish idle time.
> There is no guarantee that the container will actually delete
> the session object at the session timeout.
> As long as there is a reference to the session object GC
> cannot free the attached memory.
>
> You will have to find a way that the application keeps the
> data only for the duration of a request, or use another
> mechanism to limit the memory requirements.
>
> HTH, Alfred.
>
> -----Original Message-----
> From: Paul Joseph [mailto:pjoseph@gmail.com
> <ma...@gmail.com>]
> Sent: Montag, 24. Oktober 2011 17:16
> To: users@cocoon.apache.org <ma...@cocoon.apache.org>
> Subject: thought I had fixe it
>
> Hi there,
>
> I thought I had fixed this memory issue but...
>
> I am using Tomcat 6 in a Windows 32 bit environment (Windows
> 2003) with
> Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
> Postgres 9.1, JDBC 4 driver.
>
>
> The behavior is this:
>
> The user fires of a LARGE query that returns 25,000 large objects.
>
> The user repeats this 20 times within 5 minutes to show me he
> can freeze
> the app.
>
> On the 20th time, it says out of memory (heap space).
>
>
> The JVM indicates that it is maxed to the limit specified in
> Xms and
> that there is only about 2MB of memory free.
>
> I then ask her to log off.
>
> The session time out is set to 20 minutes.
>
> But even after an hour, the memory is not reclaimed by the
> JVM--it still
> reports that only about 2MB is still free.
>
> Is the fact that it is not reclaiming memory an indication of
> a memory leak?
>
> I am using the following settings in my repository.database:
>
> <jdbc-connection-descriptor
> jcd-alias="WebApp"
> default-connection="true"
> platform="PostgreSQL"
> jdbc-level="4.0"
> driver="org.postgresql.Driver"
> protocol="jdbc"
> subprotocol="postgresql"
> dbalias="//localhost:5432/WebApp"
> username="******"
> password="******"
> eager-release="false"
> batch-mode="false">
>
> <connection-pool maxActive="200" validationQuery="" />
> <sequence-manager
> className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
> />
> </jdbc-connection-descriptor>
>
>
> Thanks much!
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> <ma...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
> <ma...@cocoon.apache.org>
>
> The content of this e-mail is intended only for the
> confidential use of the person addressed.
> If you are not the intended recipient, please notify the
> sender and delete this e-mail immediately.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> <ma...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
> <ma...@cocoon.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> <ma...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
> <ma...@cocoon.apache.org>
>
>
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@apache.org>.
On 2011-10-24 19:17, Jasha Joachimsthal wrote:
> On 24 October 2011 18:55, Leszek Gawron <lgawron@apache.org
> <ma...@apache.org>> wrote:
>
> Hello,
>
> @Jasha: advising a user to limit the amount of data usually does not
> work. :)
>
>
> You can tell the user his search query has a result of 25000 items but I
> guess he doesn't want them all in 1 screen. Postprocessing that amount
> of items can be tried but many have failed before...
It you use cocoon as REST service this might not be that bad idea. Using
cocoon I am feeding mobile devices with sync data. Rowsets of 120k
records are not that uncommon. Cocoon is actually one of few (if there
is anything else) frameworks that is to push this much dynamic data into
HTTP response in streaming - the memory footprint on server is minimal
when using a custom generator that "converts" ResultSet right into SAX
events.
Still you're right: presenting this much data on a web page is a sign of
poor design (or stupid customer requirements - sometimes they just don't
listen :) )
lg
--
Leszek Gawron http://lgawron.posterous.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Jasha Joachimsthal <j....@onehippo.com>.
On 24 October 2011 18:55, Leszek Gawron <lg...@apache.org> wrote:
> Hello,
>
> @Jasha: advising a user to limit the amount of data usually does not work.
> :)
You can tell the user his search query has a result of 25000 items but I
guess he doesn't want them all in 1 screen. Postprocessing that amount of
items can be tried but many have failed before...
>
>
> On 2011-10-24 18:29, Paul Joseph wrote:
>
>> Actually the real reason is that the user is using a query that required
>> crafting to return 25K results. He is seeking to show the system "does
>> not scale."
>>
>> Actually I am using in my XML template, a cocoon flowscript repeater
>> widget that does paging. But this is well after the Java logic has
>> built the large array of results (the array results and the vector
>> secureResults below)
>>
>
> I suspect what might be the problem:
>
> 1. Are your continuations session bound? They are not by default - even
> though you set a short session expiry time most of the data is still held by
> continuations which are being expired with totally different mechanism.
> Container bound continuations are useful in only one case: you build a
> session-less site. For "session" exprience ALWAYS use
> session-bound-continuations.
>
> 2. Your flowscript continuation probably holds the ENTIRE rowset and will
> hold it until it expires (and all children expire). You are not supposed to
> put heavyweight object into your continuation.
>
> Your solutions are:
>
> a. use a generator instead of flowscript. Implementing own generator is
> actually dead easy. Style your data with XSLT.
>
> b. make flowscript "forget" the data after the report has been rendered.
> You probably regenerate your rowset with each request anyway.
>
> This one is actually also easy. This is an example of my production code:
>
> /* TEMPLATE
>> var config = {
>> defaultOrderBy: "name",
>> defaultDirection: "asc",
>> valueListProvider: cocoon.getComponent( "valueListBeanName"
>> ),
>> filterModelName: "filterModel",
>> viewName: "viewName",
>> errorRedirect: "/",
>> rowSelectionReturns: false,
>> bizData: bizData,
>> selections: selections,
>> formHandler: function ( action, bizData ) {
>> },
>> rowHandler: function( id, action, bizData ) {
>> }
>> }
>> */
>> function sortedFilteredView( config ) {
>> var defaultMaxResults = new Integer( 100 );
>> var modelName = ( config.filterModelName != undefined ) ?
>> config.filterModelName : config.viewName;
>> var form = new Form( "cocoon:/form-def/" + modelName );
>> //form.createBinding("cocoon:/**form-bind/" +
>> config.filterModelName );
>>
>> form.locale = determineLocale();
>> var model = form.getModel();
>>
>> var bizData = ( config.bizData != undefined ) ? config.bizData :
>> new java.util.HashMap();
>> model.orderBy = config.defaultOrderBy;
>> model.direction = ( config.defaultDirection != undefined ) ?
>> config.defaultDirection : "asc";
>> model.maxResults = ( config.maxResults != undefined ) ?
>> config.maxResults : defaultMaxResults;
>>
>> if ( config.formInitializer != undefined ) {
>> config.formInitializer( model );
>> }
>>
>> model.skipResults = new Integer( 0 );
>> while ( true ) {
>> model.pageNo = new java.lang.Integer( model.skipResults /
>> model.maxResults + 1 );
>> var filterContext = Packages.org.apache.cocoon.**
>> forms.util.**ContainerWidgetAsMap( form.form );
>> var items = config.valueListProvider.**generateResults(
>> filterContext, bizData );
>>
>> var totalItemCount = config.valueListProvider.**countEntries(
>> filterContext, bizData );
>> var pageCount = new java.lang.Integer( java.lang.Math.ceil(
>> totalItemCount / model.maxResults ) );
>> var firstPage = ( model.skipResults == 0 );
>> var lastPage = ( model.skipResults + model.maxResults >=
>> totalItemCount );
>>
>> form.showForm( "form/" + config.viewName,
>> { items : items,
>> orderBy :
>> model.orderBy,
>> direction:
>> model.direction,
>> maxResults:
>> model.maxResults,
>> pageNumber:
>> model.pageNo,
>> pageCount:
>> pageCount,
>> firstPage:
>> firstPage,
>> lastPage :
>> lastPage,
>> viewConfig: config,
>> startIndex:
>> ((model.pageNo - 1) * model.maxResults + 1),
>> bizData : bizData,
>> selections:
>> config.selections },
>> function() {
>> delete
>> items;
>> }
>> );
>>
>> if ( form.submitId == "finish" ) { //search
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId == "cancel" ) {
>> return null;
>> } else if ( form.submitId == "next" ) {
>> model.skipResults = new Integer( model.skipResults
>> + model.maxResults );
>> } else if ( form.submitId == "prev" ) {
>> model.skipResults = new Integer( model.skipResults
>> - model.maxResults );
>> if ( model.skipResults < 0 )
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId == "first" ) {
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId == "last" ) {
>> model.skipResults = new java.lang.Integer( (
>> java.lang.Math.ceil( totalItemCount / model.maxResults ) - 1 ) *
>> model.maxResults );
>> } else if ( form.submitId == "changePage" ) {
>> if ( model.pageNo != null ) {
>> var skipCount = ( model.pageNo - 1 ) *
>> model.maxResults;
>> if ( skipCount < totalItemCount )
>> model.skipResults = new
>> Integer( skipCount );
>> else
>> model.skipResults = new
>> java.lang.Integer( ( java.lang.Math.ceil( totalItemCount / model.maxResults
>> ) - 1 ) * model.maxResults );
>> if ( skipCount < 0 )
>> model.skipResults = new
>> Integer( 0 );
>> }
>> } else if ( form.submitId == "changeMaxResults" ) {
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId != null ) {
>> if ( form.submitId == "selectRow" &&
>> config.rowSelectionReturns == true )
>> return model.rowId;
>> if ( config.rowHandler != undefined && model.rowId
>> != null )
>> config.rowHandler( model.rowId,
>> form.submitId, bizData );
>> if ( config.formHandler != undefined && model.rowId
>> == null )
>> config.formHandler( form.submitId, bizData
>> );
>> }
>> if ( model.maxResults <= 0 )
>> model.maxResults = defaultMaxResults;
>> model.rowId = null;
>> }
>> }
>>
>
> The code is responsible for showing a pageable sortable filterable rowset.
>
> The most important part is :
>
> var items = config.valueListProvider.**generateResults( filterContext,
> bizData );
>
> This object is VERY heavy (it might any row count you might like).
>
> The next important thing is after rendering this object is deleted:
>
> form.showForm( "form/" + config.viewName,
>> { items : items,
>> orderBy :
>> model.orderBy,
>> direction:
>> model.direction,
>> maxResults:
>> model.maxResults,
>> pageNumber:
>> model.pageNo,
>> pageCount:
>> pageCount,
>> firstPage:
>> firstPage,
>> lastPage :
>> lastPage,
>> viewConfig: config,
>> startIndex:
>> ((model.pageNo - 1) * model.maxResults + 1),
>> bizData : bizData,
>> selections:
>> config.selections },
>> function() {
>> delete
>> items;
>> }
>> );
>>
>
> There is a parameter for cocoon.sendPageAndWait which is not widely known:
> a cleanup function.
>
> http://cocoon.apache.org/2.1/**userdocs/flow/api.html#cocoon<http://cocoon.apache.org/2.1/userdocs/flow/api.html#cocoon>
>
> So your code has to look something like that:
>
> var report = generateYourHorriblyBigReport(**);
>
> cocoon.sendPageAndWait( "myreport.html",
> { report: report },
> function() {
> delete report;
> }
> );
>
>
>
> I advice you to use some kind of profiler. Money spent on YourProfiler
> might be one of best spent dollars. You can:
>
> - make a memory snapshot
> - do your request
> - make a subsequent memory snapshot
> - compare both snapshots: there should be NO significant memory
> consumption difference.
>
> HTH
>
> lg
>
> --
> Leszek Gawron http://lgawron.posterous.com
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.**apache.org<us...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
RE: thought I had fixe it
Posted by Laurent Medioni <lm...@temenos.com>.
Oups, never fully trust JavaDoc, correct...
A continuation branch is disposed of only when its leaf expires, see
ContinuationsManagerImpl.
handleParentContinuationExpiration(WebContinuation):
"When a new continuation is created ... its parent continuation is
removed from the expiration set. This way only leaf continuations are
part of the expiration set."
And true from double code check ;)
So depending on how your navigation works the currently navigated branch
can survive a long time...
But when a "leaf" continuations expires (according to the configured
time to live) it is binned, and all of its recursive parents having no
other chidren...
Laurent
-----Original Message-----
From: Leszek Gawron [mailto:lgawron@gmail.com] On Behalf Of Leszek
Gawron
Sent: mercredi 26 octobre 2011 09:44
To: users@cocoon.apache.org
Subject: Re: thought I had fixe it
On 2011-10-26 09:36, Laurent Medioni wrote:
> According to JavaDoc:
> = "How long does a continuation exist in memory since the last access?
> The time is in miliseconds, and the default is 1 hour"
> And reading the code confirms this statement.
> Laurent
which is probably not true.
Thing is: continuations are stored "chained". What i mean is :
- user creates a first continuation in some UI processing
- cocoon stores that continuation
- user naviagates back and creates a new continuation
- that new continuation is linked to the old one
- user may navigate back to some previous continuation and start a
branch
- a continuation tree gets removed from memory when ALL the leaves
expire (still I don't remember if older "root" get expired so even
if in memory you cannot access these continuations)
- this means you can grow continuation tree indefinitely
My knowledge is at least 4-5 years old. I advise to check the source
code of ContinuationsManagerImpl
lg
--
Leszek Gawron http://lgawron.posterous.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
The information in this e-mail and any attachments is confidential and may be legally privileged.
It is intended solely for the addressee or addressees. Any use or disclosure of the contents
of this e-mail/attachments by a not intended recipient is unauthorized and may be unlawful.
If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and
do not necessarily represent those of TEMENOS.
We recommend that you check this e-mail and any attachments against viruses.
TEMENOS accepts no liability for any damage caused by any malicious code or virus transmitted by this e-mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@apache.org>.
On 2011-10-26 09:36, Laurent Medioni wrote:
> According to JavaDoc:
> = "How long does a continuation exist in memory since the last access?
> The time is in miliseconds, and the default is 1 hour"
> And reading the code confirms this statement.
> Laurent
which is probably not true.
Thing is: continuations are stored "chained". What i mean is :
- user creates a first continuation in some UI processing
- cocoon stores that continuation
- user naviagates back and creates a new continuation
- that new continuation is linked to the old one
- user may navigate back to some previous continuation and start a
branch
- a continuation tree gets removed from memory when ALL the leaves
expire (still I don't remember if older "root" get expired so even
if in memory you cannot access these continuations)
- this means you can grow continuation tree indefinitely
My knowledge is at least 4-5 years old. I advise to check the source
code of ContinuationsManagerImpl
lg
--
Leszek Gawron http://lgawron.posterous.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: thought I had fixe it
Posted by Laurent Medioni <lm...@temenos.com>.
According to JavaDoc:
= "How long does a continuation exist in memory since the last access?
The time is in miliseconds, and the default is 1 hour"
And reading the code confirms this statement.
Laurent
-----Original Message-----
From: Paul Joseph [mailto:pjoseph@gmail.com]
Sent: mardi 25 octobre 2011 15:38
To: users@cocoon.apache.org
Subject: Re: thought I had fixe it
These settings are very helpful...
Question--does time-to-live mean between inactivity or regardless of
inactivity? IF the former then I'd like to use a more agressive
value...like the equivalent of 15 minutes?
The information in this e-mail and any attachments is confidential and may be legally privileged.
It is intended solely for the addressee or addressees. Any use or disclosure of the contents
of this e-mail/attachments by a not intended recipient is unauthorized and may be unlawful.
If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and
do not necessarily represent those of TEMENOS.
We recommend that you check this e-mail and any attachments against viruses.
TEMENOS accepts no liability for any damage caused by any malicious code or virus transmitted by this e-mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
These settings are very helpful...
Question--does time-to-live mean between inactivity or regardless of
inactivity? IF the former then I'd like to use a more agressive
value...like the equivalent of 15 minutes?
On 10/25/2011 3:10 AM, Laurent Medioni wrote:
> + No1 potential memory leak ever: be sure to close your jdbc objects
> whatever happens, provide catch blocks to close objects when
> instantiations go wrong + make sure the close() methods are always
> called at some point from your code (as soon as you have read the
> resultSet).
>
> But, as already said, storing large objects in continuations is a sure
> recipe for failure ;)
> You should configure the Continuation Manager with something like:
> <continuations-manager logger="flow" session-bound-continuations="true"
> time-to-live="3600000">
> <expirations-check type="periodic">
> <offset>180000</offset>
> <period>180000</period>
> </expirations-check>
> </continuations-manager>
> This will give continuations a time to live of 1 hour, which is probably
> more than enough for replaying continuations if needed. And when the
> session expires everything will be GCed, as soon as possible/needed...
>
> Laurent
>
> The information in this e-mail and any attachments is confidential and may be legally privileged.
> It is intended solely for the addressee or addressees. Any use or disclosure of the contents
> of this e-mail/attachments by a not intended recipient is unauthorized and may be unlawful.
> If you have received this e-mail in error please notify the sender.
> Please note that any views or opinions presented in this e-mail are solely those of the author and
> do not necessarily represent those of TEMENOS.
> We recommend that you check this e-mail and any attachments against viruses.
> TEMENOS accepts no liability for any damage caused by any malicious code or virus transmitted by this e-mail.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
ah thanks...good optimizations...
I guess nothing was grievously broken in the original version...was
hoping something was so I could get back lost memory! :-)
On 10/25/2011 6:02 AM, Leszek Gawron wrote:
> On 2011-10-25 11:43, Robby Pelssers wrote:
>> He should have known that ;-)
>>
>> For all I know this is still Java
>>
>> Criteria criteria = null;
>>
>> While this is the javascript version:
>>
>> var criteria = null;
>>
>> Robby
>
>
> oops you got me :) That is probably the fault of no-morning-coffee-yet
>
> Anyways the code should looke somewhat like this:
>
>> public Object[] getObjectListbySQL(Object obj, String sql) {
>> if(obj == null)
>> return null;
>>
>> PersistenceBroker broker = null;
>> try {
>> broker = PersistenceBrokerFactory.defaultPersistenceBroker();
>> broker.clearCache();
>>
>> // get the QueryBySQL
>> QueryBySQL q = QueryFactory.newQuery(obj.getClass(), sql);
>> return ((Collection)broker.getCollectionByQuery(q)).toArray();
>> } catch (Exception sqle) {
>> // doing printStackTrace is usually bad practice
>> // either log it or rethrow it
>> // sqle.printStackTrace();
>> throw new MyRuntimeDataAcessException( "unable to fetch
>> data", sqle );
>> } finally {
>> if ( broker != null )
>> broker.close();
>> }
>> }
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@mobilebox.pl>.
On 2011-10-25 11:43, Robby Pelssers wrote:
> He should have known that ;-)
>
> For all I know this is still Java
>
> Criteria criteria = null;
>
> While this is the javascript version:
>
> var criteria = null;
>
> Robby
oops you got me :) That is probably the fault of no-morning-coffee-yet
Anyways the code should looke somewhat like this:
> public Object[] getObjectListbySQL(Object obj, String sql) {
> if(obj == null)
> return null;
>
> PersistenceBroker broker = null;
> try {
> broker = PersistenceBrokerFactory.defaultPersistenceBroker();
> broker.clearCache();
>
> // get the QueryBySQL
> QueryBySQL q = QueryFactory.newQuery(obj.getClass(), sql);
> return ((Collection)broker.getCollectionByQuery(q)).toArray();
> } catch (Exception sqle) {
> // doing printStackTrace is usually bad practice
> // either log it or rethrow it
> // sqle.printStackTrace();
> throw new MyRuntimeDataAcessException( "unable to fetch data", sqle );
> } finally {
> if ( broker != null )
> broker.close();
> }
> }
--
Leszek Gawron http://www.mobilebox.pl/krs.html
CTO at MobileBox S.A.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: thought I had fixe it
Posted by Robby Pelssers <Ro...@nxp.com>.
He should have known that ;-)
For all I know this is still Java
Criteria criteria = null;
While this is the javascript version:
var criteria = null;
Robby
-----Original Message-----
From: Paul Joseph [mailto:pjoseph@gmail.com]
Sent: Tuesday, October 25, 2011 11:41 AM
To: users@cocoon.apache.org
Subject: Re: thought I had fixe it
Sorry about that, I should have been clearer...the code I listed is Java :-)
On 10/25/2011 5:38 AM, Leszek Gawron wrote:
> On 2011-10-25 11:09, Paul Joseph wrote:
>> Would appreciate a "code review" of the same to make sure things are
>> handled correctly, memory wise.
>
> My code review says: do not do business logic in flowscript.
>
> Move ALL of this code quoted into java classes.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
Sorry about that, I should have been clearer...the code I listed is Java :-)
On 10/25/2011 5:38 AM, Leszek Gawron wrote:
> On 2011-10-25 11:09, Paul Joseph wrote:
>> Would appreciate a "code review" of the same to make sure things are
>> handled correctly, memory wise.
>
> My code review says: do not do business logic in flowscript.
>
> Move ALL of this code quoted into java classes.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@mobilebox.pl>.
On 2011-10-25 11:09, Paul Joseph wrote:
> Would appreciate a "code review" of the same to make sure things are
> handled correctly, memory wise.
My code review says: do not do business logic in flowscript.
Move ALL of this code quoted into java classes.
--
Leszek Gawron http://www.mobilebox.pl/krs.html
CTO at MobileBox S.A.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
Good points, thank you.
Re. pont 1 (closing jdbc objects) I observe the query associated with
retrieving these large objects is getObjectListbySQL
My code for this, which I don't quite recall how I came up with and
which I have been using for many years now, is as below....
Would appreciate a "code review" of the same to make sure things are
handled correctly, memory wise.
best
Paul
public Object[] getObjectListbySQL(Object obj, String sql) {
Criteria criteria = null;
Collection objects = null;
PersistenceBroker broker = null;
if(obj == null)
return null;
try {
broker = PersistenceBrokerFactory.defaultPersistenceBroker();
broker.clearCache();
// get the QueryBySQL
QueryBySQL q = QueryFactory.newQuery(obj.getClass(), sql);
objects = (Collection)broker.getCollectionByQuery(q);
//System.out.println("from getObjectListbySQL, collection
length is: " + objects.size());
} catch (Exception sqle) {
sqle.printStackTrace();
String msg = sqle.getMessage();
broker.close();
return null;
}
if(broker != null)
{
broker.close(); //
Release broker instance to the broker-pool
}
return objects.toArray();
}
On 10/25/2011 3:10 AM, Laurent Medioni wrote:
> + No1 potential memory leak ever: be sure to close your jdbc objects
> whatever happens, provide catch blocks to close objects when
> instantiations go wrong + make sure the close() methods are always
> called at some point from your code (as soon as you have read the
> resultSet).
>
> But, as already said, storing large objects in continuations is a sure
> recipe for failure ;)
> You should configure the Continuation Manager with something like:
> <continuations-manager logger="flow" session-bound-continuations="true"
> time-to-live="3600000">
> <expirations-check type="periodic">
> <offset>180000</offset>
> <period>180000</period>
> </expirations-check>
> </continuations-manager>
> This will give continuations a time to live of 1 hour, which is probably
> more than enough for replaying continuations if needed. And when the
> session expires everything will be GCed, as soon as possible/needed...
>
> Laurent
>
> The information in this e-mail and any attachments is confidential and may be legally privileged.
> It is intended solely for the addressee or addressees. Any use or disclosure of the contents
> of this e-mail/attachments by a not intended recipient is unauthorized and may be unlawful.
> If you have received this e-mail in error please notify the sender.
> Please note that any views or opinions presented in this e-mail are solely those of the author and
> do not necessarily represent those of TEMENOS.
> We recommend that you check this e-mail and any attachments against viruses.
> TEMENOS accepts no liability for any damage caused by any malicious code or virus transmitted by this e-mail.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: thought I had fixe it
Posted by Laurent Medioni <lm...@temenos.com>.
+ No1 potential memory leak ever: be sure to close your jdbc objects
whatever happens, provide catch blocks to close objects when
instantiations go wrong + make sure the close() methods are always
called at some point from your code (as soon as you have read the
resultSet).
But, as already said, storing large objects in continuations is a sure
recipe for failure ;)
You should configure the Continuation Manager with something like:
<continuations-manager logger="flow" session-bound-continuations="true"
time-to-live="3600000">
<expirations-check type="periodic">
<offset>180000</offset>
<period>180000</period>
</expirations-check>
</continuations-manager>
This will give continuations a time to live of 1 hour, which is probably
more than enough for replaying continuations if needed. And when the
session expires everything will be GCed, as soon as possible/needed...
Laurent
The information in this e-mail and any attachments is confidential and may be legally privileged.
It is intended solely for the addressee or addressees. Any use or disclosure of the contents
of this e-mail/attachments by a not intended recipient is unauthorized and may be unlawful.
If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and
do not necessarily represent those of TEMENOS.
We recommend that you check this e-mail and any attachments against viruses.
TEMENOS accepts no liability for any damage caused by any malicious code or virus transmitted by this e-mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
These are excellent ideas Leszek, I sincerely thank you.
Will implement them over the coming days, especially the pipeline one below!
brgds
Paul
On 10/24/2011 1:40 PM, Leszek Gawron wrote:
> On 2011-10-24 19:22, Paul Joseph wrote:
>>
>> I think you may have the solution--I am indeed putting in a very heavy
>> weight object into the continuation. Let me go through this code and
>> get back to you with questions if I may.
>
> I simply remember what used to kill my site and what made up my first
> commits as cocoon commiter :)
>
> have fun
>
> Please DO enable session-bound-continuations! You gain:
> - all continuations simply "vanish" for an expired session
> - there is no single continuations collection
> - you will prevent very strange errors where there is still a
> continuation that you can invoke but there actually is no user session
> anymore. Been there, broke that ;)
>
> Remember that even though you delete heavy object from flowscript your
> implementation is still not "memory nice". You need to generate
> VeryLargeObject, render it and then forget it. Imagine 10 users doing
> it at the same time - kills your server anyway.
>
> There are two solutions:
>
> - some Result object which does not hold all data but actually is an
> open rowset wrapper. Your jx:template iterates the rowset. The data is
> fetched from db as you go. Remember to finalize your resources in a
> cleanup function.
>
> - custom generator that queries database directly and emits SAX events.
>
>
> Both solutions let you generate results on almost ANY size. One more
> gotcha: if you generate large responses please mind that by default
> cocoon buffers the whole response before it's streamed to client (just
> in case an exception throws in the middle it can send a nice HTTP 500
> instead of data broken in the middle). This introduces a momory strain
> on the server and a significant response delay (data is not being sent
> to client until it's fully generated).
>
> If you are willing to accept inferior error handling you can disable
> this buffering for memory intensive pipelines (outputBufferSize):
>
>> <map:pipeline>
>> <map:parameter name="outputBufferSize" value="0"/>
>> <map:act type="set-header">
>> <map:parameter name="Expires" value="-1"/>
>> <map:parameter name="Cache-Control" value="no-cache"/>
>> <map:parameter name="Pragma" value="no-cache"/>
>> </map:act>
>> <map:match pattern="view/*.jx">
>> <map:generate type="jx" src="view/{1}.jx"/>
>> <map:transform src="stylesheets/prefixes.xsl"/>
>> <map:serialize type="xml"/>
>> </map:match>
>> <map:match pattern="text-view/*.jx">
>> <map:generate type="jx" src="view/{1}.jx"/>
>> <map:serialize type="text"/>
>> </map:match>
>> <map:match pattern="listCoupons.do">
>> <map:generate type="coupons"/>
>> <map:serialize type="xml"/>
>> </map:match>
>> <map:match pattern="*.do">
>> <map:call function="{1}"/>
>> </map:match>
>> </map:pipeline>
>
> lg
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@mobilebox.pl>.
On 2011-10-24 19:22, Paul Joseph wrote:
>
> I think you may have the solution--I am indeed putting in a very heavy
> weight object into the continuation. Let me go through this code and
> get back to you with questions if I may.
I simply remember what used to kill my site and what made up my first
commits as cocoon commiter :)
have fun
Please DO enable session-bound-continuations! You gain:
- all continuations simply "vanish" for an expired session
- there is no single continuations collection
- you will prevent very strange errors where there is still a
continuation that you can invoke but there actually is no user session
anymore. Been there, broke that ;)
Remember that even though you delete heavy object from flowscript your
implementation is still not "memory nice". You need to generate
VeryLargeObject, render it and then forget it. Imagine 10 users doing it
at the same time - kills your server anyway.
There are two solutions:
- some Result object which does not hold all data but actually is an
open rowset wrapper. Your jx:template iterates the rowset. The data is
fetched from db as you go. Remember to finalize your resources in a
cleanup function.
- custom generator that queries database directly and emits SAX events.
Both solutions let you generate results on almost ANY size. One more
gotcha: if you generate large responses please mind that by default
cocoon buffers the whole response before it's streamed to client (just
in case an exception throws in the middle it can send a nice HTTP 500
instead of data broken in the middle). This introduces a momory strain
on the server and a significant response delay (data is not being sent
to client until it's fully generated).
If you are willing to accept inferior error handling you can disable
this buffering for memory intensive pipelines (outputBufferSize):
> <map:pipeline>
> <map:parameter name="outputBufferSize" value="0"/>
> <map:act type="set-header">
> <map:parameter name="Expires" value="-1"/>
> <map:parameter name="Cache-Control" value="no-cache"/>
> <map:parameter name="Pragma" value="no-cache"/>
> </map:act>
> <map:match pattern="view/*.jx">
> <map:generate type="jx" src="view/{1}.jx"/>
> <map:transform src="stylesheets/prefixes.xsl"/>
> <map:serialize type="xml"/>
> </map:match>
> <map:match pattern="text-view/*.jx">
> <map:generate type="jx" src="view/{1}.jx"/>
> <map:serialize type="text"/>
> </map:match>
> <map:match pattern="listCoupons.do">
> <map:generate type="coupons"/>
> <map:serialize type="xml"/>
> </map:match>
> <map:match pattern="*.do">
> <map:call function="{1}"/>
> </map:match>
> </map:pipeline>
lg
--
Leszek Gawron http://www.mobilebox.pl/krs.html
CTO at MobileBox S.A.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@apache.org>.
On 2011-10-24 19:22, Paul Joseph wrote:
>
> I think you may have the solution--I am indeed putting in a very heavy
> weight object into the continuation. Let me go through this code and
> get back to you with questions if I may.
I simply remember what used to kill my site and what made up my first
commits as cocoon commiter :)
have fun
Please DO enable session-bound-continuations! You gain:
- all continuations simply "vanish" for an expired session
- there is no single continuations collection
- you will prevent very strange errors where there is still a
continuation that you can invoke but there actually is no user session
anymore. Been there, broke that ;)
Remember that even though you delete heavy object from flowscript your
implementation is still not "memory nice". You need to generate
VeryLargeObject, render it and then forget it. Imagine 10 users doing it
at the same time - kills your server anyway.
There are two solutions:
- some Result object which does not hold all data but actually is an
open rowset wrapper. Your jx:template iterates the rowset. The data is
fetched from db as you go. Remember to finalize your resources in a
cleanup function.
- custom generator that queries database directly and emits SAX events.
Both solutions let you generate results on almost ANY size. One more
gotcha: if you generate large responses please mind that by default
cocoon buffers the whole response before it's streamed to client (just
in case an exception throws in the middle it can send a nice HTTP 500
instead of data broken in the middle). This introduces a momory strain
on the server and a significant response delay (data is not being sent
to client until it's fully generated).
If you are willing to accept inferior error handling you can disable
this buffering for memory intensive pipelines (outputBufferSize):
> <map:pipeline>
> <map:parameter name="outputBufferSize" value="0"/>
> <map:act type="set-header">
> <map:parameter name="Expires" value="-1"/>
> <map:parameter name="Cache-Control" value="no-cache"/>
> <map:parameter name="Pragma" value="no-cache"/>
> </map:act>
> <map:match pattern="view/*.jx">
> <map:generate type="jx" src="view/{1}.jx"/>
> <map:transform src="stylesheets/prefixes.xsl"/>
> <map:serialize type="xml"/>
> </map:match>
> <map:match pattern="text-view/*.jx">
> <map:generate type="jx" src="view/{1}.jx"/>
> <map:serialize type="text"/>
> </map:match>
> <map:match pattern="listCoupons.do">
> <map:generate type="coupons"/>
> <map:serialize type="xml"/>
> </map:match>
> <map:match pattern="*.do">
> <map:call function="{1}"/>
> </map:match>
> </map:pipeline>
lg
--
Leszek Gawron http://lgawron.posterous.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
Hi there,
I think you may have the solution--I am indeed putting in a very heavy
weight object into the continuation. Let me go through this code and
get back to you with questions if I may.
Paul
On 10/24/2011 12:55 PM, Leszek Gawron wrote:
> Hello,
>
> @Jasha: advising a user to limit the amount of data usually does not
> work. :)
>
>
> On 2011-10-24 18:29, Paul Joseph wrote:
>> Actually the real reason is that the user is using a query that required
>> crafting to return 25K results. He is seeking to show the system "does
>> not scale."
>>
>> Actually I am using in my XML template, a cocoon flowscript repeater
>> widget that does paging. But this is well after the Java logic has
>> built the large array of results (the array results and the vector
>> secureResults below)
>
> I suspect what might be the problem:
>
> 1. Are your continuations session bound? They are not by default -
> even though you set a short session expiry time most of the data is
> still held by continuations which are being expired with totally
> different mechanism. Container bound continuations are useful in only
> one case: you build a session-less site. For "session" exprience
> ALWAYS use session-bound-continuations.
>
> 2. Your flowscript continuation probably holds the ENTIRE rowset and
> will hold it until it expires (and all children expire). You are not
> supposed to put heavyweight object into your continuation.
>
> Your solutions are:
>
> a. use a generator instead of flowscript. Implementing own generator
> is actually dead easy. Style your data with XSLT.
>
> b. make flowscript "forget" the data after the report has been
> rendered. You probably regenerate your rowset with each request anyway.
>
> This one is actually also easy. This is an example of my production code:
>
>> /* TEMPLATE
>> var config = {
>> defaultOrderBy: "name",
>> defaultDirection: "asc",
>> valueListProvider: cocoon.getComponent( "valueListBeanName" ),
>> filterModelName: "filterModel",
>> viewName: "viewName",
>> errorRedirect: "/",
>> rowSelectionReturns: false,
>> bizData: bizData,
>> selections: selections,
>> formHandler: function ( action, bizData ) {
>> },
>> rowHandler: function( id, action, bizData ) {
>> }
>> }
>> */
>> function sortedFilteredView( config ) {
>> var defaultMaxResults = new Integer( 100 );
>> var modelName = ( config.filterModelName != undefined ) ?
>> config.filterModelName : config.viewName;
>> var form = new Form( "cocoon:/form-def/" + modelName );
>> //form.createBinding("cocoon:/form-bind/" +
>> config.filterModelName );
>>
>> form.locale = determineLocale();
>> var model = form.getModel();
>>
>> var bizData = ( config.bizData != undefined ) ? config.bizData :
>> new java.util.HashMap();
>> model.orderBy = config.defaultOrderBy;
>> model.direction = ( config.defaultDirection != undefined ) ?
>> config.defaultDirection : "asc";
>> model.maxResults = ( config.maxResults != undefined ) ?
>> config.maxResults : defaultMaxResults;
>>
>> if ( config.formInitializer != undefined ) {
>> config.formInitializer( model );
>> }
>>
>> model.skipResults = new Integer( 0 );
>> while ( true ) {
>> model.pageNo = new java.lang.Integer( model.skipResults /
>> model.maxResults + 1 );
>> var filterContext =
>> Packages.org.apache.cocoon.forms.util.ContainerWidgetAsMap( form.form );
>> var items = config.valueListProvider.generateResults(
>> filterContext, bizData );
>>
>> var totalItemCount = config.valueListProvider.countEntries(
>> filterContext, bizData );
>> var pageCount = new java.lang.Integer( java.lang.Math.ceil(
>> totalItemCount / model.maxResults ) );
>> var firstPage = ( model.skipResults == 0 );
>> var lastPage = ( model.skipResults + model.maxResults >=
>> totalItemCount );
>>
>> form.showForm( "form/" + config.viewName,
>> { items : items,
>> orderBy : model.orderBy,
>> direction: model.direction,
>> maxResults: model.maxResults,
>> pageNumber: model.pageNo,
>> pageCount: pageCount,
>> firstPage: firstPage,
>> lastPage : lastPage,
>> viewConfig: config,
>> startIndex: ((model.pageNo - 1) *
>> model.maxResults + 1),
>> bizData : bizData,
>> selections: config.selections },
>> function() {
>> delete items;
>> }
>> );
>>
>> if ( form.submitId == "finish" ) { //search
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId == "cancel" ) {
>> return null;
>> } else if ( form.submitId == "next" ) {
>> model.skipResults = new Integer( model.skipResults +
>> model.maxResults );
>> } else if ( form.submitId == "prev" ) {
>> model.skipResults = new Integer( model.skipResults -
>> model.maxResults );
>> if ( model.skipResults < 0 )
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId == "first" ) {
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId == "last" ) {
>> model.skipResults = new java.lang.Integer( (
>> java.lang.Math.ceil( totalItemCount / model.maxResults ) - 1 ) *
>> model.maxResults );
>> } else if ( form.submitId == "changePage" ) {
>> if ( model.pageNo != null ) {
>> var skipCount = ( model.pageNo - 1 ) *
>> model.maxResults;
>> if ( skipCount < totalItemCount )
>> model.skipResults = new Integer( skipCount );
>> else
>> model.skipResults = new java.lang.Integer( (
>> java.lang.Math.ceil( totalItemCount / model.maxResults ) - 1 ) *
>> model.maxResults );
>> if ( skipCount < 0 )
>> model.skipResults = new Integer( 0 );
>> }
>> } else if ( form.submitId == "changeMaxResults" ) {
>> model.skipResults = new Integer( 0 );
>> } else if ( form.submitId != null ) {
>> if ( form.submitId == "selectRow" &&
>> config.rowSelectionReturns == true )
>> return model.rowId;
>> if ( config.rowHandler != undefined && model.rowId != null )
>> config.rowHandler( model.rowId, form.submitId,
>> bizData );
>> if ( config.formHandler != undefined && model.rowId ==
>> null )
>> config.formHandler( form.submitId, bizData );
>> }
>> if ( model.maxResults <= 0 )
>> model.maxResults = defaultMaxResults;
>> model.rowId = null;
>> }
>> }
>
> The code is responsible for showing a pageable sortable filterable
> rowset.
>
> The most important part is :
>
> var items = config.valueListProvider.generateResults( filterContext,
> bizData );
>
> This object is VERY heavy (it might any row count you might like).
>
> The next important thing is after rendering this object is deleted:
>
>> form.showForm( "form/" + config.viewName,
>> { items : items,
>> orderBy : model.orderBy,
>> direction: model.direction,
>> maxResults: model.maxResults,
>> pageNumber: model.pageNo,
>> pageCount: pageCount,
>> firstPage: firstPage,
>> lastPage : lastPage,
>> viewConfig: config,
>> startIndex: ((model.pageNo - 1) *
>> model.maxResults + 1),
>> bizData : bizData,
>> selections: config.selections },
>> function() {
>> delete items;
>> }
>> );
>
> There is a parameter for cocoon.sendPageAndWait which is not widely
> known: a cleanup function.
>
> http://cocoon.apache.org/2.1/userdocs/flow/api.html#cocoon
>
> So your code has to look something like that:
>
> var report = generateYourHorriblyBigReport();
>
> cocoon.sendPageAndWait( "myreport.html",
> { report: report },
> function() {
> delete report;
> }
> );
>
>
>
> I advice you to use some kind of profiler. Money spent on YourProfiler
> might be one of best spent dollars. You can:
>
> - make a memory snapshot
> - do your request
> - make a subsequent memory snapshot
> - compare both snapshots: there should be NO significant memory
> consumption difference.
>
> HTH
>
> lg
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@apache.org>.
Hello,
@Jasha: advising a user to limit the amount of data usually does not
work. :)
On 2011-10-24 18:29, Paul Joseph wrote:
> Actually the real reason is that the user is using a query that required
> crafting to return 25K results. He is seeking to show the system "does
> not scale."
>
> Actually I am using in my XML template, a cocoon flowscript repeater
> widget that does paging. But this is well after the Java logic has
> built the large array of results (the array results and the vector
> secureResults below)
I suspect what might be the problem:
1. Are your continuations session bound? They are not by default - even
though you set a short session expiry time most of the data is still
held by continuations which are being expired with totally different
mechanism. Container bound continuations are useful in only one case:
you build a session-less site. For "session" exprience ALWAYS use
session-bound-continuations.
2. Your flowscript continuation probably holds the ENTIRE rowset and
will hold it until it expires (and all children expire). You are not
supposed to put heavyweight object into your continuation.
Your solutions are:
a. use a generator instead of flowscript. Implementing own generator is
actually dead easy. Style your data with XSLT.
b. make flowscript "forget" the data after the report has been rendered.
You probably regenerate your rowset with each request anyway.
This one is actually also easy. This is an example of my production code:
> /* TEMPLATE
> var config = {
> defaultOrderBy: "name",
> defaultDirection: "asc",
> valueListProvider: cocoon.getComponent( "valueListBeanName" ),
> filterModelName: "filterModel",
> viewName: "viewName",
> errorRedirect: "/",
> rowSelectionReturns: false,
> bizData: bizData,
> selections: selections,
> formHandler: function ( action, bizData ) {
> },
> rowHandler: function( id, action, bizData ) {
> }
> }
> */
> function sortedFilteredView( config ) {
> var defaultMaxResults = new Integer( 100 );
> var modelName = ( config.filterModelName != undefined ) ? config.filterModelName : config.viewName;
> var form = new Form( "cocoon:/form-def/" + modelName );
> //form.createBinding("cocoon:/form-bind/" + config.filterModelName );
>
> form.locale = determineLocale();
> var model = form.getModel();
>
> var bizData = ( config.bizData != undefined ) ? config.bizData : new java.util.HashMap();
> model.orderBy = config.defaultOrderBy;
> model.direction = ( config.defaultDirection != undefined ) ? config.defaultDirection : "asc";
> model.maxResults = ( config.maxResults != undefined ) ? config.maxResults : defaultMaxResults;
>
> if ( config.formInitializer != undefined ) {
> config.formInitializer( model );
> }
>
> model.skipResults = new Integer( 0 );
> while ( true ) {
> model.pageNo = new java.lang.Integer( model.skipResults / model.maxResults + 1 );
> var filterContext = Packages.org.apache.cocoon.forms.util.ContainerWidgetAsMap( form.form );
> var items = config.valueListProvider.generateResults( filterContext, bizData );
>
> var totalItemCount = config.valueListProvider.countEntries( filterContext, bizData );
> var pageCount = new java.lang.Integer( java.lang.Math.ceil( totalItemCount / model.maxResults ) );
> var firstPage = ( model.skipResults == 0 );
> var lastPage = ( model.skipResults + model.maxResults >= totalItemCount );
>
> form.showForm( "form/" + config.viewName,
> { items : items,
> orderBy : model.orderBy,
> direction: model.direction,
> maxResults: model.maxResults,
> pageNumber: model.pageNo,
> pageCount: pageCount,
> firstPage: firstPage,
> lastPage : lastPage,
> viewConfig: config,
> startIndex: ((model.pageNo - 1) * model.maxResults + 1),
> bizData : bizData,
> selections: config.selections },
> function() {
> delete items;
> }
> );
>
> if ( form.submitId == "finish" ) { //search
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId == "cancel" ) {
> return null;
> } else if ( form.submitId == "next" ) {
> model.skipResults = new Integer( model.skipResults + model.maxResults );
> } else if ( form.submitId == "prev" ) {
> model.skipResults = new Integer( model.skipResults - model.maxResults );
> if ( model.skipResults < 0 )
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId == "first" ) {
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId == "last" ) {
> model.skipResults = new java.lang.Integer( ( java.lang.Math.ceil( totalItemCount / model.maxResults ) - 1 ) * model.maxResults );
> } else if ( form.submitId == "changePage" ) {
> if ( model.pageNo != null ) {
> var skipCount = ( model.pageNo - 1 ) * model.maxResults;
> if ( skipCount < totalItemCount )
> model.skipResults = new Integer( skipCount );
> else
> model.skipResults = new java.lang.Integer( ( java.lang.Math.ceil( totalItemCount / model.maxResults ) - 1 ) * model.maxResults );
> if ( skipCount < 0 )
> model.skipResults = new Integer( 0 );
> }
> } else if ( form.submitId == "changeMaxResults" ) {
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId != null ) {
> if ( form.submitId == "selectRow" && config.rowSelectionReturns == true )
> return model.rowId;
> if ( config.rowHandler != undefined && model.rowId != null )
> config.rowHandler( model.rowId, form.submitId, bizData );
> if ( config.formHandler != undefined && model.rowId == null )
> config.formHandler( form.submitId, bizData );
> }
> if ( model.maxResults <= 0 )
> model.maxResults = defaultMaxResults;
> model.rowId = null;
> }
> }
The code is responsible for showing a pageable sortable filterable rowset.
The most important part is :
var items = config.valueListProvider.generateResults( filterContext,
bizData );
This object is VERY heavy (it might any row count you might like).
The next important thing is after rendering this object is deleted:
> form.showForm( "form/" + config.viewName,
> { items : items,
> orderBy : model.orderBy,
> direction: model.direction,
> maxResults: model.maxResults,
> pageNumber: model.pageNo,
> pageCount: pageCount,
> firstPage: firstPage,
> lastPage : lastPage,
> viewConfig: config,
> startIndex: ((model.pageNo - 1) * model.maxResults + 1),
> bizData : bizData,
> selections: config.selections },
> function() {
> delete items;
> }
> );
There is a parameter for cocoon.sendPageAndWait which is not widely
known: a cleanup function.
http://cocoon.apache.org/2.1/userdocs/flow/api.html#cocoon
So your code has to look something like that:
var report = generateYourHorriblyBigReport();
cocoon.sendPageAndWait( "myreport.html",
{ report: report },
function() {
delete report;
}
);
I advice you to use some kind of profiler. Money spent on YourProfiler
might be one of best spent dollars. You can:
- make a memory snapshot
- do your request
- make a subsequent memory snapshot
- compare both snapshots: there should be NO significant memory
consumption difference.
HTH
lg
--
Leszek Gawron http://lgawron.posterous.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Leszek Gawron <lg...@mobilebox.pl>.
Hello,
@Jasha: advising a user to limit the amount of data usually does not
work. :)
On 2011-10-24 18:29, Paul Joseph wrote:
> Actually the real reason is that the user is using a query that required
> crafting to return 25K results. He is seeking to show the system "does
> not scale."
>
> Actually I am using in my XML template, a cocoon flowscript repeater
> widget that does paging. But this is well after the Java logic has
> built the large array of results (the array results and the vector
> secureResults below)
I suspect what might be the problem:
1. Are your continuations session bound? They are not by default - even
though you set a short session expiry time most of the data is still
held by continuations which are being expired with totally different
mechanism. Container bound continuations are useful in only one case:
you build a session-less site. For "session" exprience ALWAYS use
session-bound-continuations.
2. Your flowscript continuation probably holds the ENTIRE rowset and
will hold it until it expires (and all children expire). You are not
supposed to put heavyweight object into your continuation.
Your solutions are:
a. use a generator instead of flowscript. Implementing own generator is
actually dead easy. Style your data with XSLT.
b. make flowscript "forget" the data after the report has been rendered.
You probably regenerate your rowset with each request anyway.
This one is actually also easy. This is an example of my production code:
> /* TEMPLATE
> var config = {
> defaultOrderBy: "name",
> defaultDirection: "asc",
> valueListProvider: cocoon.getComponent( "valueListBeanName" ),
> filterModelName: "filterModel",
> viewName: "viewName",
> errorRedirect: "/",
> rowSelectionReturns: false,
> bizData: bizData,
> selections: selections,
> formHandler: function ( action, bizData ) {
> },
> rowHandler: function( id, action, bizData ) {
> }
> }
> */
> function sortedFilteredView( config ) {
> var defaultMaxResults = new Integer( 100 );
> var modelName = ( config.filterModelName != undefined ) ? config.filterModelName : config.viewName;
> var form = new Form( "cocoon:/form-def/" + modelName );
> //form.createBinding("cocoon:/form-bind/" + config.filterModelName );
>
> form.locale = determineLocale();
> var model = form.getModel();
>
> var bizData = ( config.bizData != undefined ) ? config.bizData : new java.util.HashMap();
> model.orderBy = config.defaultOrderBy;
> model.direction = ( config.defaultDirection != undefined ) ? config.defaultDirection : "asc";
> model.maxResults = ( config.maxResults != undefined ) ? config.maxResults : defaultMaxResults;
>
> if ( config.formInitializer != undefined ) {
> config.formInitializer( model );
> }
>
> model.skipResults = new Integer( 0 );
> while ( true ) {
> model.pageNo = new java.lang.Integer( model.skipResults / model.maxResults + 1 );
> var filterContext = Packages.org.apache.cocoon.forms.util.ContainerWidgetAsMap( form.form );
> var items = config.valueListProvider.generateResults( filterContext, bizData );
>
> var totalItemCount = config.valueListProvider.countEntries( filterContext, bizData );
> var pageCount = new java.lang.Integer( java.lang.Math.ceil( totalItemCount / model.maxResults ) );
> var firstPage = ( model.skipResults == 0 );
> var lastPage = ( model.skipResults + model.maxResults >= totalItemCount );
>
> form.showForm( "form/" + config.viewName,
> { items : items,
> orderBy : model.orderBy,
> direction: model.direction,
> maxResults: model.maxResults,
> pageNumber: model.pageNo,
> pageCount: pageCount,
> firstPage: firstPage,
> lastPage : lastPage,
> viewConfig: config,
> startIndex: ((model.pageNo - 1) * model.maxResults + 1),
> bizData : bizData,
> selections: config.selections },
> function() {
> delete items;
> }
> );
>
> if ( form.submitId == "finish" ) { //search
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId == "cancel" ) {
> return null;
> } else if ( form.submitId == "next" ) {
> model.skipResults = new Integer( model.skipResults + model.maxResults );
> } else if ( form.submitId == "prev" ) {
> model.skipResults = new Integer( model.skipResults - model.maxResults );
> if ( model.skipResults < 0 )
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId == "first" ) {
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId == "last" ) {
> model.skipResults = new java.lang.Integer( ( java.lang.Math.ceil( totalItemCount / model.maxResults ) - 1 ) * model.maxResults );
> } else if ( form.submitId == "changePage" ) {
> if ( model.pageNo != null ) {
> var skipCount = ( model.pageNo - 1 ) * model.maxResults;
> if ( skipCount < totalItemCount )
> model.skipResults = new Integer( skipCount );
> else
> model.skipResults = new java.lang.Integer( ( java.lang.Math.ceil( totalItemCount / model.maxResults ) - 1 ) * model.maxResults );
> if ( skipCount < 0 )
> model.skipResults = new Integer( 0 );
> }
> } else if ( form.submitId == "changeMaxResults" ) {
> model.skipResults = new Integer( 0 );
> } else if ( form.submitId != null ) {
> if ( form.submitId == "selectRow" && config.rowSelectionReturns == true )
> return model.rowId;
> if ( config.rowHandler != undefined && model.rowId != null )
> config.rowHandler( model.rowId, form.submitId, bizData );
> if ( config.formHandler != undefined && model.rowId == null )
> config.formHandler( form.submitId, bizData );
> }
> if ( model.maxResults <= 0 )
> model.maxResults = defaultMaxResults;
> model.rowId = null;
> }
> }
The code is responsible for showing a pageable sortable filterable rowset.
The most important part is :
var items = config.valueListProvider.generateResults( filterContext,
bizData );
This object is VERY heavy (it might any row count you might like).
The next important thing is after rendering this object is deleted:
> form.showForm( "form/" + config.viewName,
> { items : items,
> orderBy : model.orderBy,
> direction: model.direction,
> maxResults: model.maxResults,
> pageNumber: model.pageNo,
> pageCount: pageCount,
> firstPage: firstPage,
> lastPage : lastPage,
> viewConfig: config,
> startIndex: ((model.pageNo - 1) * model.maxResults + 1),
> bizData : bizData,
> selections: config.selections },
> function() {
> delete items;
> }
> );
There is a parameter for cocoon.sendPageAndWait which is not widely
known: a cleanup function.
http://cocoon.apache.org/2.1/userdocs/flow/api.html#cocoon
So your code has to look something like that:
var report = generateYourHorriblyBigReport();
cocoon.sendPageAndWait( "myreport.html",
{ report: report },
function() {
delete report;
}
);
I advice you to use some kind of profiler. Money spent on YourProfiler
might be one of best spent dollars. You can:
- make a memory snapshot
- do your request
- make a subsequent memory snapshot
- compare both snapshots: there should be NO significant memory
consumption difference.
HTH
lg
--
Leszek Gawron http://www.mobilebox.pl/krs.html
CTO at MobileBox S.A.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
Setting a limit is "Plan B". You are correct re. the paging needing to
be done at the db level--will do that in the next release.
On 10/24/2011 12:41 PM, Jasha Joachimsthal wrote:
>
>
> On 24 October 2011 18:29, Paul Joseph <pjoseph@gmail.com
> <ma...@gmail.com>> wrote:
>
> Actually the real reason is that the user is using a query that
> required crafting to return 25K results. He is seeking to show
> the system "does not scale."
>
>
> Then set a limit when building the query
>
>
> Actually I am using in my XML template, a cocoon flowscript
> repeater widget that does paging. But this is well after the Java
> logic has built the large array of results (the array results and
> the vector secureResults below)
>
>
> The actual paging should be done on DB level, not after you've fetched
> the enormous amount of results.
>
>
> On 10/24/2011 12:22 PM, Jasha Joachimsthal wrote:
>> The first question that comes to my mind: why would you need
>> 25000 objects as query result? Can't you do 1 query that only
>> returns the total possible amount of objects and another query
>> that returns e.g. the first 10 results and build in some paging
>> logic for the next 10.
>>
>> Jasha Joachimsthal
>>
>> Europe - Amsterdam - Oosteinde 11, 1017 WT Amsterdam - +31(0)20
>> 522 4466 <tel:%2B31%280%2920%20522%204466>
>> US - Boston - 1 Broadway, Cambridge, MA 02142 - +1 877 414 4776
>> (toll free)
>>
>> www.onehippo.com <http://www.onehippo.com/>
>>
>>
>> On 24 October 2011 18:13, Paul Joseph <pjoseph@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> I think you hit the nail on the head.
>>
>> I assumed that the memory is freed at session time out. I did
>> not realize this was not the case.
>>
>> I know what the culprit is in this case--the array that I use
>> to store the objects returned from the query.
>>
>> I do this in two places--once in the business logic (java
>> code) and once in the flow script.
>>
>> The code in the java business logic is as follows--the item
>> of interest is a "Task". How would I optimize it in terms of
>> memory collection?
>>
>> public Object[] getSearchResults() throws Exception {
>>
>> Object[] results = getSearchResultBeans(sql_query);
>>
>> Vector secureResults = new Vector();
>>
>> for (int n=0; n < results.length; n++) {
>>
>> Task t = (Task)results[n];
>>
>> if (securityManager.canIViewTask(t))
>> secureResults.add(t);
>>
>> }
>>
>> return secureResults.toArray();
>>
>> }
>>
>> On 10/24/2011 11:50 AM, Nathaniel, Alfred wrote:
>>
>> Hi Paul,
>>
>> I don't think that it is a database issue.
>> It is rather the question where the application places
>> the large amounts of data and how it is cleaned up.
>>
>> I interprete your statements that it is placed in the
>> session object assuming that the memory is freed at
>> session timeout.
>> Unfortunately that is not the case.
>>
>> The session timeout is a security feature to force a new
>> login if the same user comes back after a longish idle time.
>> There is no guarantee that the container will actually
>> delete the session object at the session timeout.
>> As long as there is a reference to the session object GC
>> cannot free the attached memory.
>>
>> You will have to find a way that the application keeps
>> the data only for the duration of a request, or use
>> another mechanism to limit the memory requirements.
>>
>> HTH, Alfred.
>>
>> -----Original Message-----
>> From: Paul Joseph [mailto:pjoseph@gmail.com
>> <ma...@gmail.com>]
>> Sent: Montag, 24. Oktober 2011 17:16
>> To: users@cocoon.apache.org <ma...@cocoon.apache.org>
>> Subject: thought I had fixe it
>>
>> Hi there,
>>
>> I thought I had fixed this memory issue but...
>>
>> I am using Tomcat 6 in a Windows 32 bit environment
>> (Windows 2003) with
>> Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with
>> the latest
>> Postgres 9.1, JDBC 4 driver.
>>
>>
>> The behavior is this:
>>
>> The user fires of a LARGE query that returns 25,000 large
>> objects.
>>
>> The user repeats this 20 times within 5 minutes to show
>> me he can freeze
>> the app.
>>
>> On the 20th time, it says out of memory (heap space).
>>
>>
>> The JVM indicates that it is maxed to the limit specified
>> in Xms and
>> that there is only about 2MB of memory free.
>>
>> I then ask her to log off.
>>
>> The session time out is set to 20 minutes.
>>
>> But even after an hour, the memory is not reclaimed by
>> the JVM--it still
>> reports that only about 2MB is still free.
>>
>> Is the fact that it is not reclaiming memory an
>> indication of a memory leak?
>>
>> I am using the following settings in my repository.database:
>>
>> <jdbc-connection-descriptor
>> jcd-alias="WebApp"
>> default-connection="true"
>> platform="PostgreSQL"
>> jdbc-level="4.0"
>> driver="org.postgresql.Driver"
>> protocol="jdbc"
>> subprotocol="postgresql"
>> dbalias="//localhost:5432/WebApp"
>> username="******"
>> password="******"
>> eager-release="false"
>> batch-mode="false">
>>
>> <connection-pool maxActive="200" validationQuery="" />
>> <sequence-manager
>> className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
>> />
>> </jdbc-connection-descriptor>
>>
>>
>> Thanks much!
>> Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> users-unsubscribe@cocoon.apache.org
>> <ma...@cocoon.apache.org>
>> For additional commands, e-mail:
>> users-help@cocoon.apache.org
>> <ma...@cocoon.apache.org>
>>
>> The content of this e-mail is intended only for the
>> confidential use of the person addressed.
>> If you are not the intended recipient, please notify the
>> sender and delete this e-mail immediately.
>> Thank you.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> users-unsubscribe@cocoon.apache.org
>> <ma...@cocoon.apache.org>
>> For additional commands, e-mail:
>> users-help@cocoon.apache.org
>> <ma...@cocoon.apache.org>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>> <ma...@cocoon.apache.org>
>> For additional commands, e-mail: users-help@cocoon.apache.org
>> <ma...@cocoon.apache.org>
>>
>>
>
Re: thought I had fixe it
Posted by Jasha Joachimsthal <j....@onehippo.com>.
On 24 October 2011 18:29, Paul Joseph <pj...@gmail.com> wrote:
> Actually the real reason is that the user is using a query that required
> crafting to return 25K results. He is seeking to show the system "does not
> scale."
>
Then set a limit when building the query
>
> Actually I am using in my XML template, a cocoon flowscript repeater widget
> that does paging. But this is well after the Java logic has built the large
> array of results (the array results and the vector secureResults below)
>
The actual paging should be done on DB level, not after you've fetched the
enormous amount of results.
>
> On 10/24/2011 12:22 PM, Jasha Joachimsthal wrote:
>
> The first question that comes to my mind: why would you need 25000 objects
> as query result? Can't you do 1 query that only returns the total possible
> amount of objects and another query that returns e.g. the first 10 results
> and build in some paging logic for the next 10.
>
> Jasha Joachimsthal
>
> Europe - Amsterdam - Oosteinde 11, 1017 WT Amsterdam - +31(0)20 522 4466
> US - Boston - 1 Broadway, Cambridge, MA 02142 - +1 877 414 4776 (toll free)
>
> www.onehippo.com
>
>
> On 24 October 2011 18:13, Paul Joseph <pj...@gmail.com> wrote:
>
>> I think you hit the nail on the head.
>>
>> I assumed that the memory is freed at session time out. I did not realize
>> this was not the case.
>>
>> I know what the culprit is in this case--the array that I use to store the
>> objects returned from the query.
>>
>> I do this in two places--once in the business logic (java code) and once
>> in the flow script.
>>
>> The code in the java business logic is as follows--the item of interest is
>> a "Task". How would I optimize it in terms of memory collection?
>>
>> public Object[] getSearchResults() throws Exception {
>>
>> Object[] results = getSearchResultBeans(sql_query);
>>
>> Vector secureResults = new Vector();
>>
>> for (int n=0; n < results.length; n++) {
>>
>> Task t = (Task)results[n];
>>
>> if (securityManager.canIViewTask(t)) secureResults.add(t);
>>
>> }
>>
>> return secureResults.toArray();
>>
>> }
>>
>> On 10/24/2011 11:50 AM, Nathaniel, Alfred wrote:
>>
>>> Hi Paul,
>>>
>>> I don't think that it is a database issue.
>>> It is rather the question where the application places the large amounts
>>> of data and how it is cleaned up.
>>>
>>> I interprete your statements that it is placed in the session object
>>> assuming that the memory is freed at session timeout.
>>> Unfortunately that is not the case.
>>>
>>> The session timeout is a security feature to force a new login if the
>>> same user comes back after a longish idle time.
>>> There is no guarantee that the container will actually delete the session
>>> object at the session timeout.
>>> As long as there is a reference to the session object GC cannot free the
>>> attached memory.
>>>
>>> You will have to find a way that the application keeps the data only for
>>> the duration of a request, or use another mechanism to limit the memory
>>> requirements.
>>>
>>> HTH, Alfred.
>>>
>>> -----Original Message-----
>>> From: Paul Joseph [mailto:pjoseph@gmail.com]
>>> Sent: Montag, 24. Oktober 2011 17:16
>>> To: users@cocoon.apache.org
>>> Subject: thought I had fixe it
>>>
>>> Hi there,
>>>
>>> I thought I had fixed this memory issue but...
>>>
>>> I am using Tomcat 6 in a Windows 32 bit environment (Windows 2003) with
>>> Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
>>> Postgres 9.1, JDBC 4 driver.
>>>
>>>
>>> The behavior is this:
>>>
>>> The user fires of a LARGE query that returns 25,000 large objects.
>>>
>>> The user repeats this 20 times within 5 minutes to show me he can freeze
>>> the app.
>>>
>>> On the 20th time, it says out of memory (heap space).
>>>
>>>
>>> The JVM indicates that it is maxed to the limit specified in Xms and
>>> that there is only about 2MB of memory free.
>>>
>>> I then ask her to log off.
>>>
>>> The session time out is set to 20 minutes.
>>>
>>> But even after an hour, the memory is not reclaimed by the JVM--it still
>>> reports that only about 2MB is still free.
>>>
>>> Is the fact that it is not reclaiming memory an indication of a memory
>>> leak?
>>>
>>> I am using the following settings in my repository.database:
>>>
>>> <jdbc-connection-descriptor
>>> jcd-alias="WebApp"
>>> default-connection="true"
>>> platform="PostgreSQL"
>>> jdbc-level="4.0"
>>> driver="org.postgresql.Driver"
>>> protocol="jdbc"
>>> subprotocol="postgresql"
>>> dbalias="//localhost:5432/WebApp"
>>> username="******"
>>> password="******"
>>> eager-release="false"
>>> batch-mode="false">
>>>
>>> <connection-pool maxActive="200" validationQuery="" />
>>> <sequence-manager
>>>
>>> className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
>>> />
>>> </jdbc-connection-descriptor>
>>>
>>>
>>> Thanks much!
>>> Paul
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>> For additional commands, e-mail: users-help@cocoon.apache.org
>>>
>>> The content of this e-mail is intended only for the confidential use of
>>> the person addressed.
>>> If you are not the intended recipient, please notify the sender and
>>> delete this e-mail immediately.
>>> Thank you.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>> For additional commands, e-mail: users-help@cocoon.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>> For additional commands, e-mail: users-help@cocoon.apache.org
>>
>>
>
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
Actually the real reason is that the user is using a query that required
crafting to return 25K results. He is seeking to show the system "does
not scale."
Actually I am using in my XML template, a cocoon flowscript repeater
widget that does paging. But this is well after the Java logic has
built the large array of results (the array results and the vector
secureResults below)
On 10/24/2011 12:22 PM, Jasha Joachimsthal wrote:
> The first question that comes to my mind: why would you need 25000
> objects as query result? Can't you do 1 query that only returns the
> total possible amount of objects and another query that returns e.g.
> the first 10 results and build in some paging logic for the next 10.
>
> Jasha Joachimsthal
>
> Europe - Amsterdam - Oosteinde 11, 1017 WT Amsterdam - +31(0)20 522 4466
> US - Boston - 1 Broadway, Cambridge, MA 02142 - +1 877 414 4776 (toll
> free)
>
> www.onehippo.com <http://www.onehippo.com/>
>
>
> On 24 October 2011 18:13, Paul Joseph <pjoseph@gmail.com
> <ma...@gmail.com>> wrote:
>
> I think you hit the nail on the head.
>
> I assumed that the memory is freed at session time out. I did not
> realize this was not the case.
>
> I know what the culprit is in this case--the array that I use to
> store the objects returned from the query.
>
> I do this in two places--once in the business logic (java code)
> and once in the flow script.
>
> The code in the java business logic is as follows--the item of
> interest is a "Task". How would I optimize it in terms of memory
> collection?
>
> public Object[] getSearchResults() throws Exception {
>
> Object[] results = getSearchResultBeans(sql_query);
>
> Vector secureResults = new Vector();
>
> for (int n=0; n < results.length; n++) {
>
> Task t = (Task)results[n];
>
> if (securityManager.canIViewTask(t)) secureResults.add(t);
>
> }
>
> return secureResults.toArray();
>
> }
>
> On 10/24/2011 11:50 AM, Nathaniel, Alfred wrote:
>
> Hi Paul,
>
> I don't think that it is a database issue.
> It is rather the question where the application places the
> large amounts of data and how it is cleaned up.
>
> I interprete your statements that it is placed in the session
> object assuming that the memory is freed at session timeout.
> Unfortunately that is not the case.
>
> The session timeout is a security feature to force a new login
> if the same user comes back after a longish idle time.
> There is no guarantee that the container will actually delete
> the session object at the session timeout.
> As long as there is a reference to the session object GC
> cannot free the attached memory.
>
> You will have to find a way that the application keeps the
> data only for the duration of a request, or use another
> mechanism to limit the memory requirements.
>
> HTH, Alfred.
>
> -----Original Message-----
> From: Paul Joseph [mailto:pjoseph@gmail.com
> <ma...@gmail.com>]
> Sent: Montag, 24. Oktober 2011 17:16
> To: users@cocoon.apache.org <ma...@cocoon.apache.org>
> Subject: thought I had fixe it
>
> Hi there,
>
> I thought I had fixed this memory issue but...
>
> I am using Tomcat 6 in a Windows 32 bit environment (Windows
> 2003) with
> Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
> Postgres 9.1, JDBC 4 driver.
>
>
> The behavior is this:
>
> The user fires of a LARGE query that returns 25,000 large objects.
>
> The user repeats this 20 times within 5 minutes to show me he
> can freeze
> the app.
>
> On the 20th time, it says out of memory (heap space).
>
>
> The JVM indicates that it is maxed to the limit specified in
> Xms and
> that there is only about 2MB of memory free.
>
> I then ask her to log off.
>
> The session time out is set to 20 minutes.
>
> But even after an hour, the memory is not reclaimed by the
> JVM--it still
> reports that only about 2MB is still free.
>
> Is the fact that it is not reclaiming memory an indication of
> a memory leak?
>
> I am using the following settings in my repository.database:
>
> <jdbc-connection-descriptor
> jcd-alias="WebApp"
> default-connection="true"
> platform="PostgreSQL"
> jdbc-level="4.0"
> driver="org.postgresql.Driver"
> protocol="jdbc"
> subprotocol="postgresql"
> dbalias="//localhost:5432/WebApp"
> username="******"
> password="******"
> eager-release="false"
> batch-mode="false">
>
> <connection-pool maxActive="200" validationQuery="" />
> <sequence-manager
> className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
> />
> </jdbc-connection-descriptor>
>
>
> Thanks much!
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> <ma...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
> <ma...@cocoon.apache.org>
>
> The content of this e-mail is intended only for the
> confidential use of the person addressed.
> If you are not the intended recipient, please notify the
> sender and delete this e-mail immediately.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> <ma...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
> <ma...@cocoon.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> <ma...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
> <ma...@cocoon.apache.org>
>
>
Re: thought I had fixe it
Posted by Jasha Joachimsthal <j....@onehippo.com>.
The first question that comes to my mind: why would you need 25000 objects
as query result? Can't you do 1 query that only returns the total possible
amount of objects and another query that returns e.g. the first 10 results
and build in some paging logic for the next 10.
Jasha Joachimsthal
Europe - Amsterdam - Oosteinde 11, 1017 WT Amsterdam - +31(0)20 522 4466
US - Boston - 1 Broadway, Cambridge, MA 02142 - +1 877 414 4776 (toll free)
www.onehippo.com
On 24 October 2011 18:13, Paul Joseph <pj...@gmail.com> wrote:
> I think you hit the nail on the head.
>
> I assumed that the memory is freed at session time out. I did not realize
> this was not the case.
>
> I know what the culprit is in this case--the array that I use to store the
> objects returned from the query.
>
> I do this in two places--once in the business logic (java code) and once in
> the flow script.
>
> The code in the java business logic is as follows--the item of interest is
> a "Task". How would I optimize it in terms of memory collection?
>
> public Object[] getSearchResults() throws Exception {
>
> Object[] results = getSearchResultBeans(sql_**query);
>
> Vector secureResults = new Vector();
>
> for (int n=0; n < results.length; n++) {
>
> Task t = (Task)results[n];
>
> if (securityManager.canIViewTask(**t)) secureResults.add(t);
>
> }
>
> return secureResults.toArray();
>
> }
>
> On 10/24/2011 11:50 AM, Nathaniel, Alfred wrote:
>
>> Hi Paul,
>>
>> I don't think that it is a database issue.
>> It is rather the question where the application places the large amounts
>> of data and how it is cleaned up.
>>
>> I interprete your statements that it is placed in the session object
>> assuming that the memory is freed at session timeout.
>> Unfortunately that is not the case.
>>
>> The session timeout is a security feature to force a new login if the same
>> user comes back after a longish idle time.
>> There is no guarantee that the container will actually delete the session
>> object at the session timeout.
>> As long as there is a reference to the session object GC cannot free the
>> attached memory.
>>
>> You will have to find a way that the application keeps the data only for
>> the duration of a request, or use another mechanism to limit the memory
>> requirements.
>>
>> HTH, Alfred.
>>
>> -----Original Message-----
>> From: Paul Joseph [mailto:pjoseph@gmail.com]
>> Sent: Montag, 24. Oktober 2011 17:16
>> To: users@cocoon.apache.org
>> Subject: thought I had fixe it
>>
>> Hi there,
>>
>> I thought I had fixed this memory issue but...
>>
>> I am using Tomcat 6 in a Windows 32 bit environment (Windows 2003) with
>> Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
>> Postgres 9.1, JDBC 4 driver.
>>
>>
>> The behavior is this:
>>
>> The user fires of a LARGE query that returns 25,000 large objects.
>>
>> The user repeats this 20 times within 5 minutes to show me he can freeze
>> the app.
>>
>> On the 20th time, it says out of memory (heap space).
>>
>>
>> The JVM indicates that it is maxed to the limit specified in Xms and
>> that there is only about 2MB of memory free.
>>
>> I then ask her to log off.
>>
>> The session time out is set to 20 minutes.
>>
>> But even after an hour, the memory is not reclaimed by the JVM--it still
>> reports that only about 2MB is still free.
>>
>> Is the fact that it is not reclaiming memory an indication of a memory
>> leak?
>>
>> I am using the following settings in my repository.database:
>>
>> <jdbc-connection-descriptor
>> jcd-alias="WebApp"
>> default-connection="true"
>> platform="PostgreSQL"
>> jdbc-level="4.0"
>> driver="org.postgresql.Driver"
>> protocol="jdbc"
>> subprotocol="postgresql"
>> dbalias="//localhost:5432/**WebApp"
>> username="******"
>> password="******"
>> eager-release="false"
>> batch-mode="false">
>>
>> <connection-pool maxActive="200" validationQuery="" />
>> <sequence-manager
>> className="org.apache.ojb.**broker.util.sequence.**
>> SequenceManagerNextValImpl"
>> />
>> </jdbc-connection-descriptor>
>>
>>
>> Thanks much!
>> Paul
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: users-unsubscribe@cocoon.**apache.org<us...@cocoon.apache.org>
>> For additional commands, e-mail: users-help@cocoon.apache.org
>>
>> The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> If you are not the intended recipient, please notify the sender and delete
>> this e-mail immediately.
>> Thank you.
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: users-unsubscribe@cocoon.**apache.org<us...@cocoon.apache.org>
>> For additional commands, e-mail: users-help@cocoon.apache.org
>>
>>
>>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.**apache.org<us...@cocoon.apache.org>
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
Re: thought I had fixe it
Posted by Paul Joseph <pj...@gmail.com>.
I think you hit the nail on the head.
I assumed that the memory is freed at session time out. I did not
realize this was not the case.
I know what the culprit is in this case--the array that I use to store
the objects returned from the query.
I do this in two places--once in the business logic (java code) and once
in the flow script.
The code in the java business logic is as follows--the item of interest
is a "Task". How would I optimize it in terms of memory collection?
public Object[] getSearchResults() throws Exception {
Object[] results = getSearchResultBeans(sql_query);
Vector secureResults = new Vector();
for (int n=0; n < results.length; n++) {
Task t = (Task)results[n];
if (securityManager.canIViewTask(t)) secureResults.add(t);
}
return secureResults.toArray();
}
On 10/24/2011 11:50 AM, Nathaniel, Alfred wrote:
> Hi Paul,
>
> I don't think that it is a database issue.
> It is rather the question where the application places the large amounts of data and how it is cleaned up.
>
> I interprete your statements that it is placed in the session object assuming that the memory is freed at session timeout.
> Unfortunately that is not the case.
>
> The session timeout is a security feature to force a new login if the same user comes back after a longish idle time.
> There is no guarantee that the container will actually delete the session object at the session timeout.
> As long as there is a reference to the session object GC cannot free the attached memory.
>
> You will have to find a way that the application keeps the data only for the duration of a request, or use another mechanism to limit the memory requirements.
>
> HTH, Alfred.
>
> -----Original Message-----
> From: Paul Joseph [mailto:pjoseph@gmail.com]
> Sent: Montag, 24. Oktober 2011 17:16
> To: users@cocoon.apache.org
> Subject: thought I had fixe it
>
> Hi there,
>
> I thought I had fixed this memory issue but...
>
> I am using Tomcat 6 in a Windows 32 bit environment (Windows 2003) with
> Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
> Postgres 9.1, JDBC 4 driver.
>
>
> The behavior is this:
>
> The user fires of a LARGE query that returns 25,000 large objects.
>
> The user repeats this 20 times within 5 minutes to show me he can freeze
> the app.
>
> On the 20th time, it says out of memory (heap space).
>
>
> The JVM indicates that it is maxed to the limit specified in Xms and
> that there is only about 2MB of memory free.
>
> I then ask her to log off.
>
> The session time out is set to 20 minutes.
>
> But even after an hour, the memory is not reclaimed by the JVM--it still
> reports that only about 2MB is still free.
>
> Is the fact that it is not reclaiming memory an indication of a memory leak?
>
> I am using the following settings in my repository.database:
>
> <jdbc-connection-descriptor
> jcd-alias="WebApp"
> default-connection="true"
> platform="PostgreSQL"
> jdbc-level="4.0"
> driver="org.postgresql.Driver"
> protocol="jdbc"
> subprotocol="postgresql"
> dbalias="//localhost:5432/WebApp"
> username="******"
> password="******"
> eager-release="false"
> batch-mode="false">
>
> <connection-pool maxActive="200" validationQuery="" />
> <sequence-manager
> className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
> />
> </jdbc-connection-descriptor>
>
>
> Thanks much!
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
> The content of this e-mail is intended only for the confidential use of the person addressed.
> If you are not the intended recipient, please notify the sender and delete this e-mail immediately.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: thought I had fixe it
Posted by "Nathaniel, Alfred" <Al...@six-group.com>.
Hi Paul,
I don't think that it is a database issue.
It is rather the question where the application places the large amounts of data and how it is cleaned up.
I interprete your statements that it is placed in the session object assuming that the memory is freed at session timeout.
Unfortunately that is not the case.
The session timeout is a security feature to force a new login if the same user comes back after a longish idle time.
There is no guarantee that the container will actually delete the session object at the session timeout.
As long as there is a reference to the session object GC cannot free the attached memory.
You will have to find a way that the application keeps the data only for the duration of a request, or use another mechanism to limit the memory requirements.
HTH, Alfred.
-----Original Message-----
From: Paul Joseph [mailto:pjoseph@gmail.com]
Sent: Montag, 24. Oktober 2011 17:16
To: users@cocoon.apache.org
Subject: thought I had fixe it
Hi there,
I thought I had fixed this memory issue but...
I am using Tomcat 6 in a Windows 32 bit environment (Windows 2003) with
Cocoon 2.1.11 and Java 1.6, agains Postgresql 8.4 with the latest
Postgres 9.1, JDBC 4 driver.
The behavior is this:
The user fires of a LARGE query that returns 25,000 large objects.
The user repeats this 20 times within 5 minutes to show me he can freeze
the app.
On the 20th time, it says out of memory (heap space).
The JVM indicates that it is maxed to the limit specified in Xms and
that there is only about 2MB of memory free.
I then ask her to log off.
The session time out is set to 20 minutes.
But even after an hour, the memory is not reclaimed by the JVM--it still
reports that only about 2MB is still free.
Is the fact that it is not reclaiming memory an indication of a memory leak?
I am using the following settings in my repository.database:
<jdbc-connection-descriptor
jcd-alias="WebApp"
default-connection="true"
platform="PostgreSQL"
jdbc-level="4.0"
driver="org.postgresql.Driver"
protocol="jdbc"
subprotocol="postgresql"
dbalias="//localhost:5432/WebApp"
username="******"
password="******"
eager-release="false"
batch-mode="false">
<connection-pool maxActive="200" validationQuery="" />
<sequence-manager
className="org.apache.ojb.broker.util.sequence.SequenceManagerNextValImpl"
/>
</jdbc-connection-descriptor>
Thanks much!
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
The content of this e-mail is intended only for the confidential use of the person addressed.
If you are not the intended recipient, please notify the sender and delete this e-mail immediately.
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org