You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tapestry.apache.org by "Blower, Andy" <An...@proquest.co.uk> on 2010/06/29 17:48:04 UTC
RE: Tapestry using 1.3Gb of heap space after capacity testing
I doubt anyone remembers this thread except me, but we're still having problems that I could do with help on.
So, two months later, we have managed to improve things significantly, but still have very large pages and we're having a lot of trouble with the page pool size. What we see when we push the app too hard is that the heap fills until the gc thread is permanently running, and the app is essentially dead and unresponsive. I've been testing with 2g & 4g heaps up to now, with 100ms soft-wait & 200 hard limit because we need a lot of instances of some pages, and were running out with a 100 limit. I know I need to understand and do more with these values, but it's worrying because our tests are only requesting English pages at present and we'll have 17 languages for launch! So I'm rather concerned to say the least...
I have a theory about what is happening that I'd like to run by the list and see what others think. I'm having some trouble figuring out what's happening at load.
What I think is that when the heap gets full-ish at high load, the gc kicks in more and more often, slowing request processing down a little. (there's a fair amount of soft referenced stuff it can boot) This means that pages are returned to the page pool a bit later causing more page instances to be created which fills the heap and enters an vicious cycle with the gc trying to free memory and the T5 page pool trying to create more page instances. The reason I suspect this is because heap dumps show about 40 PageImpl's that have not completed loading, and many more that have but are much smaller in size than others for the same page. (e.g. a Results PageImpl retaining 28k when a normal one retains 7.5Mb - very big I know, but I've cut it down from 14Mb)
Our site has 223 pages which consume 213Mb if 1 instance of each is instantiated. Multiply this by 200 (hard limit) and then by 17 languages and worst case we'll need a heap size of 725Gb which is a little ridiculous! I will do more work on reducing the size of the pages but I've already done the easy stuff.
Here's a list of the heap dominators using Eclipse MAT, http://lh6.ggpht.com/_YwJn8TJTqJU/TCoNT11ouZI/AAAAAAAAACI/G8eTGsth4zM/HeapDominators.png
Anyone think I'm on the right track, or barking up the wrong tree completely?
Thanks,
Andy
> -----Original Message-----
> From: Blower, Andy [mailto:Andy.Blower@proquest.co.uk]
> Sent: 20 April 2010 17:42
> To: 'Tapestry users'
> Subject: RE: Tapestry using 1.3Gb of heap space after capacity testing
>
> Thanks for the link. As I said before, I don't think that we've fallen
> prey to that exactly. Doing some local testing and looking at one of
> our largest pages which displays search results, it has 76 component
> objects, each of those will have components nested within (pretty
> heavily in some cases) but nothing which has a bunch of components
> where only one is used and the others are redundant. (what I think is
> this uber-component thing)
>
> It looks like explicit repeated use of a component, rather than using a
> single component in a loop, creates a lot more component objects in the
> heap. I've never seen anything warning against this, and we do usually
> use loops but it's not always the best solution.
>
> Again, looking at our results page, a single instance seems to get
> 11,283 ComponentPageElementImpl instances created for it. I'm finding
> it hard to find a good view of the tree to figure out what component(s)
> are the main culprits for this huge number of instances.
>
> Thanks for any help or guidance you can give me.
>
> Andy
>
> > -----Original Message-----
> > From: Thiago H. de Paula Figueiredo [mailto:thiagohp@gmail.com]
> > Sent: 20 April 2010 15:56
> > To: Tapestry users
> > Subject: Re: Tapestry using 1.3Gb of heap space after capacity
> testing
> >
> > On Tue, 20 Apr 2010 11:41:25 -0300, Blower, Andy
> > <An...@proquest.co.uk> wrote:
> >
> > > Uber-component anti-pattern, not to my knowledge, but we have a lot
> > of
> > > pages and components now. And a lot of logic in them. The project's
> > > large enough I can't be sure. Do you have a handy link to Howard's
> > > description so I can get my team to check?
> >
> > It's here:
> > http://old.nabble.com/-T5.0.18--Out-of-Memory-Error---Potential-Leak-
> > (doesn't-reduce-after-forced-GC)-to25403474s302.html#a25497441
> > That thread is interesting, by the way. The problem is more about how
> > components and pages are written than the number of them.
> > What Tapestry version are you using?
> >
> > > Page pool config is default, I don't know enough to fiddle yet.
> Also,
> > > the test is on a single stand alone server with no clustering, we
> are
> > > using Tomcat clustering for production.
> >
> > You should take a look a them, as they can affect the memory
> > consumption
> > directly.
> >
> > > I did fire the GC manually before getting concerned BTW. A lot of
> > heap
> > > seems to be taken up with 100-300k descriptions on the
> > > InternalClassTransformationImpl class. An example of one is
> attached
> > > below - seems wrong to me and more like the debug that can be
> > requested
> > > through log4j rather than a description.
> >
> > They look really bigger than they should. 300k is a lot. I have one
> > application with a reasonable number of pages and components and it
> > runs
> > inside a 512 MB slide at SliceHost without using the whole memory.
> >
> > --
> > Thiago H. de Paula Figueiredo
> > Independent Java, Apache Tapestry 5 and Hibernate consultant,
> > developer,
> > and instructor
> > Owner, software architect and developer, Ars Machina Tecnologia da
> > Informação Ltda.
> > http://www.arsmachina.com.br
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> > For additional commands, e-mail: users-help@tapestry.apache.org
> >
Re: Tapestry using 1.3Gb of heap space after capacity testing
Posted by Pierce Wetter <pi...@paceap.com>.
>
>
> Anyone think I'm on the right track, or barking up the wrong tree completely?
Here's a completely obscure question out of left field. Are you launching the JVM with -server?
Here's a pretty useful blog post:
http://jfarcand.wordpress.com/2009/11/27/putting-glassfish-v3-in-production-essential-surviving-guide/
There's lots of interesting meaty-but-obscure command line options to experiment with.
pierce
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Tapestry using 1.3Gb of heap space after capacity testing
Posted by Howard Lewis Ship <hl...@gmail.com>.
On Wed, Jun 30, 2010 at 9:37 AM, Blower, Andy
<An...@proquest.co.uk> wrote:
>> -----Original Message-----
>> From: Howard Lewis Ship [mailto:hlship@gmail.com]
>> Sent: 29 June 2010 18:20
>> To: Tapestry users
>> Subject: Re: Tapestry using 1.3Gb of heap space after capacity testing
>>
>> It may be time to return to a more radical idea, one that is more
>> technically feasible now (in release 5.2) than it was in the past.
>
> Why more feasible in 5.2? We're using 5.1.0.5, and are now too close to release to upgrade. I was hoping to move to 5.2 when I thought it would be released in H1 2010, but we'll need to stick with 5.1 now I think. We launch in August.
>
>> Get rid of page pooling.
>>
>> I'm not saying to re-create each page for each request; I don't think
>> that would scale.
>>
>> However, it may be possible to change Tapestry so that you need only
>> ONE instance of a given page (per locale). All transient (per-request)
>> state for the page could be accessed indirectly, stored in a
>> per-thread object (or inside the Request, as attributes).
>>
>> I can envision some small amount of extra overhead per request (due to
>> extra levels of indirection).
>>
>> However, you could then process any number of threads against the
>> same, single page object with no extra memory consumption: for
>> example, no more duplicated Binding objects, and no more extra Maps to
>> hold them all.
>>
>> Some parts of the Tapestry's internal implementation
>> (InternalComponentResourcesImpl and ComponentPageElementImpl) would
>> need some changing, i.e., a bit more synchronization around some
>> critical sections.
>>
>> It's an exciting idea ... when will I have time to investigate it?
>
> It certainly is an exciting idea, out of curiosity How long do you think it would take you to implement this - it sounds like a pretty big overhaul to me.
It's not so large an overhaul as you might think. My big concern is
that the change in semantics (not the change in API) might cause
existing ComponentClassTransformationWorkers (contributed by 3rd party
libraries) to no longer operate correctly.
>
> Anything else that I could look into? Am I on the right track?
>
>
>> -----Original Message-----
>> From: Christophe Cordenier [mailto:christophe.cordenier@gmail.com]
>> Sent: 29 June 2010 19:35
>> To: Tapestry users
>> Subject: Re: Tapestry using 1.3Gb of heap space after capacity testing
>>
>> Holy grail ! but is it really feasible...
>>
>> @Andy Sorry i didn't follow all the thread but if i understand well, you
>> have page instance of 7.5Mb for one single instance ? This is only due to
>> page and component structure ?
>
> That's correct I believe - it used to be 14Mb. Our largest page is 10Mb and we have 13 pages with over 4Mb retained size from their PageImpl's.
>
> I'm not sure how this compares with the average, but I suspect they are a bit on the large size.
>
> Andy
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
--
Howard M. Lewis Ship
Creator of Apache Tapestry
The source for Tapestry training, mentoring and support. Contact me to
learn how I can get you up and productive in Tapestry fast!
(971) 678-5210
http://howardlewisship.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Tapestry using 1.3Gb of heap space after capacity testing
Posted by "Thiago H. de Paula Figueiredo" <th...@gmail.com>.
On Wed, 30 Jun 2010 13:37:54 -0300, Blower, Andy
<An...@proquest.co.uk> wrote:
> Why more feasible in 5.2? We're using 5.1.0.5, and are now too close to
> release to upgrade. I was hoping to move to 5.2 when I thought it would
> be released in H1 2010, but we'll need to stick with 5.1 now I think. We
> launch in August.
I guess because of the changes on how class transformations are done.
--
Thiago H. de Paula Figueiredo
Independent Java, Apache Tapestry 5 and Hibernate consultant, developer,
and instructor
Owner, Ars Machina Tecnologia da Informação Ltda.
http://www.arsmachina.com.br
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
RE: Tapestry using 1.3Gb of heap space after capacity testing
Posted by "Blower, Andy" <An...@proquest.co.uk>.
> -----Original Message-----
> From: Howard Lewis Ship [mailto:hlship@gmail.com]
> Sent: 29 June 2010 18:20
> To: Tapestry users
> Subject: Re: Tapestry using 1.3Gb of heap space after capacity testing
>
> It may be time to return to a more radical idea, one that is more
> technically feasible now (in release 5.2) than it was in the past.
Why more feasible in 5.2? We're using 5.1.0.5, and are now too close to release to upgrade. I was hoping to move to 5.2 when I thought it would be released in H1 2010, but we'll need to stick with 5.1 now I think. We launch in August.
> Get rid of page pooling.
>
> I'm not saying to re-create each page for each request; I don't think
> that would scale.
>
> However, it may be possible to change Tapestry so that you need only
> ONE instance of a given page (per locale). All transient (per-request)
> state for the page could be accessed indirectly, stored in a
> per-thread object (or inside the Request, as attributes).
>
> I can envision some small amount of extra overhead per request (due to
> extra levels of indirection).
>
> However, you could then process any number of threads against the
> same, single page object with no extra memory consumption: for
> example, no more duplicated Binding objects, and no more extra Maps to
> hold them all.
>
> Some parts of the Tapestry's internal implementation
> (InternalComponentResourcesImpl and ComponentPageElementImpl) would
> need some changing, i.e., a bit more synchronization around some
> critical sections.
>
> It's an exciting idea ... when will I have time to investigate it?
It certainly is an exciting idea, out of curiosity How long do you think it would take you to implement this - it sounds like a pretty big overhaul to me.
Anything else that I could look into? Am I on the right track?
> -----Original Message-----
> From: Christophe Cordenier [mailto:christophe.cordenier@gmail.com]
> Sent: 29 June 2010 19:35
> To: Tapestry users
> Subject: Re: Tapestry using 1.3Gb of heap space after capacity testing
>
> Holy grail ! but is it really feasible...
>
> @Andy Sorry i didn't follow all the thread but if i understand well, you
> have page instance of 7.5Mb for one single instance ? This is only due to
> page and component structure ?
That's correct I believe - it used to be 14Mb. Our largest page is 10Mb and we have 13 pages with over 4Mb retained size from their PageImpl's.
I'm not sure how this compares with the average, but I suspect they are a bit on the large size.
Andy
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Tapestry using 1.3Gb of heap space after capacity testing
Posted by Christophe Cordenier <ch...@gmail.com>.
Holy grail ! but is it really feasible...
@Andy Sorry i didn't follow all the thread but if i understand well, you
have page instance of 7.5Mb for one single instance ? This is only due to
page and component structure ?
2010/6/29 Howard Lewis Ship <hl...@gmail.com>
> It may be time to return to a more radical idea, one that is more
> technically feasible now (in release 5.2) than it was in the past.
>
> Get rid of page pooling.
>
> I'm not saying to re-create each page for each request; I don't think
> that would scale.
>
> However, it may be possible to change Tapestry so that you need only
> ONE instance of a given page (per locale). All transient (per-request)
> state for the page could be accessed indirectly, stored in a
> per-thread object (or inside the Request, as attributes).
>
> I can envision some small amount of extra overhead per request (due to
> extra levels of indirection).
>
> However, you could then process any number of threads against the
> same, single page object with no extra memory consumption: for
> example, no more duplicated Binding objects, and no more extra Maps to
> hold them all.
>
> Some parts of the Tapestry's internal implementation
> (InternalComponentResourcesImpl and ComponentPageElementImpl) would
> need some changing, i.e., a bit more synchronization around some
> critical sections.
>
> It's an exciting idea ... when will I have time to investigate it?
>
> On Tue, Jun 29, 2010 at 10:02 AM, Howard Lewis Ship <hl...@gmail.com>
> wrote:
> > Just to check ... which version of Tapestry?
> >
> > On Tue, Jun 29, 2010 at 8:48 AM, Blower, Andy
> > <An...@proquest.co.uk> wrote:
> >> I doubt anyone remembers this thread except me, but we're still having
> problems that I could do with help on.
> >>
> >> So, two months later, we have managed to improve things significantly,
> but still have very large pages and we're having a lot of trouble with the
> page pool size. What we see when we push the app too hard is that the heap
> fills until the gc thread is permanently running, and the app is essentially
> dead and unresponsive. I've been testing with 2g & 4g heaps up to now, with
> 100ms soft-wait & 200 hard limit because we need a lot of instances of some
> pages, and were running out with a 100 limit. I know I need to understand
> and do more with these values, but it's worrying because our tests are only
> requesting English pages at present and we'll have 17 languages for launch!
> So I'm rather concerned to say the least...
> >>
> >> I have a theory about what is happening that I'd like to run by the list
> and see what others think. I'm having some trouble figuring out what's
> happening at load.
> >>
> >> What I think is that when the heap gets full-ish at high load, the gc
> kicks in more and more often, slowing request processing down a little.
> (there's a fair amount of soft referenced stuff it can boot) This means that
> pages are returned to the page pool a bit later causing more page instances
> to be created which fills the heap and enters an vicious cycle with the gc
> trying to free memory and the T5 page pool trying to create more page
> instances. The reason I suspect this is because heap dumps show about 40
> PageImpl's that have not completed loading, and many more that have but are
> much smaller in size than others for the same page. (e.g. a Results PageImpl
> retaining 28k when a normal one retains 7.5Mb - very big I know, but I've
> cut it down from 14Mb)
> >>
> >> Our site has 223 pages which consume 213Mb if 1 instance of each is
> instantiated. Multiply this by 200 (hard limit) and then by 17 languages and
> worst case we'll need a heap size of 725Gb which is a little ridiculous! I
> will do more work on reducing the size of the pages but I've already done
> the easy stuff.
> >>
> >> Here's a list of the heap dominators using Eclipse MAT,
> http://lh6.ggpht.com/_YwJn8TJTqJU/TCoNT11ouZI/AAAAAAAAACI/G8eTGsth4zM/HeapDominators.png
> >>
> >> Anyone think I'm on the right track, or barking up the wrong tree
> completely?
> >>
> >> Thanks,
> >>
> >> Andy
> >>
> >>> -----Original Message-----
> >>> From: Blower, Andy [mailto:Andy.Blower@proquest.co.uk]
> >>> Sent: 20 April 2010 17:42
> >>> To: 'Tapestry users'
> >>> Subject: RE: Tapestry using 1.3Gb of heap space after capacity testing
> >>>
> >>> Thanks for the link. As I said before, I don't think that we've fallen
> >>> prey to that exactly. Doing some local testing and looking at one of
> >>> our largest pages which displays search results, it has 76 component
> >>> objects, each of those will have components nested within (pretty
> >>> heavily in some cases) but nothing which has a bunch of components
> >>> where only one is used and the others are redundant. (what I think is
> >>> this uber-component thing)
> >>>
> >>> It looks like explicit repeated use of a component, rather than using a
> >>> single component in a loop, creates a lot more component objects in the
> >>> heap. I've never seen anything warning against this, and we do usually
> >>> use loops but it's not always the best solution.
> >>>
> >>> Again, looking at our results page, a single instance seems to get
> >>> 11,283 ComponentPageElementImpl instances created for it. I'm finding
> >>> it hard to find a good view of the tree to figure out what component(s)
> >>> are the main culprits for this huge number of instances.
> >>>
> >>> Thanks for any help or guidance you can give me.
> >>>
> >>> Andy
> >>>
> >>> > -----Original Message-----
> >>> > From: Thiago H. de Paula Figueiredo [mailto:thiagohp@gmail.com]
> >>> > Sent: 20 April 2010 15:56
> >>> > To: Tapestry users
> >>> > Subject: Re: Tapestry using 1.3Gb of heap space after capacity
> >>> testing
> >>> >
> >>> > On Tue, 20 Apr 2010 11:41:25 -0300, Blower, Andy
> >>> > <An...@proquest.co.uk> wrote:
> >>> >
> >>> > > Uber-component anti-pattern, not to my knowledge, but we have a lot
> >>> > of
> >>> > > pages and components now. And a lot of logic in them. The project's
> >>> > > large enough I can't be sure. Do you have a handy link to Howard's
> >>> > > description so I can get my team to check?
> >>> >
> >>> > It's here:
> >>> >
> http://old.nabble.com/-T5.0.18--Out-of-Memory-Error---Potential-Leak-
> >>> > (doesn't-reduce-after-forced-GC)-to25403474s302.html#a25497441
> >>> > That thread is interesting, by the way. The problem is more about how
> >>> > components and pages are written than the number of them.
> >>> > What Tapestry version are you using?
> >>> >
> >>> > > Page pool config is default, I don't know enough to fiddle yet.
> >>> Also,
> >>> > > the test is on a single stand alone server with no clustering, we
> >>> are
> >>> > > using Tomcat clustering for production.
> >>> >
> >>> > You should take a look a them, as they can affect the memory
> >>> > consumption
> >>> > directly.
> >>> >
> >>> > > I did fire the GC manually before getting concerned BTW. A lot of
> >>> > heap
> >>> > > seems to be taken up with 100-300k descriptions on the
> >>> > > InternalClassTransformationImpl class. An example of one is
> >>> attached
> >>> > > below - seems wrong to me and more like the debug that can be
> >>> > requested
> >>> > > through log4j rather than a description.
> >>> >
> >>> > They look really bigger than they should. 300k is a lot. I have one
> >>> > application with a reasonable number of pages and components and it
> >>> > runs
> >>> > inside a 512 MB slide at SliceHost without using the whole memory.
> >>> >
> >>> > --
> >>> > Thiago H. de Paula Figueiredo
> >>> > Independent Java, Apache Tapestry 5 and Hibernate consultant,
> >>> > developer,
> >>> > and instructor
> >>> > Owner, software architect and developer, Ars Machina Tecnologia da
> >>> > Informação Ltda.
> >>> > http://www.arsmachina.com.br
> >>> >
> >>> > ---------------------------------------------------------------------
> >>> > To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> >>> > For additional commands, e-mail: users-help@tapestry.apache.org
> >>> >
> >>
> >>
> >
> >
> >
> > --
> > Howard M. Lewis Ship
> >
> > Creator of Apache Tapestry
> >
> > The source for Tapestry training, mentoring and support. Contact me to
> > learn how I can get you up and productive in Tapestry fast!
> >
> > (971) 678-5210
> > http://howardlewisship.com
> >
>
>
>
> --
> Howard M. Lewis Ship
>
> Creator of Apache Tapestry
>
> The source for Tapestry training, mentoring and support. Contact me to
> learn how I can get you up and productive in Tapestry fast!
>
> (971) 678-5210
> http://howardlewisship.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
--
Regards,
Christophe Cordenier.
Committer on Apache Tapestry 5
Co-creator of wooki @wookicentral.com
Re: Tapestry using 1.3Gb of heap space after capacity testing
Posted by Howard Lewis Ship <hl...@gmail.com>.
It may be time to return to a more radical idea, one that is more
technically feasible now (in release 5.2) than it was in the past.
Get rid of page pooling.
I'm not saying to re-create each page for each request; I don't think
that would scale.
However, it may be possible to change Tapestry so that you need only
ONE instance of a given page (per locale). All transient (per-request)
state for the page could be accessed indirectly, stored in a
per-thread object (or inside the Request, as attributes).
I can envision some small amount of extra overhead per request (due to
extra levels of indirection).
However, you could then process any number of threads against the
same, single page object with no extra memory consumption: for
example, no more duplicated Binding objects, and no more extra Maps to
hold them all.
Some parts of the Tapestry's internal implementation
(InternalComponentResourcesImpl and ComponentPageElementImpl) would
need some changing, i.e., a bit more synchronization around some
critical sections.
It's an exciting idea ... when will I have time to investigate it?
On Tue, Jun 29, 2010 at 10:02 AM, Howard Lewis Ship <hl...@gmail.com> wrote:
> Just to check ... which version of Tapestry?
>
> On Tue, Jun 29, 2010 at 8:48 AM, Blower, Andy
> <An...@proquest.co.uk> wrote:
>> I doubt anyone remembers this thread except me, but we're still having problems that I could do with help on.
>>
>> So, two months later, we have managed to improve things significantly, but still have very large pages and we're having a lot of trouble with the page pool size. What we see when we push the app too hard is that the heap fills until the gc thread is permanently running, and the app is essentially dead and unresponsive. I've been testing with 2g & 4g heaps up to now, with 100ms soft-wait & 200 hard limit because we need a lot of instances of some pages, and were running out with a 100 limit. I know I need to understand and do more with these values, but it's worrying because our tests are only requesting English pages at present and we'll have 17 languages for launch! So I'm rather concerned to say the least...
>>
>> I have a theory about what is happening that I'd like to run by the list and see what others think. I'm having some trouble figuring out what's happening at load.
>>
>> What I think is that when the heap gets full-ish at high load, the gc kicks in more and more often, slowing request processing down a little. (there's a fair amount of soft referenced stuff it can boot) This means that pages are returned to the page pool a bit later causing more page instances to be created which fills the heap and enters an vicious cycle with the gc trying to free memory and the T5 page pool trying to create more page instances. The reason I suspect this is because heap dumps show about 40 PageImpl's that have not completed loading, and many more that have but are much smaller in size than others for the same page. (e.g. a Results PageImpl retaining 28k when a normal one retains 7.5Mb - very big I know, but I've cut it down from 14Mb)
>>
>> Our site has 223 pages which consume 213Mb if 1 instance of each is instantiated. Multiply this by 200 (hard limit) and then by 17 languages and worst case we'll need a heap size of 725Gb which is a little ridiculous! I will do more work on reducing the size of the pages but I've already done the easy stuff.
>>
>> Here's a list of the heap dominators using Eclipse MAT, http://lh6.ggpht.com/_YwJn8TJTqJU/TCoNT11ouZI/AAAAAAAAACI/G8eTGsth4zM/HeapDominators.png
>>
>> Anyone think I'm on the right track, or barking up the wrong tree completely?
>>
>> Thanks,
>>
>> Andy
>>
>>> -----Original Message-----
>>> From: Blower, Andy [mailto:Andy.Blower@proquest.co.uk]
>>> Sent: 20 April 2010 17:42
>>> To: 'Tapestry users'
>>> Subject: RE: Tapestry using 1.3Gb of heap space after capacity testing
>>>
>>> Thanks for the link. As I said before, I don't think that we've fallen
>>> prey to that exactly. Doing some local testing and looking at one of
>>> our largest pages which displays search results, it has 76 component
>>> objects, each of those will have components nested within (pretty
>>> heavily in some cases) but nothing which has a bunch of components
>>> where only one is used and the others are redundant. (what I think is
>>> this uber-component thing)
>>>
>>> It looks like explicit repeated use of a component, rather than using a
>>> single component in a loop, creates a lot more component objects in the
>>> heap. I've never seen anything warning against this, and we do usually
>>> use loops but it's not always the best solution.
>>>
>>> Again, looking at our results page, a single instance seems to get
>>> 11,283 ComponentPageElementImpl instances created for it. I'm finding
>>> it hard to find a good view of the tree to figure out what component(s)
>>> are the main culprits for this huge number of instances.
>>>
>>> Thanks for any help or guidance you can give me.
>>>
>>> Andy
>>>
>>> > -----Original Message-----
>>> > From: Thiago H. de Paula Figueiredo [mailto:thiagohp@gmail.com]
>>> > Sent: 20 April 2010 15:56
>>> > To: Tapestry users
>>> > Subject: Re: Tapestry using 1.3Gb of heap space after capacity
>>> testing
>>> >
>>> > On Tue, 20 Apr 2010 11:41:25 -0300, Blower, Andy
>>> > <An...@proquest.co.uk> wrote:
>>> >
>>> > > Uber-component anti-pattern, not to my knowledge, but we have a lot
>>> > of
>>> > > pages and components now. And a lot of logic in them. The project's
>>> > > large enough I can't be sure. Do you have a handy link to Howard's
>>> > > description so I can get my team to check?
>>> >
>>> > It's here:
>>> > http://old.nabble.com/-T5.0.18--Out-of-Memory-Error---Potential-Leak-
>>> > (doesn't-reduce-after-forced-GC)-to25403474s302.html#a25497441
>>> > That thread is interesting, by the way. The problem is more about how
>>> > components and pages are written than the number of them.
>>> > What Tapestry version are you using?
>>> >
>>> > > Page pool config is default, I don't know enough to fiddle yet.
>>> Also,
>>> > > the test is on a single stand alone server with no clustering, we
>>> are
>>> > > using Tomcat clustering for production.
>>> >
>>> > You should take a look a them, as they can affect the memory
>>> > consumption
>>> > directly.
>>> >
>>> > > I did fire the GC manually before getting concerned BTW. A lot of
>>> > heap
>>> > > seems to be taken up with 100-300k descriptions on the
>>> > > InternalClassTransformationImpl class. An example of one is
>>> attached
>>> > > below - seems wrong to me and more like the debug that can be
>>> > requested
>>> > > through log4j rather than a description.
>>> >
>>> > They look really bigger than they should. 300k is a lot. I have one
>>> > application with a reasonable number of pages and components and it
>>> > runs
>>> > inside a 512 MB slide at SliceHost without using the whole memory.
>>> >
>>> > --
>>> > Thiago H. de Paula Figueiredo
>>> > Independent Java, Apache Tapestry 5 and Hibernate consultant,
>>> > developer,
>>> > and instructor
>>> > Owner, software architect and developer, Ars Machina Tecnologia da
>>> > Informação Ltda.
>>> > http://www.arsmachina.com.br
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
>>> > For additional commands, e-mail: users-help@tapestry.apache.org
>>> >
>>
>>
>
>
>
> --
> Howard M. Lewis Ship
>
> Creator of Apache Tapestry
>
> The source for Tapestry training, mentoring and support. Contact me to
> learn how I can get you up and productive in Tapestry fast!
>
> (971) 678-5210
> http://howardlewisship.com
>
--
Howard M. Lewis Ship
Creator of Apache Tapestry
The source for Tapestry training, mentoring and support. Contact me to
learn how I can get you up and productive in Tapestry fast!
(971) 678-5210
http://howardlewisship.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Tapestry using 1.3Gb of heap space after capacity testing
Posted by Howard Lewis Ship <hl...@gmail.com>.
Just to check ... which version of Tapestry?
On Tue, Jun 29, 2010 at 8:48 AM, Blower, Andy
<An...@proquest.co.uk> wrote:
> I doubt anyone remembers this thread except me, but we're still having problems that I could do with help on.
>
> So, two months later, we have managed to improve things significantly, but still have very large pages and we're having a lot of trouble with the page pool size. What we see when we push the app too hard is that the heap fills until the gc thread is permanently running, and the app is essentially dead and unresponsive. I've been testing with 2g & 4g heaps up to now, with 100ms soft-wait & 200 hard limit because we need a lot of instances of some pages, and were running out with a 100 limit. I know I need to understand and do more with these values, but it's worrying because our tests are only requesting English pages at present and we'll have 17 languages for launch! So I'm rather concerned to say the least...
>
> I have a theory about what is happening that I'd like to run by the list and see what others think. I'm having some trouble figuring out what's happening at load.
>
> What I think is that when the heap gets full-ish at high load, the gc kicks in more and more often, slowing request processing down a little. (there's a fair amount of soft referenced stuff it can boot) This means that pages are returned to the page pool a bit later causing more page instances to be created which fills the heap and enters an vicious cycle with the gc trying to free memory and the T5 page pool trying to create more page instances. The reason I suspect this is because heap dumps show about 40 PageImpl's that have not completed loading, and many more that have but are much smaller in size than others for the same page. (e.g. a Results PageImpl retaining 28k when a normal one retains 7.5Mb - very big I know, but I've cut it down from 14Mb)
>
> Our site has 223 pages which consume 213Mb if 1 instance of each is instantiated. Multiply this by 200 (hard limit) and then by 17 languages and worst case we'll need a heap size of 725Gb which is a little ridiculous! I will do more work on reducing the size of the pages but I've already done the easy stuff.
>
> Here's a list of the heap dominators using Eclipse MAT, http://lh6.ggpht.com/_YwJn8TJTqJU/TCoNT11ouZI/AAAAAAAAACI/G8eTGsth4zM/HeapDominators.png
>
> Anyone think I'm on the right track, or barking up the wrong tree completely?
>
> Thanks,
>
> Andy
>
>> -----Original Message-----
>> From: Blower, Andy [mailto:Andy.Blower@proquest.co.uk]
>> Sent: 20 April 2010 17:42
>> To: 'Tapestry users'
>> Subject: RE: Tapestry using 1.3Gb of heap space after capacity testing
>>
>> Thanks for the link. As I said before, I don't think that we've fallen
>> prey to that exactly. Doing some local testing and looking at one of
>> our largest pages which displays search results, it has 76 component
>> objects, each of those will have components nested within (pretty
>> heavily in some cases) but nothing which has a bunch of components
>> where only one is used and the others are redundant. (what I think is
>> this uber-component thing)
>>
>> It looks like explicit repeated use of a component, rather than using a
>> single component in a loop, creates a lot more component objects in the
>> heap. I've never seen anything warning against this, and we do usually
>> use loops but it's not always the best solution.
>>
>> Again, looking at our results page, a single instance seems to get
>> 11,283 ComponentPageElementImpl instances created for it. I'm finding
>> it hard to find a good view of the tree to figure out what component(s)
>> are the main culprits for this huge number of instances.
>>
>> Thanks for any help or guidance you can give me.
>>
>> Andy
>>
>> > -----Original Message-----
>> > From: Thiago H. de Paula Figueiredo [mailto:thiagohp@gmail.com]
>> > Sent: 20 April 2010 15:56
>> > To: Tapestry users
>> > Subject: Re: Tapestry using 1.3Gb of heap space after capacity
>> testing
>> >
>> > On Tue, 20 Apr 2010 11:41:25 -0300, Blower, Andy
>> > <An...@proquest.co.uk> wrote:
>> >
>> > > Uber-component anti-pattern, not to my knowledge, but we have a lot
>> > of
>> > > pages and components now. And a lot of logic in them. The project's
>> > > large enough I can't be sure. Do you have a handy link to Howard's
>> > > description so I can get my team to check?
>> >
>> > It's here:
>> > http://old.nabble.com/-T5.0.18--Out-of-Memory-Error---Potential-Leak-
>> > (doesn't-reduce-after-forced-GC)-to25403474s302.html#a25497441
>> > That thread is interesting, by the way. The problem is more about how
>> > components and pages are written than the number of them.
>> > What Tapestry version are you using?
>> >
>> > > Page pool config is default, I don't know enough to fiddle yet.
>> Also,
>> > > the test is on a single stand alone server with no clustering, we
>> are
>> > > using Tomcat clustering for production.
>> >
>> > You should take a look a them, as they can affect the memory
>> > consumption
>> > directly.
>> >
>> > > I did fire the GC manually before getting concerned BTW. A lot of
>> > heap
>> > > seems to be taken up with 100-300k descriptions on the
>> > > InternalClassTransformationImpl class. An example of one is
>> attached
>> > > below - seems wrong to me and more like the debug that can be
>> > requested
>> > > through log4j rather than a description.
>> >
>> > They look really bigger than they should. 300k is a lot. I have one
>> > application with a reasonable number of pages and components and it
>> > runs
>> > inside a 512 MB slide at SliceHost without using the whole memory.
>> >
>> > --
>> > Thiago H. de Paula Figueiredo
>> > Independent Java, Apache Tapestry 5 and Hibernate consultant,
>> > developer,
>> > and instructor
>> > Owner, software architect and developer, Ars Machina Tecnologia da
>> > Informação Ltda.
>> > http://www.arsmachina.com.br
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
>> > For additional commands, e-mail: users-help@tapestry.apache.org
>> >
>
>
--
Howard M. Lewis Ship
Creator of Apache Tapestry
The source for Tapestry training, mentoring and support. Contact me to
learn how I can get you up and productive in Tapestry fast!
(971) 678-5210
http://howardlewisship.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Tapestry using 1.3Gb of heap space after capacity testing
Posted by Christophe Cordenier <ch...@gmail.com>.
Hi
This make me think about something i have found when i started to go deep in
the Tapestry page pool code. It's about the PagePoolImpl service. If we look
at the checkout method, we can see that the test made to check the pool's
limits is made on a inUse list. I have made the experience that in heavy
load when the pool is empty this test becomes useless because the inUse list
is updated after the page loading (that is obviously not synchronized).
Theory is that if you have huge page, you may come into this case.
I don't know if it's related to your problem, but I guess, this is why
"tapestry.page-pool.hard-limit" will be deprecated, as this is to the
container to handle the limits of request handling threads.
Anyway, in the meantime shouldn't we handle the limits check using a
'inConstruction' list ? But it sounds to simple to be true :)
2010/6/29 Blower, Andy <An...@proquest.co.uk>
> I doubt anyone remembers this thread except me, but we're still having
> problems that I could do with help on.
>
> So, two months later, we have managed to improve things significantly, but
> still have very large pages and we're having a lot of trouble with the page
> pool size. What we see when we push the app too hard is that the heap fills
> until the gc thread is permanently running, and the app is essentially dead
> and unresponsive. I've been testing with 2g & 4g heaps up to now, with 100ms
> soft-wait & 200 hard limit because we need a lot of instances of some pages,
> and were running out with a 100 limit. I know I need to understand and do
> more with these values, but it's worrying because our tests are only
> requesting English pages at present and we'll have 17 languages for launch!
> So I'm rather concerned to say the least...
>
> I have a theory about what is happening that I'd like to run by the list
> and see what others think. I'm having some trouble figuring out what's
> happening at load.
>
> What I think is that when the heap gets full-ish at high load, the gc kicks
> in more and more often, slowing request processing down a little. (there's a
> fair amount of soft referenced stuff it can boot) This means that pages are
> returned to the page pool a bit later causing more page instances to be
> created which fills the heap and enters an vicious cycle with the gc trying
> to free memory and the T5 page pool trying to create more page instances.
> The reason I suspect this is because heap dumps show about 40 PageImpl's
> that have not completed loading, and many more that have but are much
> smaller in size than others for the same page. (e.g. a Results PageImpl
> retaining 28k when a normal one retains 7.5Mb - very big I know, but I've
> cut it down from 14Mb)
>
> Our site has 223 pages which consume 213Mb if 1 instance of each is
> instantiated. Multiply this by 200 (hard limit) and then by 17 languages and
> worst case we'll need a heap size of 725Gb which is a little ridiculous! I
> will do more work on reducing the size of the pages but I've already done
> the easy stuff.
>
> Here's a list of the heap dominators using Eclipse MAT,
> http://lh6.ggpht.com/_YwJn8TJTqJU/TCoNT11ouZI/AAAAAAAAACI/G8eTGsth4zM/HeapDominators.png
>
> Anyone think I'm on the right track, or barking up the wrong tree
> completely?
>
> Thanks,
>
> Andy
>
> > -----Original Message-----
> > From: Blower, Andy [mailto:Andy.Blower@proquest.co.uk]
> > Sent: 20 April 2010 17:42
> > To: 'Tapestry users'
> > Subject: RE: Tapestry using 1.3Gb of heap space after capacity testing
> >
> > Thanks for the link. As I said before, I don't think that we've fallen
> > prey to that exactly. Doing some local testing and looking at one of
> > our largest pages which displays search results, it has 76 component
> > objects, each of those will have components nested within (pretty
> > heavily in some cases) but nothing which has a bunch of components
> > where only one is used and the others are redundant. (what I think is
> > this uber-component thing)
> >
> > It looks like explicit repeated use of a component, rather than using a
> > single component in a loop, creates a lot more component objects in the
> > heap. I've never seen anything warning against this, and we do usually
> > use loops but it's not always the best solution.
> >
> > Again, looking at our results page, a single instance seems to get
> > 11,283 ComponentPageElementImpl instances created for it. I'm finding
> > it hard to find a good view of the tree to figure out what component(s)
> > are the main culprits for this huge number of instances.
> >
> > Thanks for any help or guidance you can give me.
> >
> > Andy
> >
> > > -----Original Message-----
> > > From: Thiago H. de Paula Figueiredo [mailto:thiagohp@gmail.com]
> > > Sent: 20 April 2010 15:56
> > > To: Tapestry users
> > > Subject: Re: Tapestry using 1.3Gb of heap space after capacity
> > testing
> > >
> > > On Tue, 20 Apr 2010 11:41:25 -0300, Blower, Andy
> > > <An...@proquest.co.uk> wrote:
> > >
> > > > Uber-component anti-pattern, not to my knowledge, but we have a lot
> > > of
> > > > pages and components now. And a lot of logic in them. The project's
> > > > large enough I can't be sure. Do you have a handy link to Howard's
> > > > description so I can get my team to check?
> > >
> > > It's here:
> > > http://old.nabble.com/-T5.0.18--Out-of-Memory-Error---Potential-Leak-
> > > (doesn't-reduce-after-forced-GC)-to25403474s302.html#a25497441
> > > That thread is interesting, by the way. The problem is more about how
> > > components and pages are written than the number of them.
> > > What Tapestry version are you using?
> > >
> > > > Page pool config is default, I don't know enough to fiddle yet.
> > Also,
> > > > the test is on a single stand alone server with no clustering, we
> > are
> > > > using Tomcat clustering for production.
> > >
> > > You should take a look a them, as they can affect the memory
> > > consumption
> > > directly.
> > >
> > > > I did fire the GC manually before getting concerned BTW. A lot of
> > > heap
> > > > seems to be taken up with 100-300k descriptions on the
> > > > InternalClassTransformationImpl class. An example of one is
> > attached
> > > > below - seems wrong to me and more like the debug that can be
> > > requested
> > > > through log4j rather than a description.
> > >
> > > They look really bigger than they should. 300k is a lot. I have one
> > > application with a reasonable number of pages and components and it
> > > runs
> > > inside a 512 MB slide at SliceHost without using the whole memory.
> > >
> > > --
> > > Thiago H. de Paula Figueiredo
> > > Independent Java, Apache Tapestry 5 and Hibernate consultant,
> > > developer,
> > > and instructor
> > > Owner, software architect and developer, Ars Machina Tecnologia da
> > > Informação Ltda.
> > > http://www.arsmachina.com.br
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> > > For additional commands, e-mail: users-help@tapestry.apache.org
> > >
>
>
--
Regards,
Christophe Cordenier.
Committer on Apache Tapestry 5
Co-creator of wooki @wookicentral.com