You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Stefano Mazzocchi <st...@apache.org> on 2001/11/30 18:38:03 UTC
Cocoon 2.0 Scalability Disappointment
Michael told me privately the numbers that he has been experiencing
along with a description of his environment and I now agree with him, we
have a serious scalability problem.
He refers to 2.0RC2 but since nothing that could improve performance
changed between 2.0 and 2.0RC2, I think it's still up there.
For "serious", I mean something important that should be fixed ASAP,
otherwise, Cocoon 2.0 will rarely have any use in high-load
environments.
The numbers say, more or less, than the same functionality ported from
Cocoon 1.8.2 to Cocoon 2.0RC2 resulted in 20 times slower performance!
Yes, guys, we're not talking about 20% slower but 2000% slower.
WAIT, STOP, SIT DOWN!
Before you rush to your boss to tell that Cocoon 2.0 sucks and you
shouldn't place your million-dollars worth project on it, let me explain
what I believe the problem is.
1) their Cocoon 1.8.2 application has been up for a long time and they
carefully tuned the system for it. Their Cocoon 2.0RC2 version is an
early implementation.
2) they admit the Cocoon 2.0 cache system is likely to provide huge
benefits for them, but they are not sure they are using it properly.
3) they don't use a transparent proxy on top so the Cocoon cache is
continously stressed.
4) they report much better "perceived" performance on a single browsing
experience: this seems to show that Cocoon 2.0 isn't slow by itself, but
there is an hidden bottleneck someplace.
5) they use XSP. I'm aware of performance issues with XSP load and
execution under heavy load (some forgotten locking or synchronized
method?). This is the place where I'd concentrate profiling effort.
Net Result: I think Cocoon 2.0, as it stands, doesn't scale, but the
data provided it's not sufficient to understand *what* slows Cocoon
down.
Michael, I'd love if you guys could perform some more load tests for us:
1) disable logging. If log is DEBUG, it could generate Gigabytes of
information and disks could become the bottleneck.
[I think log might be the bottleneck]
2) disable resource reloading. Same as above, the disk I/O system could
become the bottleneck.
3) try to come up with some logarithmic stress tests: 1 req/sec, 2
req/sec, 4 req/sec, 8, 16, 32, 64, until your machines saturate.
4) turn the cache off/on. [this, compared to the logaritmic approach
should give us information on the caching system]
5) try to get the generated source code of one of your XSP, compile it
as a generator and use it as a normal generator instead. [this should
remove the XSP loading/handling phase]
With this data we will know where the problem is and if it really
resides in Cocoon.
--
Stefano Mazzocchi One must still have chaos in oneself to be
able to give birth to a dancing star.
<st...@apache.org> Friedrich Nietzsche
--------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Stefano Mazzocchi <st...@apache.org>.
Berin Loritsch wrote:
>
> Michael Hartle wrote:
>
> > Sylvain Wallez wrote:
> >
> >>> Net Result: I think Cocoon 2.0, as it stands, doesn't scale, but the
> >>> data provided it's not sufficient to understand *what* slows Cocoon
> >>> down.
> >>>
> >>> Michael, I'd love if you guys could perform some more load tests for us:
> >>>
> >> 0) before disabling logging, search for messages such as
> >> "decommissioning instance of...". This reveals some undersized pools
> >> which are corrected by tuning cocoon.xconf and sitemap.xmap. Undersized
> >> pools act like an object factory, plus the ComponentManager overhead.
> >>
> > Just a thought, as not tuning the pools is probably going to be a common
> > and easy mistake for people who are starting to evaluate Cocoon and get
> > bad performance results, what about an adaptable pool which starts
> > tuning its configuration after a while depending on usage ? Is this
> > achievable at all ?
>
> It is achievable. I have not gotten around to doing it yet.
Look into JServ, there's one already written.
By me :)
Peter didn't consider this important enough to enter Avalon, but I still
think my pool code is the most flexible one. We could move it overhere
and use it.
--
Stefano Mazzocchi One must still have chaos in oneself to be
able to give birth to a dancing star.
<st...@apache.org> Friedrich Nietzsche
--------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Berin Loritsch <bl...@apache.org>.
Michael Hartle wrote:
> Sylvain Wallez wrote:
>
>>> Net Result: I think Cocoon 2.0, as it stands, doesn't scale, but the
>>> data provided it's not sufficient to understand *what* slows Cocoon
>>> down.
>>>
>>> Michael, I'd love if you guys could perform some more load tests for us:
>>>
>> 0) before disabling logging, search for messages such as
>> "decommissioning instance of...". This reveals some undersized pools
>> which are corrected by tuning cocoon.xconf and sitemap.xmap. Undersized
>> pools act like an object factory, plus the ComponentManager overhead.
>>
> Just a thought, as not tuning the pools is probably going to be a common
> and easy mistake for people who are starting to evaluate Cocoon and get
> bad performance results, what about an adaptable pool which starts
> tuning its configuration after a while depending on usage ? Is this
> achievable at all ?
It is achievable. I have not gotten around to doing it yet. I created the
current pooling implementation--and the reason I chose the ResourceLimiting
approach is so that the JVM doesn't hold an uncontrolled number of references
which can be damaging to GC collection cycles later. Clearly, the resources
have to be controlled, but it can be done in a smarter manner.
--
"They that give up essential liberty to obtain a little temporary safety
deserve neither liberty nor safety."
- Benjamin Franklin
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Michael Hartle <mh...@hartle-klug.com>.
Sylvain Wallez wrote:
>>Net Result: I think Cocoon 2.0, as it stands, doesn't scale, but the
>>data provided it's not sufficient to understand *what* slows Cocoon
>>down.
>>
>>Michael, I'd love if you guys could perform some more load tests for us:
>>
>0) before disabling logging, search for messages such as
>"decommissioning instance of...". This reveals some undersized pools
>which are corrected by tuning cocoon.xconf and sitemap.xmap. Undersized
>pools act like an object factory, plus the ComponentManager overhead.
>
Just a thought, as not tuning the pools is probably going to be a common
and easy mistake for people who are starting to evaluate Cocoon and get
bad performance results, what about an adaptable pool which starts
tuning its configuration after a while depending on usage ? Is this
achievable at all ?
Best regards,
Michael Hartle
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
XSP speedup [was Re: Cocoon 2.0 Scalability Disappointment]
Posted by Sylvain Wallez <sy...@anyware-tech.com>.
Sylvain Wallez a écrit :
>
> Stefano Mazzocchi a écrit :
> >
> > Michael told me privately the numbers that he has been experiencing
> > along with a description of his environment and I now agree with him, we
> > have a serious scalability problem.
<snip/>
> There's a CPU/memory eater in ServerPagesGenerator : each startElement
> is stored in a stack in order to properly close all opened elements in
> case of premature end of the XSP (return statement or exception). Cocoon
> 1 didn't have this because of the use of DOM : even if partially
> generated, the DOM is always a valid XML document.
>
> This stack is a LinkedList to which EventData objects are appended. This
> means for each element, 2 objects are allocated : the EventData object,
> and a LinkedList node. We can change this stuff to use a single
> ArrayStack (from excalibur.collections) and no EventData object, which
> should significantly reduce CPU consumption.
>
> But the real question is : can an XSP be considered valid if it contains
> a "return" statement ? I personnaly never saw such XSPs. Also, this
> stuff is only a half protection against bogus code in the XSP : it will
> take care of having as much closed elements as opened ones, but doesn't
> check proper balancing nor do anything for constructs like a "break" in
> a loop inside an XML element.
>
> So my opinion is remove this costly stuff and forbid return statements
> in xsp:logic. This will make a speedy XSP engine.
After more thinking, I came to the conclusion that using a "return" to
break prematurely the XSP generation is **really bad** : some
logicsheets (e.g. ESQL) generate additional "segmentation methods" to
circumvent the 64 kb limit imposed by the JVM for the bytecode size of a
single method. If someone uses return statements to prematurely end an
XSP that uses such a logicsheet, he may either end the full XSP or end a
part of it if the "return" gets generated in a segmentation method, in
an uncontrolled way (decided by the logicsheet).
Since we cannot really know what the return statement will actually do,
this should be considered a bad practice which should be avoided (once
again, I never saw such a return statement in an XSP). And IMO we
shouldn't harm performances of the XSP engine to support bad practices.
So I refactored ServerPagesGenerator in both 2.0 and 2.1 to make
document completion optional (it is disabled by default), and faster if
enabled through the use of an ArrayStack instead of a LinkedList.
Michael, could you please update your performance measurements and tell
us if this leads to a significant change ?
Sylvain.
--
Sylvain Wallez
Anyware Technologies - http://www.anyware-tech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Sylvain Wallez <sy...@anyware-tech.com>.
Berin Loritsch a écrit :
>
> Sylvain Wallez wrote:
>
> >
> > There's a CPU/memory eater in ServerPagesGenerator : each startElement
> > is stored in a stack in order to properly close all opened elements in
> > case of premature end of the XSP (return statement or exception). Cocoon
> > 1 didn't have this because of the use of DOM : even if partially
> > generated, the DOM is always a valid XML document.
> >
> > This stack is a LinkedList to which EventData objects are appended. This
> > means for each element, 2 objects are allocated : the EventData object,
> > and a LinkedList node. We can change this stuff to use a single
> > ArrayStack (from excalibur.collections) and no EventData object, which
> > should significantly reduce CPU consumption.
> >
> > But the real question is : can an XSP be considered valid if it contains
> > a "return" statement ? I personnaly never saw such XSPs. Also, this
> > stuff is only a half protection against bogus code in the XSP : it will
> > take care of having as much closed elements as opened ones, but doesn't
> > check proper balancing nor do anything for constructs like a "break" in
> > a loop inside an XML element.
> >
> > So my opinion is remove this costly stuff and forbid return statements
> > in xsp:logic. This will make a speedy XSP engine.
> >
> > Your thoughts ?
>
> Before you throw out the baby with the bath water, we have to consider the
> creation of methods in XSP. If a method is created by the <xsp:logic>
> section outside of the root element, then you must allow a return statement.
Poor baby ;)
> for example:
>
> <xsp:page>
> <xsp:logic>
> /** should be legal */
> private int getUserId() {
> return ((Integer) request.getSession().getAttribute("userid")).intValue();
> }
> </xsp:logic>
>
> <mypage>
> <xsp:logic>
> int userid = getUserId();
> return; // should be Illegal
> </xsp:logic>
> </mypage>
> </xsl:page>
>
> Do you notice the key distinction?
>
Sure, and I was aware of that : by "forbid return statements", I was
meaning an XSP writing guideline (coding standard) saying not to use a
return statement to break the normal control flow, and not a rule
enforced by the engine : this would be difficult to implement, since it
would require a Java parser. The return statement at the end of a method
doesn't break the flow and is thus perfectly legal.
Sylvain
--
Sylvain Wallez
Anyware Technologies - http://www.anyware-tech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Berin Loritsch <bl...@apache.org>.
Sylvain Wallez wrote:
>
> There's a CPU/memory eater in ServerPagesGenerator : each startElement
> is stored in a stack in order to properly close all opened elements in
> case of premature end of the XSP (return statement or exception). Cocoon
> 1 didn't have this because of the use of DOM : even if partially
> generated, the DOM is always a valid XML document.
>
> This stack is a LinkedList to which EventData objects are appended. This
> means for each element, 2 objects are allocated : the EventData object,
> and a LinkedList node. We can change this stuff to use a single
> ArrayStack (from excalibur.collections) and no EventData object, which
> should significantly reduce CPU consumption.
>
> But the real question is : can an XSP be considered valid if it contains
> a "return" statement ? I personnaly never saw such XSPs. Also, this
> stuff is only a half protection against bogus code in the XSP : it will
> take care of having as much closed elements as opened ones, but doesn't
> check proper balancing nor do anything for constructs like a "break" in
> a loop inside an XML element.
>
> So my opinion is remove this costly stuff and forbid return statements
> in xsp:logic. This will make a speedy XSP engine.
>
> Your thoughts ?
Before you throw out the baby with the bath water, we have to consider the
creation of methods in XSP. If a method is created by the <xsp:logic>
section outside of the root element, then you must allow a return statement.
for example:
<xsp:page>
<xsp:logic>
/** should be legal */
private int getUserId() {
return ((Integer) request.getSession().getAttribute("userid")).intValue();
}
</xsp:logic>
<mypage>
<xsp:logic>
int userid = getUserId();
return; // should be Illegal
</xsp:logic>
</mypage>
</xsl:page>
Do you notice the key distinction?
--
"They that give up essential liberty to obtain a little temporary safety
deserve neither liberty nor safety."
- Benjamin Franklin
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Michael Hartle <mh...@hartle-klug.com>.
Peter Royal wrote:
>>>From a caching standpoint, I've removed everything dynamic out of my XSP
>documents so the results are cacheable. (I still like XSP because I can
>precompute static final java objects to stick in as Request attributes).
>
You can keep your XSPs dynamic and yet cachable when you are able to
classify the parameters your XSP is based on and use them appropriately
in the cache-relevant functions; some time ago, I mapped some URI space
to a XSP page which in turn generated a JPEG navigation button via SVG;
depending on what was entered in the path, the pictures and labels on
the button changed. By using these parameters in cache key
generation/validation functions in the XSP (not sure how exactly, but
the code is around somewhere), I got the individual results
appropriately cached.
Best regards,
Michael Hartle
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Peter Royal <pr...@managingpartners.com>.
On Friday 30 November 2001 01:22 pm, you wrote:
> This stack is a LinkedList to which EventData objects are appended. This
> means for each element, 2 objects are allocated : the EventData object,
> and a LinkedList node. We can change this stuff to use a single
> ArrayStack (from excalibur.collections) and no EventData object, which
> should significantly reduce CPU consumption.
<snip>
> So my opinion is remove this costly stuff and forbid return statements
> in xsp:logic. This will make a speedy XSP engine.
Yes! How about making that an option in the xsp:page element?
enable-error-handling or something as such.
>From a caching standpoint, I've removed everything dynamic out of my XSP
documents so the results are cacheable. (I still like XSP because I can
precompute static final java objects to stick in as Request attributes).
As such, I know for certain that I don't need the event cache that XSP keeps,
the only code in the generate() function is calls to the SAX functions and
some if's on variables that have already been defined.
> 0) before disabling logging, search for messages such as
> "decommissioning instance of...". This reveals some undersized pools
> which are corrected by tuning cocoon.xconf and sitemap.xmap. Undersized
> pools act like an object factory, plus the ComponentManager overhead.
Definitly do the above. That can literal kill performance of the server,
especially if it turns into a factory for XSLT transformers *shudder*.
Also cache as much as you can. If you have a pipeline that is different each
request, reorganize it so that any static pieces are done first so that bit
can be cached. Pipeline organization is huge.
For some numbers (the initial ones are sad, I'm almost embarassed to admit
them) from our tests:
Baseline:
1 thread * 10 iter: 10s
5 thread * 10 iter: 30s + --> i eventually gave up. pools turned into
factories and it was ugly
Reorganized pipline (static pieces up front to utilize caching):
1 thread * 10iter: 2s
5 thread * 10iter: 4s
This is a webapp on an intranet so concurrency isn't as high as a public
website might be. Each request also generates at least a single RMI call,
which is a few hundred msec minimum. I'm not done tuning yet, I just got it
to the point of "good enough" so that other things to do have higher
priority. My next point of attack will be to stick a profiler on the system
to see where to focus on next. I know the RMI is an issue, and I plan to
attempt to get Catalina running under Phoenix to fix that. The RMI calls are
a few 100 msec's each so I'm sure that will help a bit.
-pete
--
peter royal -> proyal@managingpartners.com
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Sylvain Wallez <sy...@anyware-tech.com>.
Stefano Mazzocchi a écrit :
>
> Michael told me privately the numbers that he has been experiencing
> along with a description of his environment and I now agree with him, we
> have a serious scalability problem.
>
> He refers to 2.0RC2 but since nothing that could improve performance
> changed between 2.0 and 2.0RC2, I think it's still up there.
>
> For "serious", I mean something important that should be fixed ASAP,
> otherwise, Cocoon 2.0 will rarely have any use in high-load
> environments.
>
> The numbers say, more or less, than the same functionality ported from
> Cocoon 1.8.2 to Cocoon 2.0RC2 resulted in 20 times slower performance!
>
> Yes, guys, we're not talking about 20% slower but 2000% slower.
>
> WAIT, STOP, SIT DOWN!
>
> Before you rush to your boss to tell that Cocoon 2.0 sucks and you
> shouldn't place your million-dollars worth project on it, let me explain
> what I believe the problem is.
>
> 1) their Cocoon 1.8.2 application has been up for a long time and they
> carefully tuned the system for it. Their Cocoon 2.0RC2 version is an
> early implementation.
>
> 2) they admit the Cocoon 2.0 cache system is likely to provide huge
> benefits for them, but they are not sure they are using it properly.
>
> 3) they don't use a transparent proxy on top so the Cocoon cache is
> continously stressed.
>
> 4) they report much better "perceived" performance on a single browsing
> experience: this seems to show that Cocoon 2.0 isn't slow by itself, but
> there is an hidden bottleneck someplace.
>
> 5) they use XSP. I'm aware of performance issues with XSP load and
> execution under heavy load (some forgotten locking or synchronized
> method?). This is the place where I'd concentrate profiling effort.
There's a CPU/memory eater in ServerPagesGenerator : each startElement
is stored in a stack in order to properly close all opened elements in
case of premature end of the XSP (return statement or exception). Cocoon
1 didn't have this because of the use of DOM : even if partially
generated, the DOM is always a valid XML document.
This stack is a LinkedList to which EventData objects are appended. This
means for each element, 2 objects are allocated : the EventData object,
and a LinkedList node. We can change this stuff to use a single
ArrayStack (from excalibur.collections) and no EventData object, which
should significantly reduce CPU consumption.
But the real question is : can an XSP be considered valid if it contains
a "return" statement ? I personnaly never saw such XSPs. Also, this
stuff is only a half protection against bogus code in the XSP : it will
take care of having as much closed elements as opened ones, but doesn't
check proper balancing nor do anything for constructs like a "break" in
a loop inside an XML element.
So my opinion is remove this costly stuff and forbid return statements
in xsp:logic. This will make a speedy XSP engine.
Your thoughts ?
> Net Result: I think Cocoon 2.0, as it stands, doesn't scale, but the
> data provided it's not sufficient to understand *what* slows Cocoon
> down.
>
> Michael, I'd love if you guys could perform some more load tests for us:
0) before disabling logging, search for messages such as
"decommissioning instance of...". This reveals some undersized pools
which are corrected by tuning cocoon.xconf and sitemap.xmap. Undersized
pools act like an object factory, plus the ComponentManager overhead.
> 1) disable logging. If log is DEBUG, it could generate Gigabytes of
> information and disks could become the bottleneck.
>
> [I think log might be the bottleneck]
>
> 2) disable resource reloading. Same as above, the disk I/O system could
> become the bottleneck.
>
> 3) try to come up with some logarithmic stress tests: 1 req/sec, 2
> req/sec, 4 req/sec, 8, 16, 32, 64, until your machines saturate.
>
> 4) turn the cache off/on. [this, compared to the logaritmic approach
> should give us information on the caching system]
>
> 5) try to get the generated source code of one of your XSP, compile it
> as a generator and use it as a normal generator instead. [this should
> remove the XSP loading/handling phase]
And will also avoid the stack management mentioned above.
> With this data we will know where the problem is and if it really
> resides in Cocoon.
>
> --
> Stefano Mazzocchi One must still have chaos in oneself to be
> able to give birth to a dancing star.
> <st...@apache.org> Friedrich Nietzsche
--
Sylvain Wallez
Anyware Technologies - http://www.anyware-tech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Sebastien Koechlin <sk...@ivision.fr>.
Sylvain Wallez wrote:
> Maybe it should be difficult to access this information from the status
> page, but this is certainly something we can output in the logs.
>
> Do you have an implementation idea for this, or better, a patch ?
I can try if someone can tell me where to look at (files or classes).
I did not spend as much time in Cocoon2 code as I did in Cocoon1,
and I'm often lost.
- What isa StoreJanitor?
- How does org.apache.cocoon.caching.CachedEventObject,
org.apache.cocoon.components.pipeline.CachingEventPipeline
and org.apache.cocoon.components.store.MRUMemoryStore
interact?
- How can I have access to the store?
- Where are object added into the store?
--
Sébastien Koechlin - IVision - skoechlin@ivision.fr
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Sylvain Wallez <sy...@anyware-tech.com>.
Sebastien Koechlin a écrit :
>
> Stefano Mazzocchi wrote:
>
> > 2) they admit the Cocoon 2.0 cache system is likely to provide huge
> > benefits for them, but they are not sure they are using it properly.
>
> It was to monitor who does cache is working that I wrote something
> about "Improving StatusGenerator".
>
> I think that for each cached stream/event, we should record a creation
> timestamp and increment a counter each time the cached object is served.
>
> On the status page, displaying thoses values allow us to see if cache
> is used where it is expected to.
>
> - If nothing is listed : cache is not active.
> - If counter is 0 or 1 : cache is re-generated each time.
>
> --
> Sébastien Koechlin - IVision - skoechlin@ivision.fr
>
Hi, Sébastien
Maybe it should be difficult to access this information from the status
page, but this is certainly something we can output in the logs.
Do you have an implementation idea for this, or better, a patch ?
Sylvain
--
Sylvain Wallez
Anyware Technologies - http://www.anyware-tech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Sebastien Koechlin <sk...@ivision.fr>.
Stefano Mazzocchi wrote:
> 2) they admit the Cocoon 2.0 cache system is likely to provide huge
> benefits for them, but they are not sure they are using it properly.
It was to monitor who does cache is working that I wrote something
about "Improving StatusGenerator".
I think that for each cached stream/event, we should record a creation
timestamp and increment a counter each time the cached object is served.
On the status page, displaying thoses values allow us to see if cache
is used where it is expected to.
- If nothing is listed : cache is not active.
- If counter is 0 or 1 : cache is re-generated each time.
--
Sébastien Koechlin - IVision - skoechlin@ivision.fr
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by giacomo <gi...@apache.org>.
On Fri, 30 Nov 2001, Gianugo Rabellino wrote:
> Stefano Mazzocchi wrote:
>
> > 3) they don't use a transparent proxy on top so the Cocoon cache is
> > continously stressed.
>
>
> If they are using XSP (or anyway are dynamically generating data) a
> transparent proxy won't help a lot: it will help the data flow (since
> Apache will be run on a fast network or even on a loopback) but not much
> more.
>
> > 5) they use XSP. I'm aware of performance issues with XSP load and
> > execution under heavy load (some forgotten locking or synchronized
> > method?). This is the place where I'd concentrate profiling effort.
>
>
> So do I.
>
> > 1) disable logging. If log is DEBUG, it could generate Gigabytes of
> > information and disks could become the bottleneck.
> >
> > [I think log might be the bottleneck]
>
>
> I think this can be an appropriate timing to start cleaning up the
> logging code. We have to find a policy and stick to it: I tend to say
> that we should *always* do a isDebugEnabled() or a isInfoEnabled()
> before spitting out Strings. I'm not that sure that such a policy should
> be enforced for levels of warn and above.
>
> We should also come up with a sort of best practice/code convention
> about what should be logged at each level: from a quick look at the code
> it seems to me that many warnings and even some errors are logged at the
> debug level: this is to be avoided in all cases.
>
> The next question is about Avalon: once we come up with a great logging
> policy we will we have control over Cocoon logging, but what about
> Avalon generated messagess ? Is the Avalon code itself "log wise" or
> will we end up with tons of debug strings being created anyway?
I had a talk there about the pooling messages as they are important for
pool tuning. The consensus so far will be to use a different category to
log onto concerning this issue.
Anyway, with the logkit management system you are able to configure each
component to the logger it should get and thus you can easily
concentrate on a component base. But yes, Avalon itself (ie. Component
Managent System) doesn't behave that way. Today it is using the logger
passed into the ExcaliburComponentManager for all CM operations.
Maybe we should move this logging aspect to the Avvalon list.
Giacomo
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Gianugo Rabellino <gi...@rabellino.it>.
Stefano Mazzocchi wrote:
> 3) they don't use a transparent proxy on top so the Cocoon cache is
> continously stressed.
If they are using XSP (or anyway are dynamically generating data) a
transparent proxy won't help a lot: it will help the data flow (since
Apache will be run on a fast network or even on a loopback) but not much
more.
> 5) they use XSP. I'm aware of performance issues with XSP load and
> execution under heavy load (some forgotten locking or synchronized
> method?). This is the place where I'd concentrate profiling effort.
So do I.
> 1) disable logging. If log is DEBUG, it could generate Gigabytes of
> information and disks could become the bottleneck.
>
> [I think log might be the bottleneck]
I think this can be an appropriate timing to start cleaning up the
logging code. We have to find a policy and stick to it: I tend to say
that we should *always* do a isDebugEnabled() or a isInfoEnabled()
before spitting out Strings. I'm not that sure that such a policy should
be enforced for levels of warn and above.
We should also come up with a sort of best practice/code convention
about what should be logged at each level: from a quick look at the code
it seems to me that many warnings and even some errors are logged at the
debug level: this is to be avoided in all cases.
The next question is about Avalon: once we come up with a great logging
policy we will we have control over Cocoon logging, but what about
Avalon generated messagess ? Is the Avalon code itself "log wise" or
will we end up with tons of debug strings being created anyway?
Ciao,
--
Gianugo
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Berin Loritsch <bl...@apache.org>.
Gerhard Froehlich wrote:
>>-----Original Message-----
>>From: Berin Loritsch [mailto:bloritsch@apache.org]
>>Sent: Saturday, December 01, 2001 2:50 PM
>>To: cocoon-dev@xml.apache.org
>>Subject: Re: Cocoon 2.0 Scalability Disappointment
>>
>>
>>Gerhard Froehlich wrote:
>>
>>
>>>>-----Original Message-----
>>>>From: Chris Finn [mailto:yarnfish2001@yahoo.com]
>>>>
>>>>
>>>As I did load tests with the RC1 I made the same experience.
>>>With the help of OptimizeIt (http://www.vmgear.com/) I discovered
>>>that most resource is burning the ClassLoader and stuff.
>>>See thread http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=99857216131771&w=2
>>>
>>>Berin was talking about a Active Monitor. But I don't know if
>>>he implemented this.
>>>
>>
>>It's in Avalon Excalibur 4.0 release. Yes, I did implement it.
>>
>
> I should know this, shouldn't I ;-).
> I crawled the Mail Archive and it seems that the ActiveMonitor is
> not worked in, yet. Are there any issues open or difficulties?
Not that I know of. I just don't have time, so I need other people
to do it... Hint, hint.
----------------------------------------------------
Sign Up for NetZero Platinum Today
Only $9.95 per month!
http://my.netzero.net/s/signup?r=platinum&refcd=PT97
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: Cocoon 2.0 Scalability Disappointment
Posted by Gerhard Froehlich <g-...@gmx.de>.
>-----Original Message-----
>From: Berin Loritsch [mailto:bloritsch@apache.org]
>Sent: Saturday, December 01, 2001 2:50 PM
>To: cocoon-dev@xml.apache.org
>Subject: Re: Cocoon 2.0 Scalability Disappointment
>
>
>Gerhard Froehlich wrote:
>
>>>-----Original Message-----
>>>From: Chris Finn [mailto:yarnfish2001@yahoo.com]
>>>
>> As I did load tests with the RC1 I made the same experience.
>> With the help of OptimizeIt (http://www.vmgear.com/) I discovered
>> that most resource is burning the ClassLoader and stuff.
>> See thread http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=99857216131771&w=2
>>
>> Berin was talking about a Active Monitor. But I don't know if
>> he implemented this.
>
>
>It's in Avalon Excalibur 4.0 release. Yes, I did implement it.
I should know this, shouldn't I ;-).
I crawled the Mail Archive and it seems that the ActiveMonitor is
not worked in, yet. Are there any issues open or difficulties?
Cheers
Gerhard
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
Re: Cocoon 2.0 Scalability Disappointment
Posted by Berin Loritsch <bl...@apache.org>.
Gerhard Froehlich wrote:
>>-----Original Message-----
>>From: Chris Finn [mailto:yarnfish2001@yahoo.com]
>>
> As I did load tests with the RC1 I made the same experience.
> With the help of OptimizeIt (http://www.vmgear.com/) I discovered
> that most resource is burning the ClassLoader and stuff.
> See thread http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=99857216131771&w=2
>
> Berin was talking about a Active Monitor. But I don't know if
> he implemented this.
It's in Avalon Excalibur 4.0 release. Yes, I did implement it.
----------------------------------------------------
Sign Up for NetZero Platinum Today
Only $9.95 per month!
http://my.netzero.net/s/signup?r=platinum&refcd=PT97
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: Cocoon 2.0 Scalability Disappointment
Posted by Gerhard Froehlich <g-...@gmx.de>.
>-----Original Message-----
>From: Chris Finn [mailto:yarnfish2001@yahoo.com]
>
>--- Gerhard Froehlich <g-...@gmx.de> wrote:
>> 6) Look at the store parameters
>> (store,store-janitor,...) in the
>> cocoon.xconf file and customize them -as recommended
>> in the comments- for
>> your testing machine(s).
>
>This is one of the most important things to get right.
> The other thing I found to have a significant
>improvement on scalability was setting the -Xnoclassgc
>parameter on the Sun JDK 1.3.1--if I understand the
>documentation for this option correctly, it reduces
>the frequency of need for garbage collection by
>permitting the memory allocated to unused classes to
>be reused (instead of having to be collected and/or
>compacted). Less fragmentation means less collection
>means better response times.
>
>My particular experience with RC1 was that performance
>degrades over time under high stress--not clear if it
>is my particular app or XSPs in general. With 10
>simultaneous threads hitting the same page, requests
>1-500 perform lightning fast, 500-100 about 50%
>slower, and then from there on out gradually slowing
>down even further until a significant drop in load
>occurs.
As I did load tests with the RC1 I made the same experience.
With the help of OptimizeIt (http://www.vmgear.com/) I discovered
that most resource is burning the ClassLoader and stuff.
See thread http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=99857216131771&w=2
Berin was talking about a Active Monitor. But I don't know if
he implemented this.
>I've also found that *not* specifying an -Xms
>parameter seems to improve performance as well. That
>could be because I don't have the store janitor
>settings right, but I did see mention on an IBM site
>that suggested this for some cases.
Ok the "algorithm" is:
JVM.totalMemory() > Heapsize && JVM.freeMemory < Freememory
Heapsize and Freememory are the paramters from the cocoon.xconf
file.
If this is true the StoreJanitor kicks in, forces the gc() and
frees some Objects out of the stores.
See
http://xml.apache.org/cocoon/userdocs/concepts/storejanitor.html
This document describes how to set the parameters.
>Finally, turn on the verbose garbage collection JVM
>options to see if there are issues with too much
>GCing.
>
>For the long term, let me make two proposals:
>
>1) Would it make sense to start a document, either as
>part of the Cocoon document base, or as part of the
>Complete Idiot's Guide to Cocoon, of the best
>practices for tuning an application? I think we all
>have our tips and tricks we could collect, and based
>on the periodic complaints that come up in
>cocoon-users about performance, it might be a useful
>guide. I can start collecting info and putting
>something together with a little guidance on where it
>should reside, format, etc. And of course, your best
>suggestions.
Yeah something like a performance guide. Because every
release rises the same questions up ;).
>2) In observing the verbose GC messages and behavior
>under stress, I got to thinking about whether it would
>make sense to add a feature to the janitor that caused
>a full GC after a given period of idle time
>(regardless of the current memory state)?
>
>The goal would be to reduce the probability of those
>nasty full GC calls (which can hang the server for
>long periods of time) hitting during peak load times
>by keeping the decks clean during idle times.
>If the server has been idle for a minute (or a time
>interval specified in the cocoon.xconf) do a GC. If
>you have bursty sets of requests, the pending cleanup
>can be like a timebomb waiting for the next request to
>cause all the clean up and GC to occur.
I see, that's to consider. Maybe we should change this.
But it would be interesting the hear the other devs
to this issue.
But we should avoid to much active threads.
>By trying to distribute some of this "deep cleaning"
>into the idle times, we might be able to avoid some of
>the nasty requests.
>
>Or, it could be that I still haven't figured out the
>best way to set the janitor settings. ;) In which
>case, see suggestion 1.
>
>Chris
>
>
>
>__________________________________________________
>Do You Yahoo!?
>Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month.
>http://geocities.yahoo.com/ps/info1
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
>For additional commands, email: cocoon-dev-help@xml.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: Cocoon 2.0 Scalability Disappointment
Posted by Chris Finn <ya...@yahoo.com>.
--- Gerhard Froehlich <g-...@gmx.de> wrote:
> 6) Look at the store parameters
> (store,store-janitor,...) in the
> cocoon.xconf file and customize them -as recommended
> in the comments- for
> your testing machine(s).
This is one of the most important things to get right.
The other thing I found to have a significant
improvement on scalability was setting the -Xnoclassgc
parameter on the Sun JDK 1.3.1--if I understand the
documentation for this option correctly, it reduces
the frequency of need for garbage collection by
permitting the memory allocated to unused classes to
be reused (instead of having to be collected and/or
compacted). Less fragmentation means less collection
means better response times.
My particular experience with RC1 was that performance
degrades over time under high stress--not clear if it
is my particular app or XSPs in general. With 10
simultaneous threads hitting the same page, requests
1-500 perform lightning fast, 500-100 about 50%
slower, and then from there on out gradually slowing
down even further until a significant drop in load
occurs.
I've also found that *not* specifying an -Xms
parameter seems to improve performance as well. That
could be because I don't have the store janitor
settings right, but I did see mention on an IBM site
that suggested this for some cases.
Finally, turn on the verbose garbage collection JVM
options to see if there are issues with too much
GCing.
For the long term, let me make two proposals:
1) Would it make sense to start a document, either as
part of the Cocoon document base, or as part of the
Complete Idiot's Guide to Cocoon, of the best
practices for tuning an application? I think we all
have our tips and tricks we could collect, and based
on the periodic complaints that come up in
cocoon-users about performance, it might be a useful
guide. I can start collecting info and putting
something together with a little guidance on where it
should reside, format, etc. And of course, your best
suggestions.
2) In observing the verbose GC messages and behavior
under stress, I got to thinking about whether it would
make sense to add a feature to the janitor that caused
a full GC after a given period of idle time
(regardless of the current memory state)?
The goal would be to reduce the probability of those
nasty full GC calls (which can hang the server for
long periods of time) hitting during peak load times
by keeping the decks clean during idle times.
If the server has been idle for a minute (or a time
interval specified in the cocoon.xconf) do a GC. If
you have bursty sets of requests, the pending cleanup
can be like a timebomb waiting for the next request to
cause all the clean up and GC to occur.
By trying to distribute some of this "deep cleaning"
into the idle times, we might be able to avoid some of
the nasty requests.
Or, it could be that I still haven't figured out the
best way to set the janitor settings. ;) In which
case, see suggestion 1.
Chris
__________________________________________________
Do You Yahoo!?
Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month.
http://geocities.yahoo.com/ps/info1
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org
RE: Cocoon 2.0 Scalability Disappointment
Posted by Gerhard Froehlich <g-...@gmx.de>.
>1) disable logging. If log is DEBUG, it could generate Gigabytes of
>information and disks could become the bottleneck.
>
>[I think log might be the bottleneck]
>
>2) disable resource reloading. Same as above, the disk I/O system could
>become the bottleneck.
>
>3) try to come up with some logarithmic stress tests: 1 req/sec, 2
>req/sec, 4 req/sec, 8, 16, 32, 64, until your machines saturate.
>
>4) turn the cache off/on. [this, compared to the logaritmic approach
>should give us information on the caching system]
>
>5) try to get the generated source code of one of your XSP, compile it
>as a generator and use it as a normal generator instead. [this should
>remove the XSP loading/handling phase]
6) Look at the store parameters (store,store-janitor,...) in the
cocoon.xconf file and customize them -as recommended in the comments- for
your testing machine(s).
Cheers
Gerhard
---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org