Performance problems with 2.1.10 using FOM JavaScript cocoon.load()

["(Berry) A.W. van Halderen" <b....@hippo.nl>]

Dear all,

The following issue I've put up to the users mailing list, with no prevail, so
I'd kindly ask you people for some help.

The problem is that when upgrading to Cocoon 2.1.10 for one of our software
packages to make use of all the improvements in 2.1.10, we're hitting an
enormous performance hit.  We basically see that requests which would normally
take 10ths to 100thms of microseconds now takes seconds upto a minut to
execute, where the processor is fully used.

Now basically we needed to make no modifications to our software to use 2.1.10
apart from updating dependencies.  We think therefor that the bad performance
comes from some way we use Cocoon, especially some classloading issue.

I've profiles the application to some extent, and I'm seeing that the
application is in fact slow because when executing a server-side JavaScript
file, it is loading other JavaScript files (through cocoon.load()).  This
loading is slow, because even though it seems to be cached, it is calling the
classloader to look-up the cache definitions of all referenced Java code.
This makes an enormous hotspot in the Jetty ContextLoader classloader, which
can return a reference to the class quite quickly, but has to give a lot of
them.

Now the total application, accessing a single page results in a number of
execution of JavaScript files, and a JavaScript file is executed more than
once.  The most important hint probably is that I've looked at the most
important JavaScript file, and I'm seeing that when executing this script more
than once that in its execution of cocoon.load()'s of other files it is
ALTERNATING fast and slow.  And I really mean one time fast, then slow, then
fast, then slow...  When tracing back, there is no difference in its call
stack between the two cases.

I'd really appreciate any hints at the moment where to look at, because I'm
running out of ideas where to trace the beast next.

Thanx,
\Berry

-- 
Berry A.W. van Halderen           b.vanhalderen@hippo.nl / berry@halderen.net
Disclaimer: the above is the author's personal opinion and is not the opinion
or policy of his employer or of the little green men that have been following
him all day.

Re: Performance problems with 2.1.10 using FOM JavaScript cocoon.load()

["(Berry) A.W. van Halderen" <b....@hippo.nl>]

On Tue, Mar 06, 2007 at 08:21:33AM -0500, Vadim Gritsenko wrote:
> Cocoon 2.1.10 has latest version of rhino (js-1.6r5), while cocoon 2.1.9 
> comes with older 1.5r4. This change probably is the cause of the issue, try 
> comparing behavior of these two rhino versions.

Actually, even the js1.6r5 causes problems, I'm using a recent snapshot
of the Cocoon source tree which already includes js1.6r5.  However, after
some remarks from others too, I've tried to build Rhino from source, to
use the 1.6R6pre.
And this improves things a lot.  I'm not so far to say everything is back
to normal speed, but it is a dawn lot closer.

> Vadim

Thanx for your comments.

\Berry
-- 
Berry A.W. van Halderen           b.vanhalderen@hippo.nl / berry@halderen.net
Disclaimer: the above is the author's personal opinion and is not the opinion
or policy of his employer or of the little green men that have been following
him all day.

Re: Performance problems with 2.1.10 using FOM JavaScript cocoon.load()

[Vadim Gritsenko <va...@reverycodes.com>]

(Berry) A.W. van Halderen wrote:
> The problem is that when upgrading to Cocoon 2.1.10 for one of our software
> packages to make use of all the improvements in 2.1.10, we're hitting an
> enormous performance hit.  We basically see that requests which would normally
> take 10ths to 100thms of microseconds now takes seconds upto a minut to
> execute, where the processor is fully used.
> 
> Now basically we needed to make no modifications to our software to use 2.1.10
> apart from updating dependencies.  We think therefor that the bad performance
> comes from some way we use Cocoon, especially some classloading issue.
> 
> I've profiles the application to some extent, and I'm seeing that the
> application is in fact slow because when executing a server-side JavaScript
> file, it is loading other JavaScript files (through cocoon.load()).  This
> loading is slow, because even though it seems to be cached, it is calling the
> classloader to look-up the cache definitions of all referenced Java code.
> This makes an enormous hotspot in the Jetty ContextLoader classloader, which
> can return a reference to the class quite quickly, but has to give a lot of
> them.

Cocoon 2.1.10 has latest version of rhino (js-1.6r5), while cocoon 2.1.9 comes 
with older 1.5r4. This change probably is the cause of the issue, try comparing 
behavior of these two rhino versions.

Vadim

> Now the total application, accessing a single page results in a number of
> execution of JavaScript files, and a JavaScript file is executed more than
> once.  The most important hint probably is that I've looked at the most
> important JavaScript file, and I'm seeing that when executing this script more
> than once that in its execution of cocoon.load()'s of other files it is
> ALTERNATING fast and slow.  And I really mean one time fast, then slow, then
> fast, then slow...  When tracing back, there is no difference in its call
> stack between the two cases.
> 
> I'd really appreciate any hints at the moment where to look at, because I'm
> running out of ideas where to trace the beast next.
> 
> Thanx,
> \Berry

Re: Performance problems with 2.1.10 using FOM JavaScript cocoon.load()

["(Berry) A.W. van Halderen" <b....@hippo.nl>]

On Tue, Mar 06, 2007 at 10:37:04AM +0100, Bertrand Delacretaz wrote:
> On 3/5/07, (Berry) A.W. van Halderen <b....@hippo.nl> wrote:
> Note that I've had to moderate your message in, please subscribe!
Hmm, I'm subscribed, but apparently I've missed that it was under an alias
rather than the actual e-mail address.  The account names are a bit
confusing.
Thanks for bringing it to my attention, I should have solved this now.

> Did you check network activity when this happens? Makes me think of
> something fetching a DTD on the web...this doesn't fit with the rest
> of your description, but it might be good to check.

Network activity isn't the problem, and I'm tracing all activity, but I'll
make sure this isn't a side problem.
I'm a lot closer though by replacing the Rhino package with a prereleased
version (see other posting).

\Berry
-- 
Berry A.W. van Halderen           b.vanhalderen@hippo.nl / berry@halderen.net
Disclaimer: the above is the author's personal opinion and is not the opinion
or policy of his employer or of the little green men that have been following
him all day.

Re: Performance problems with 2.1.10 using FOM JavaScript cocoon.load()

[Bertrand Delacretaz <bd...@apache.org>]

On 3/5/07, (Berry) A.W. van Halderen <b....@hippo.nl> wrote:
> Dear all,...

Note that I've had to moderate your message in, please subscribe!

> ...The most important hint probably is that I've looked at the most
> important JavaScript file, and I'm seeing that when executing this script more
> than once that in its execution of cocoon.load()'s of other files it is
> ALTERNATING fast and slow.  And I really mean one time fast, then slow, then
> fast, then slow...

Did you check network activity when this happens? Makes me think of
something fetching a DTD on the web...this doesn't fit with the rest
of your description, but it might be good to check.

-Bertrand