Axis Response Time appears linear with load

["Sutton, Ray" <Ra...@den.galileo.com>]

Hello,

I've been tasked (for my sins, which must have been huge!) with performance
testing an application framework that sits on top of Tomcat and uses Axis as
the soap server. My initial testing slams the server as hard as possible
with 10 - 500 users stepping up 10 ever 60 seconds, each running an
identical script in an endless loop. The purpose being to identify any
points where synchronization was occurring.  I'm seeing a response time that
is almost linear with load (97% correlation) which implies, too me, a
synchronization issue. I started to localize the problem by isolating
components. Currently I have the soap method proxy turning the response
around without calling the application framework, so the only components
under test is Axis, ignoring the infrastructure which is a 4CPU 2Gb blade
server running Tomcat & the 1.4.1 JVM on RedHat Advanced Server.

I'm still seeing the linear response time change with increasing load.
Further investigation with JProbe (under XP) pointed the finger at
javax.xml.parsers.DocumentBuilderFactory. A JVM level thread dump with 10
users hitting the server showed org.apache.xerces.DocumentBuilderFactoryImpl
holding a lock and 3 other threads waiting on the lock.   

That's the background, now the question. 

Has anybody encountered a similar problem and could give me some pointers on
possible configuration options or other any other suggestion that might help
remove this bottleneck. 

If there's a well documented solution to this then, firstly I'd love a
pointer to it as a 30min search on Google failed to find anything that
seemed relevant.  Secondly I apologize for wasting people's time,
web-services/soap is a new area for me & I'm under some time pressure on
this task so it's quite possible I've missed the obvious.

Thanks
Ray Sutton
Systems Engineer




The information in this electronic mail message is sender's business
Confidential and may be legally privileged.  It is intended solely for the
addressee(s).  Access to this Internet electronic mail message by anyone
else is unauthorized.  If you are not the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be taken
in reliance on it is prohibited and may be unlawful. 
The sender believes that this E-mail and any attachments were free of any
virus, worm, Trojan horse, and/or malicious code when sent. This message and
its attachments could have been infected during  transmission. By reading
the message and opening any attachments, the recipient accepts full
responsibility for taking protective and remedial action about viruses and
other defects. Galileo International is not liable for any loss or damage
arising in any way from this message or its attachments.

Re: Axis Response Time appears linear with load

[Mike Spreitzer <ms...@us.ibm.com>]

Well, considering that just going through the message processing overhead 
of Tomcat and Axis is work, why do you not expect the response time to 
scale linearly with load?  BTW, did you mean to complain about the 
response time being proportional to load, or merely scaling linearly with 
it (i.e., being some a + b*load)?

My group has done some response time measurements of various frameworks 
and applications.  We typically give a system more than 60 seconds to 
stabilize and take a reading.  We typically find the response time grows 
more than linearly with load, if we explore sufficiently high loads.

Thanks,
Mike

Re: Axis Response Time appears linear with load

[Toshiyuki Kimura <to...@apache.org>]

Hi Ray and dims,

  The thread dump will be useful, I think so too.

  However, I'd like to add a few comments in other point
of view. The most interesting feature of your report was
the parser-related lock.

  Have you ever had a checking whether your component doesn't
instantiate a parser object on each transactions ? I mean that
you may meet a bottle-neck with the heavy-duty use, if you don't
share (i.e recycle) the instances.

  Axis does the reusing - as far as possible with the pooling
technics internally.  However, it may not enough to solve your
problem.  I'm not sure - what is the root problem of you.

Thanks,

  Toshi <to...@apache.org>

-----Original Message-----
From: Davanum Srinivas [mailto:dims@yahoo.com]
Sent: Friday, October 03, 2003 11:31 AM
To: axis-user@ws.apache.org
Subject: Re: Axis Response Time appears linear with load

Ray,

Can you please post the thread dump?

Thanks,
dims

--- "Sutton, Ray" <Ra...@den.galileo.com> wrote:
> Hello,
>
> I've been tasked (for my sins, which must have been huge!) with
performance
> testing an application framework that sits on top of Tomcat and uses
Axis as
> the soap server. My initial testing slams the server as hard as possible
> with 10 - 500 users stepping up 10 ever 60 seconds, each running an
> identical script in an endless loop. The purpose being to identify any
> points where synchronization was occurring.  I'm seeing a response time
that
> is almost linear with load (97% correlation) which implies, too me, a
> synchronization issue. I started to localize the problem by isolating
> components. Currently I have the soap method proxy turning the response
> around without calling the application framework, so the only components
> under test is Axis, ignoring the infrastructure which is a 4CPU 2Gb
blade
> server running Tomcat & the 1.4.1 JVM on RedHat Advanced Server.
>
> I'm still seeing the linear response time change with increasing load.
> Further investigation with JProbe (under XP) pointed the finger at
> javax.xml.parsers.DocumentBuilderFactory. A JVM level thread dump with
10
> users hitting the server showed
org.apache.xerces.DocumentBuilderFactoryImpl
> holding a lock and 3 other threads waiting on the lock.
>
> That's the background, now the question.
>
> Has anybody encountered a similar problem and could give me some
pointers on
> possible configuration options or other any other suggestion that might
help
> remove this bottleneck.
>
> If there's a well documented solution to this then, firstly I'd love a
> pointer to it as a 30min search on Google failed to find anything that
> seemed relevant.  Secondly I apologize for wasting people's time,
> web-services/soap is a new area for me & I'm under some time pressure on
> this task so it's quite possible I've missed the obvious.
>
> Thanks
> Ray Sutton
> Systems Engineer
>
> The information in this electronic mail message is sender's business
> Confidential and may be legally privileged.  It is intended solely for
the
> addressee(s).  Access to this Internet electronic mail message by anyone
> else is unauthorized.  If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
taken
> in reliance on it is prohibited and may be unlawful.
> The sender believes that this E-mail and any attachments were free of
any
> virus, worm, Trojan horse, and/or malicious code when sent. This message
and
> its attachments could have been infected during  transmission. By
reading
> the message and opening any attachments, the recipient accepts full
> responsibility for taking protective and remedial action about viruses
and
> other defects. Galileo International is not liable for any loss or
damage
> arising in any way from this message or its attachments.
>

=====
Davanum Srinivas - http://webservices.apache.org/~dims/

Re: Axis Response Time appears linear with load

[Davanum Srinivas <di...@yahoo.com>]

Ray,

Can you please post the thread dump?

Thanks,
dims

--- "Sutton, Ray" <Ra...@den.galileo.com> wrote:
> Hello,
> 
> I've been tasked (for my sins, which must have been huge!) with performance
> testing an application framework that sits on top of Tomcat and uses Axis as
> the soap server. My initial testing slams the server as hard as possible
> with 10 - 500 users stepping up 10 ever 60 seconds, each running an
> identical script in an endless loop. The purpose being to identify any
> points where synchronization was occurring.  I'm seeing a response time that
> is almost linear with load (97% correlation) which implies, too me, a
> synchronization issue. I started to localize the problem by isolating
> components. Currently I have the soap method proxy turning the response
> around without calling the application framework, so the only components
> under test is Axis, ignoring the infrastructure which is a 4CPU 2Gb blade
> server running Tomcat & the 1.4.1 JVM on RedHat Advanced Server.
> 
> I'm still seeing the linear response time change with increasing load.
> Further investigation with JProbe (under XP) pointed the finger at
> javax.xml.parsers.DocumentBuilderFactory. A JVM level thread dump with 10
> users hitting the server showed org.apache.xerces.DocumentBuilderFactoryImpl
> holding a lock and 3 other threads waiting on the lock.   
> 
> That's the background, now the question. 
> 
> Has anybody encountered a similar problem and could give me some pointers on
> possible configuration options or other any other suggestion that might help
> remove this bottleneck. 
> 
> If there's a well documented solution to this then, firstly I'd love a
> pointer to it as a 30min search on Google failed to find anything that
> seemed relevant.  Secondly I apologize for wasting people's time,
> web-services/soap is a new area for me & I'm under some time pressure on
> this task so it's quite possible I've missed the obvious.
> 
> Thanks
> Ray Sutton
> Systems Engineer
> 
> 
> 
> 
> The information in this electronic mail message is sender's business
> Confidential and may be legally privileged.  It is intended solely for the
> addressee(s).  Access to this Internet electronic mail message by anyone
> else is unauthorized.  If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be taken
> in reliance on it is prohibited and may be unlawful. 
> The sender believes that this E-mail and any attachments were free of any
> virus, worm, Trojan horse, and/or malicious code when sent. This message and
> its attachments could have been infected during  transmission. By reading
> the message and opening any attachments, the recipient accepts full
> responsibility for taking protective and remedial action about viruses and
> other defects. Galileo International is not liable for any loss or damage
> arising in any way from this message or its attachments.
> 
> 
> 


=====
Davanum Srinivas - http://webservices.apache.org/~dims/