You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by David Goodenough <da...@btconnect.com> on 2007/01/23 16:43:31 UTC
How to tune JSP performance?
I have a system with two identical machines running Tomcat 5.5.17
running identical applications. There are some extra things running
on one of the machines, but these problems occur even when they are
turned off.
The application consists of two parts. There is a JSP layer, and
a servlet (handling XML requests). The JSP lay talks to the XML
Servlet layer for all requests, the XML servlet layer talks to
a Postgresql DB on another machine.
The XML portions of this app are running almost identically (i.e.
within measurement tolerance) at the same speed, if I send a
null request to the XML servlet they respond quickly (including
loading and executing the sending routing both response within
.20x seconds (the x varies from test run to test run).
But using wget to invoke the JSP code gets back the logon page
almost instantly on one machine, and takes an age on the other.
Actually to find out that it needs to go to the logon page the JSP
does not need to use the XML servlet.
On the slow machine it takes 30 seconds to send back the 302
rewriting the URL so that it has a jsessionid, and then 16
seconds to serve the HTML for the logon page.
On the fast machine it takes less than a second to do the same
sequence.
Both machines seem to be using <10% of the machine CPU. There is
more than enough memory and no shortage of disk space.
Both machines are connected to the postgresql machine by TCP/IP
and actually the slower machine has a slightly better ping time
than the fast machine, but like the XML servlet transactions the
difference is within the margin of error.
Until this morning both machines were behaving the same, one
was probably slightly slower than the other due to the extra
things running on it, but you had to be watching quite closely
to notice the difference.
There does not seem to have been any changes to the system
that correspond with this change. Although slightly more
traffic seems to be going through it that yesterday, although
less than before Christmas when this slowdown was not noticable.
The two machines sit behind a load balancer, which is run
by the hosting organisation, they deny any changes and this
effect is measured on each machine sending the requests to
localhost so it does not seem likely to be part of the problem.
The machines are running linux 2.6.16. They are behind a solid
firewall, and there is nothing that would appear to be doing
anything odd on either system, no connected sockets to strange
places, no strange binaries or processes. Only the required
services are open to the outside world. Also the fact that the
XML Servlet seems to be running as normal suggests that the
problem is localised to the JSP processing.
There are no messages either in the tomcat logs or the system
logs that talk of errors at either the hardware or software
level.
The JSPs are precompiled (by Eclipse) and the WAR files have
not changed recently (they are both identical).
Anyone got any good ideas as to how I might go about diagnosing this?
Are there any good values available anywhere which would tell me
where the tomcat system is hitting a bottleneck.
David
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: How to tune JSP performance?
Posted by Pid <p...@pidster.com>.
Leon Rosenberg wrote:
> Hi David,
>
> 1. Have you double and tripple checked that all your configs on both
> machines as well as tomcat/jdk/OS versions are identical?
Of course, (at this risk of stating the obvious), there are some things
that shouldn't be identical... what network configuration are the two
machines in, in respect of clustering, for example?
(Apologies if I missed this from earlier in the thread.)
They're sharing a database, what else are they sharing and is anything
controlling the routing of packets to them?
Try enabling the RequestDumperValve, and increasing the logging levels
to get a more forensic view of what the server is doing.
p
> 2. Have you tried rebooting? There were some memory bugs with later
> kernel, I don't remember if the 2.6.16 is affected, but just in case.
>
> 3. How exactly do you measure? From outside by jmeter or something, or
> from inside?
>
> 4. Are you testing with a single connection or with multiple connections?
>
> regards
> Leon
>
> On 1/23/07, David Goodenough <da...@btconnect.com> wrote:
>> I have a system with two identical machines running Tomcat 5.5.17
>> running identical applications. There are some extra things running
>> on one of the machines, but these problems occur even when they are
>> turned off.
>>
>> The application consists of two parts. There is a JSP layer, and
>> a servlet (handling XML requests). The JSP lay talks to the XML
>> Servlet layer for all requests, the XML servlet layer talks to
>> a Postgresql DB on another machine.
>>
>> The XML portions of this app are running almost identically (i.e.
>> within measurement tolerance) at the same speed, if I send a
>> null request to the XML servlet they respond quickly (including
>> loading and executing the sending routing both response within
>> .20x seconds (the x varies from test run to test run).
>>
>> But using wget to invoke the JSP code gets back the logon page
>> almost instantly on one machine, and takes an age on the other.
>> Actually to find out that it needs to go to the logon page the JSP
>> does not need to use the XML servlet.
>>
>> On the slow machine it takes 30 seconds to send back the 302
>> rewriting the URL so that it has a jsessionid, and then 16
>> seconds to serve the HTML for the logon page.
>>
>> On the fast machine it takes less than a second to do the same
>> sequence.
>>
>> Both machines seem to be using <10% of the machine CPU. There is
>> more than enough memory and no shortage of disk space.
>>
>> Both machines are connected to the postgresql machine by TCP/IP
>> and actually the slower machine has a slightly better ping time
>> than the fast machine, but like the XML servlet transactions the
>> difference is within the margin of error.
>>
>> Until this morning both machines were behaving the same, one
>> was probably slightly slower than the other due to the extra
>> things running on it, but you had to be watching quite closely
>> to notice the difference.
>>
>> There does not seem to have been any changes to the system
>> that correspond with this change. Although slightly more
>> traffic seems to be going through it that yesterday, although
>> less than before Christmas when this slowdown was not noticable.
>>
>> The two machines sit behind a load balancer, which is run
>> by the hosting organisation, they deny any changes and this
>> effect is measured on each machine sending the requests to
>> localhost so it does not seem likely to be part of the problem.
>>
>> The machines are running linux 2.6.16. They are behind a solid
>> firewall, and there is nothing that would appear to be doing
>> anything odd on either system, no connected sockets to strange
>> places, no strange binaries or processes. Only the required
>> services are open to the outside world. Also the fact that the
>> XML Servlet seems to be running as normal suggests that the
>> problem is localised to the JSP processing.
>>
>> There are no messages either in the tomcat logs or the system
>> logs that talk of errors at either the hardware or software
>> level.
>>
>> The JSPs are precompiled (by Eclipse) and the WAR files have
>> not changed recently (they are both identical).
>>
>> Anyone got any good ideas as to how I might go about diagnosing this?
>> Are there any good values available anywhere which would tell me
>> where the tomcat system is hitting a bottleneck.
>>
>> David
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: How to tune JSP performance?
Posted by Leon Rosenberg <ro...@googlemail.com>.
Hi David,
1. Have you double and tripple checked that all your configs on both
machines as well as tomcat/jdk/OS versions are identical?
2. Have you tried rebooting? There were some memory bugs with later
kernel, I don't remember if the 2.6.16 is affected, but just in case.
3. How exactly do you measure? From outside by jmeter or something, or
from inside?
4. Are you testing with a single connection or with multiple connections?
regards
Leon
On 1/23/07, David Goodenough <da...@btconnect.com> wrote:
> I have a system with two identical machines running Tomcat 5.5.17
> running identical applications. There are some extra things running
> on one of the machines, but these problems occur even when they are
> turned off.
>
> The application consists of two parts. There is a JSP layer, and
> a servlet (handling XML requests). The JSP lay talks to the XML
> Servlet layer for all requests, the XML servlet layer talks to
> a Postgresql DB on another machine.
>
> The XML portions of this app are running almost identically (i.e.
> within measurement tolerance) at the same speed, if I send a
> null request to the XML servlet they respond quickly (including
> loading and executing the sending routing both response within
> .20x seconds (the x varies from test run to test run).
>
> But using wget to invoke the JSP code gets back the logon page
> almost instantly on one machine, and takes an age on the other.
> Actually to find out that it needs to go to the logon page the JSP
> does not need to use the XML servlet.
>
> On the slow machine it takes 30 seconds to send back the 302
> rewriting the URL so that it has a jsessionid, and then 16
> seconds to serve the HTML for the logon page.
>
> On the fast machine it takes less than a second to do the same
> sequence.
>
> Both machines seem to be using <10% of the machine CPU. There is
> more than enough memory and no shortage of disk space.
>
> Both machines are connected to the postgresql machine by TCP/IP
> and actually the slower machine has a slightly better ping time
> than the fast machine, but like the XML servlet transactions the
> difference is within the margin of error.
>
> Until this morning both machines were behaving the same, one
> was probably slightly slower than the other due to the extra
> things running on it, but you had to be watching quite closely
> to notice the difference.
>
> There does not seem to have been any changes to the system
> that correspond with this change. Although slightly more
> traffic seems to be going through it that yesterday, although
> less than before Christmas when this slowdown was not noticable.
>
> The two machines sit behind a load balancer, which is run
> by the hosting organisation, they deny any changes and this
> effect is measured on each machine sending the requests to
> localhost so it does not seem likely to be part of the problem.
>
> The machines are running linux 2.6.16. They are behind a solid
> firewall, and there is nothing that would appear to be doing
> anything odd on either system, no connected sockets to strange
> places, no strange binaries or processes. Only the required
> services are open to the outside world. Also the fact that the
> XML Servlet seems to be running as normal suggests that the
> problem is localised to the JSP processing.
>
> There are no messages either in the tomcat logs or the system
> logs that talk of errors at either the hardware or software
> level.
>
> The JSPs are precompiled (by Eclipse) and the WAR files have
> not changed recently (they are both identical).
>
> Anyone got any good ideas as to how I might go about diagnosing this?
> Are there any good values available anywhere which would tell me
> where the tomcat system is hitting a bottleneck.
>
> David
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: How to tune JSP performance?
Posted by David Goodenough <da...@btconnect.com>.
On Tuesday 23 January 2007 18:14, Christopher Schultz wrote:
> David,
>
> David Goodenough wrote:
> > Currently it looks to me as though the problem is with the multicast
> > messages for the clustering. I found a timeout message which is
> > suggesting that for some reason although the slow machine can send
> > messages to the fast, the slow machine never gets a reply.
>
> Check the routing tables (and iptables rules) on both machines. If you
> use routing tricks to send your clustering messages through a secondary
> interface, then you might be running into something weird there.
no, we have only one active interface on each machine.
David
>
> -chris
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: How to tune JSP performance?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
David,
David Goodenough wrote:
> Currently it looks to me as though the problem is with the multicast
> messages for the clustering. I found a timeout message which is suggesting
> that for some reason although the slow machine can send messages to the fast,
> the slow machine never gets a reply.
Check the routing tables (and iptables rules) on both machines. If you
use routing tricks to send your clustering messages through a secondary
interface, then you might be running into something weird there.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFtlBx9CaO5/Lv0PARAmlJAKCqevhUmzrEpaQgITUFuTXU8npdcwCgg8Tj
vezYZft30OxaxeW0VWrJ9Aw=
=0EDW
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: How to tune JSP performance?
Posted by David Goodenough <da...@btconnect.com>.
On Tuesday 23 January 2007 15:51, Mikolaj Rydzewski wrote:
> David Goodenough wrote:
> > On the slow machine it takes 30 seconds to send back the 302
> > rewriting the URL so that it has a jsessionid, and then 16
> > seconds to serve the HTML for the logon page.
>
> Looks like a DNS timeout value. Is the DNS system properly configured on
> both machines?
DNS configs are the same on both machines, and on both machines if I type
ping bbc.co.uk I get a header line with the IP address back instantly on
both machines.
Currently it looks to me as though the problem is with the multicast
messages for the clustering. I found a timeout message which is suggesting
that for some reason although the slow machine can send messages to the fast,
the slow machine never gets a reply.
David
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: How to tune JSP performance?
Posted by Mikolaj Rydzewski <mi...@ceti.pl>.
David Goodenough wrote:
> On the slow machine it takes 30 seconds to send back the 302
> rewriting the URL so that it has a jsessionid, and then 16
> seconds to serve the HTML for the logon page.
>
Looks like a DNS timeout value. Is the DNS system properly configured on
both machines?
--
Mikolaj Rydzewski <mi...@ceti.pl>