Strange delays after about 800 users on a server -- tomcat 4.1.27, apache 2.0.54, LB/SSL on a Cisco ACE

[Shaun Qualheim <kb...@yahoo.com>]

Hello all --

Running with the following:
*3 Dell Poweredge 1950 servers (2x quad core xeons, 8 GB RAM, 146 GB 15k SAS in Raid 1) running CentOS 5.1 x86_64
   *Apache 2.0.54 with mod_jk
   *Tomcat 4.1.27-LE
   *jdk1.5.0_15

These 3 servers are behind a Cisco ACE load balancer.  We're using SSL offloading with the load balancer.

Apache is listening on ports 80 and 81 using http.  Port 80 immediately rewrites everything to an https:// url.  That https:// url goes to the load balancer on 443 and is passed to the appliance on port 81.

Using mod_jk on Apache to balance our  Tomcats.  We have 3 tomcat instances running.

When we use the our application with all http (port 80 doesn't rewrite,) the system works fine when we run a load test.  We can push 2100 concurrent users out of each server.

However... using the same setup beyond the load balancer, we are only about to get to about 2500 concurrent users across the three servers (about 800 per server) before we start seeing very long delays (1-2 minutes where we should be seeing a few seconds) on miscellaneous functions throughout the application.  

We've run load tests on this several times using values that are known to work well when we're using HTTP only (5000 concurrent users with a ramp-up of 50 users/sec.)  We've monitored resources on the Cisco ACE load balancer and things there seem just fine... no bandwidth issues, no system resource concerns.

We did have to modify the server.xml files for each tomcat instance to have a "scheme=https proxyPort=443" on the toms to let the tomcats and app know that we're behind an SSL offloader, not simply the http traffic that they see traffic as.

We're pretty perplexed on why the sudden slowdown happens at about 800 users per server.  It works fine when we're http only.  We don't see anything that stands out in the apache or catalina logs that would seem to be concerning (broken pipes, abnormal timeouts, etc.)  I would greatly appreciate any help anyone can offer us.

Thank you very much,

Shaun Qualheim



      

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Strange delays after about 800 users on a server -- tomcat 4.1.27, apache 2.0.54, LB/SSL on a Cisco ACE

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Shaun,

On 7/8/2009 1:35 PM, Shaun Qualheim wrote:
>    *Tomcat 4.1.27-LE

You might consider upgrading at some point. 4.1 is getting ready to be
retired, and the 3 (yes 3!) versions since then all have significant
performance improvements that may help your situation.

> Apache is listening on ports 80 and 81 using http. Port 80 
> immediately rewrites everything to an https:// url. That https:// url
> goes to the load balancer on 443 and is passed to the appliance on port 81.

Wait... what?

client -- http:80 --> httpd -- https:443 --> lb -- http:81 --> Tomcat?

You said you have 3 TC instances. Is that 1 instance on each of 3
separate physical servers, so port 81 is used on them all? Why bother
with SSL after the request is within your network? Also, why have httpd
forward all traffic to a load-balancer instead of just doing the
load-balancing itself?

Maybe I have misunderstood your setup, but it seems overly complicated.

> When we use the our application with all http (port 80 doesn't
> rewrite,) the system works fine when we run a load test. We can push
> 2100 concurrent users out of each server.

Okay.

> However... using the same setup beyond the load balancer, we are 
> only about to get to about 2500 concurrent users across the three
> servers (about 800 per server) before we start seeing very long
> delays (1-2 minutes where we should be seeing a few seconds) on
> miscellaneous functions throughout the application.

Does the load-balancer have any kind of traffic shaping or
bandwidth/connection limiting configured?

What do your <Connector> elements look like in Tomcat's server.xml?

> We're pretty perplexed on why the sudden slowdown happens at about
> 800 users per server. It works fine when we're http only. We don't see
> anything that stands out in the apache or catalina logs that would seem
> to be concerning (broken pipes, abnormal timeouts, etc.) I would greatly
> appreciate any help anyone can offer us.

So, does it look like you are hitting a wall (like there aren't enough
connections allowed) or does the application/server start to experience
an actual slowdown (like high CPU load, lots of paging, etc.)?

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkpY/oEACgkQ9CaO5/Lv0PDlnwCgn1wLVEtYDVZiAc3JIDVL9LQY
ZLIAoIwDuDf8tP1UAthX1aYxzk0xjC6n
=NJun
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org