You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Vasanth Kumar ravi <jo...@gmail.com> on 2008/11/19 03:42:35 UTC
Question on Performance Tuning
Folks,
I have some queries related to the performance tuning of Tomcat V6.
I did search some search on the internet and found very little with regards
to tuning.
My application setup is as follows.
2 Apache web servers.
4 Tomcat application servers
2 database servers.
We did many rounds of load/performance testing with 50 Virtual users, to
find bottlenecks with the application, major problem found was CPU
utilization.
1. We did load testing with 1 apache and 1 tomcat, the CPU was arnd 70%
2. We did load testing with 2 apache and 2 tomcat, the CPU was arnd 70%
3.We did load testing with 2 apache and 3 tomcat, the CPU was arnd 70%
4.We did load testing with 2 apache and 4 tomcat, the CPU was arnd 70%
Each tomcat has a heap size of min/max = 2 GB.
JAVA_OPTS="$JAVA_OPTS "-Xms2048m" "-Xmx2048m" "-XX:MaxPermSize=128m"
"-XX:PermSize=128m" "-XX:NewRatio=3" "-XX:NewSize=256m"
"-XX:MaxNewSize=256m" "-XX:SurvivorRatio=6" "-XX:+UseConcMarkSweepGC"
"-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+DisableExplicitGC"
"-Xloggc:logs/gc.log" "-Denv=beesptesxapp14" "-Denvironment=etestqa"
"-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser"
"-Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl"
"-Dinstance=homepage_beesptesxapp14"
"-javaagent:/opt/shared/applications/homepage/lib/aspectjweaver-1.5.3.jar
We did take thread dumps while performing the test, and found nothing
related to application bottlenecks.
Can I get some pointers on anything to be tuned in the tomcat to reduce the
cpu utilization.
Thanks.
--
Regards&Thanks,
Vasanth Kumar Ravi
Re: Question on Performance Tuning
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Vasanth,
Vasanth Kumar ravi wrote:
> Chuck
> We tried using the jprobe to profile the application. Now we are trying out
> lighter profiler as the jprobe was a heavy one.
Don't worry too much about the "weight" of the profiler: the whole point
is to see what part of your /application/ is using the CPU. Even if the
app runs at 50% speed, you should be able to get good data. Don't sit
around waiting for a better profiler when you have one in front of you.
> But found nothing suspicious.
...? So Java is using a lot of your CPU and when you profile it, it doesn't?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkk0MIAACgkQ9CaO5/Lv0PAPzQCcDnjYDJuv5NN408noNP4Xtkkp
0OMAn0R3Qsk9Su3INjNfuf10WTsOk8/K
=4Tyw
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Question on Performance Tuning
Posted by Vasanth Kumar ravi <jo...@gmail.com>.
Chuck
We tried using the jprobe to profile the application.Now we are trying out
lighter profiler as the jprobe was a heavy one.
But found nothing suspicious.
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_16-b02, mixed mode)
On Fri, Nov 21, 2008 at 9:49 AM, Caldarale, Charles R <
Chuck.Caldarale@unisys.com> wrote:
> > From: Vasanth Kumar ravi [mailto:josvasanth@gmail.com]
> > Subject: Re: Question on Performance Tuning
> >
> > But we have not specified the -server option in the JAVA_OPTS.
> > Is it must to have -server option specified when you are
> > using a 64 bit Java.
>
> Typically, 64-bit JVMs run only in -server mode, so you don't have to
> specify anything. Do "java -version" from a shell window to verify.
>
> > To identify the surge in CPU, I tested the following scenario's yday.
>
> Fine, but what about the profiling we suggested you do? Without that data,
> everything else is just speculation. Run a profiler, and find out what's
> using up the CPU cycles.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Regards&Thanks,
Vasanth Kumar Ravi
RE: Question on Performance Tuning
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Vasanth Kumar ravi [mailto:josvasanth@gmail.com]
> Subject: Re: Question on Performance Tuning
>
> But we have not specified the -server option in the JAVA_OPTS.
> Is it must to have -server option specified when you are
> using a 64 bit Java.
Typically, 64-bit JVMs run only in -server mode, so you don't have to specify anything. Do "java -version" from a shell window to verify.
> To identify the surge in CPU, I tested the following scenario's yday.
Fine, but what about the profiling we suggested you do? Without that data, everything else is just speculation. Run a profiler, and find out what's using up the CPU cycles.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Question on Performance Tuning
Posted by Vasanth Kumar ravi <jo...@gmail.com>.
Currently we are running Java 1.5 with 64 bit version on the superdome
server with Redhat O/S.
But we have not specified the -server option in the JAVA_OPTS.
Is it must to have -server option specified when you are using a 64 bit
Java.
The CPU utilization nags me very much.
The performance of the application pages/transactions are meeting our
targets except the cpu target.
We have the same application running on Weblogic/Apache/Oracle on Solaris
O/S in which the CPU for 100 virtual user is 50-60%.
To identify the surge in CPU, I tested the following scenario's yday.
2 users - Single Tomcat - 12-18% of CPU
4 users - Single Tomcat - 20-30% of CPU
6 users - Single Tomcat - 40-50% of CPU
More than 6 users - Single/Double Tomcat - CPU utilization is 70-80%.
Even if I test with 25/50/100 users the CPU maintains at 70%.
We have max threads as 40 and I increased to 60 but did not find any
difference in CPU.
My application is an internet based one which currently has half a million
page views everyday.
In the near future it may exceed one million.
I cannot test more than 100 users as we do not have license for more than
100 users.
Is it a normal behavior of the tomcat ?
On Wed, Nov 19, 2008 at 11:01 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Vasanth,
>
> Vasanth Kumar ravi wrote:
> > We did many rounds of load/performance testing with 50 Virtual users, to
> > find bottlenecks with the application, major problem found was CPU
> > utilization.
> >
> > 1. We did load testing with 1 apache and 1 tomcat, the CPU was arnd 70%
> > 2. We did load testing with 2 apache and 2 tomcat, the CPU was arnd 70%
> > 3. We did load testing with 2 apache and 3 tomcat, the CPU was arnd 70%
> > 4. We did load testing with 2 apache and 4 tomcat, the CPU was arnd 70%
>
> Was your load test designed to test your peak load expectations? IMO,
> 70% CPU utilization means that you have appropriately sized your
> hardware for your demand. I would call that a successful capacity
> planning job: you even have a bit of room to grow before adding more
> hardware.
>
> Are you just upset that you are using such a high percentage of your CPU?
>
> - -chris
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkkkKmYACgkQ9CaO5/Lv0PAwsgCghztrPqEqZr8qOgU9D/wQICcd
> oFsAoKeDf44p9qPHOr0jlEA3LA1lEUhc
> =lcEb
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Regards&Thanks,
Vasanth Kumar Ravi
Re: Question on Performance Tuning
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Vasanth,
Vasanth Kumar ravi wrote:
> We did many rounds of load/performance testing with 50 Virtual users, to
> find bottlenecks with the application, major problem found was CPU
> utilization.
>
> 1. We did load testing with 1 apache and 1 tomcat, the CPU was arnd 70%
> 2. We did load testing with 2 apache and 2 tomcat, the CPU was arnd 70%
> 3. We did load testing with 2 apache and 3 tomcat, the CPU was arnd 70%
> 4. We did load testing with 2 apache and 4 tomcat, the CPU was arnd 70%
Was your load test designed to test your peak load expectations? IMO,
70% CPU utilization means that you have appropriately sized your
hardware for your demand. I would call that a successful capacity
planning job: you even have a bit of room to grow before adding more
hardware.
Are you just upset that you are using such a high percentage of your CPU?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkkkKmYACgkQ9CaO5/Lv0PAwsgCghztrPqEqZr8qOgU9D/wQICcd
oFsAoKeDf44p9qPHOr0jlEA3LA1lEUhc
=lcEb
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Question on Performance Tuning
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Vasanth Kumar ravi [mailto:josvasanth@gmail.com]
> Subject: Re: Question on Performance Tuning
>
> One question thou..
>
> in the JAVA_OPTS do we have to specify -server cos it runs as
> client by default. Will it bring any change to performance ?
Usually it will help reduce the path length of the Java code by 10-15%; whether or not that will have any measurable effect in your environment depends on where you're consuming the CPU cycles. The server JIT takes longer to kick in than the client JIT does, so there's a longer ramp-up period as the JVM optimizes the heavily used methods.
Note that a 64-bit JVM runs only in -server mode, and allows a much bigger heap.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Question on Performance Tuning
Posted by Vasanth Kumar ravi <jo...@gmail.com>.
Thanks Chuck..
One question thou..
in the JAVA_OPTS do we have to specify -server cos it runs as client by
default.
Will it bring any change to performance ?
On Wed, Nov 19, 2008 at 12:27 PM, Caldarale, Charles R <
Chuck.Caldarale@unisys.com> wrote:
> > From: Vasanth Kumar ravi [mailto:josvasanth@gmail.com]
> > Subject: Question on Performance Tuning
> >
> > We did take thread dumps while performing the test, and
> > found nothing related to application bottlenecks.
>
> Thread dumps are appropriate for diagnosing hangs, but they're not terribly
> useful for analyzing CPU usage. You'd be better off running a profiler to
> see where time is being spent. Even the JVM's built-in hprof running in
> sample mode should give you a pretty good idea of where the cycles are going
> (or use JVisualVM).
>
> > Can I get some pointers on anything to be tuned in
> > the tomcat to reduce the cpu utilization.
>
> First, find out what's really using the CPU before fiddling with anything.
> Tweaking Tomcat is not likely to change much assuming the thread pool and
> heap are adequate for your load.
>
> You've specified several GC options that may or may not be appropriate.
> Unless you've proven that changing NewRatio, NewSize, MaxNewSize, and
> SurvivorRatio actually helps your throughput, you're likely better off
> leaving them alone. 128M MaxPermSize may be too small if you have a lot of
> classes in your webapps. Enabling concurrent GC produces more regular
> response times, but will actually *reduce* throughput in most cases.
>
> These might help slightly:
> * Remove AspectJ.
> * Turn off PrintGCDetails.
> * Remove unused webapps.
> * Precompile JSPs.
> * Turn off auto-deployment.
> * Turn off monitored resources.
> * Set the parameters for the JSP servlet (in conf/web.xml) for production.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Regards&Thanks,
Vasanth Kumar Ravi
RE: Question on Performance Tuning
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Vasanth Kumar ravi [mailto:josvasanth@gmail.com]
> Subject: Question on Performance Tuning
>
> We did take thread dumps while performing the test, and
> found nothing related to application bottlenecks.
Thread dumps are appropriate for diagnosing hangs, but they're not terribly useful for analyzing CPU usage. You'd be better off running a profiler to see where time is being spent. Even the JVM's built-in hprof running in sample mode should give you a pretty good idea of where the cycles are going (or use JVisualVM).
> Can I get some pointers on anything to be tuned in
> the tomcat to reduce the cpu utilization.
First, find out what's really using the CPU before fiddling with anything. Tweaking Tomcat is not likely to change much assuming the thread pool and heap are adequate for your load.
You've specified several GC options that may or may not be appropriate. Unless you've proven that changing NewRatio, NewSize, MaxNewSize, and SurvivorRatio actually helps your throughput, you're likely better off leaving them alone. 128M MaxPermSize may be too small if you have a lot of classes in your webapps. Enabling concurrent GC produces more regular response times, but will actually *reduce* throughput in most cases.
These might help slightly:
* Remove AspectJ.
* Turn off PrintGCDetails.
* Remove unused webapps.
* Precompile JSPs.
* Turn off auto-deployment.
* Turn off monitored resources.
* Set the parameters for the JSP servlet (in conf/web.xml) for production.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org