You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Talha Fazal <tf...@credera.com> on 2011/09/02 15:31:36 UTC
Tomcat Performance Turning.
Background:
We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
The Issue - Load Testing:
In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
Setup for Load Testing:
We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
Mentioned below is the version information:
Apache Version 2.2.14 (with mod_jk module)
Tomcat: 6.0.29
Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
Connection Pool: DBCP.
Mentioned below are connector settings in conf/server.xml:
<Connector
address="stagingTCserver01"
backlog="300"
connectionTimeout="60000"
enableLookups="false"
maxPostSize="2097152"
maxSpareThreads="10"
maxThreads="30"
minSpareThreads="5"
port="8006"
protocol="AJP/1.3"
tcpNoDelay="true"
/>
Mentioned below are the settings for JNDI resource configured in conf/context.xml:
<Resource
name="jdbc/onlinedb"
auth="Container"
type="javax.sql.DataSource"
driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
initialSize="1"
maxActive="30"
minIdle="1"
maxIdle="5"
maxWait="300000"
poolPreparedStatements="true"
maxOpenPreparedStatements="300"
validationQuery="SELECT 1 FROM BB_DUAL"
testOnBorrow="true"
validationInterval="10000"
testWhileIdle="true"
/>
JVM Parameters:
-Xms512m -Xmx2048m
-XX:PermSize=128m -XX:MaxPermSize=256m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
-verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
It must be noted that Weblogic setup performs very well using similar settings.
Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
Any tips/pointers will be greatly appreciated.
Talha.
Re: Tomcat Performance Turning.
Posted by Leon Rosenberg <ro...@gmail.com>.
Hello Talha,
On Fri, Sep 2, 2011 at 11:34 PM, Talha Fazal <tf...@credera.com> wrote:
> Q. Have you tried running directly against tomcat without apache inbetween?
> A. Yes; this resolves the performance bottleneck.
Than its pretty clear where to search, isn't it?
>
> Q. <holywar>from my experience there is no need for apache in your setup anyway </holywar>.
> A. :-) The reasons for having apache in front:
>A) Create a DMZ and protect Tomcat from being exposed to outside attacks;
You know that httpd does nothing but forwards each and single request
to tomcat? So you are not really protect them, you just add another
(time consuming) layer.
> B) Load balancing.
You mentioned that you have 2 httpd servers, so all the traffic to 6
(or was it eight) tomcat servers are going through the two httpds?
And how do you loadbalancer the 2 httpd servers?
Regards
Leon
>
> Thanks Leon!
>
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
> Sent: Friday, September 02, 2011 3:19 PM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> On Fri, Sep 2, 2011 at 6:16 PM, Talha Fazal <tf...@credera.com> wrote:
>> PLEASE SEE MY ANSWERS BELOW IN UPPER CAPS.
>>
>> ONE IMPORTANT OBSERVATION: Even though, tomcat's response time is decent, for some reason, apache is taking a lot longer to serve requests. We are using mod_jk with Tomcat using AJP1.3 protocol in the connector setting in server.xml. We plan to try mod_proxy_ajp and mod_proxy instead of mod_jk. Any thoughts here?
>
> Interesting observation. Have you tried running directly against
> tomcat without apache inbetween?
> <holywar>from my experience there is no need for apache in your setup
> anyway </holywar>.
>
> In the past we used both mod_jk and mod_proxy_ajp, and even
> mod_proxy_ajp is somewhat faster, the difference where never 17
> seconds.
> Your best guess would probably be to test without httpd to rule out if
> tomcat or httpd is the bottleneck.
>
> regards
> Leon
>
>>
>> Thanks,
>>
>>
>> -----Original Message-----
>> From: R Batchelor [mailto:rsbatch8@yahoo.com]
>> Sent: Friday, September 02, 2011 10:10 AM
>> To: Tomcat Users List
>> Subject: Re: Tomcat Performance Turning.
>>
>> Have you ruled out issues with db connection pooling? NO. IN FACT, THERE ARE KNOWN DBCP CONNECTION POOLING DEADLOCK ISSUES IN TOMCAT 6.0.26 [https://issues.apache.org/jira/browse/DBCP-270]. I AM LOBBYING TO UPGRADE TO TOMCAT 7.0.20 AND USE THE JDBC CONNECTION POOL INSTEAD OF DBCP CONNECTION POOL.
>>
>> You might consider setting your dbcp maxWait to 8000 or less and watch
>> for "timeout waiting for idle object" exceptions. Also, you could
>> monitor database connections/active-users on the DB side to see if your
>> dbcp pools are max'd but all the database users/connections are idle.
>> These can be symptoms of an application mis-managing its dbcp pools.
>> I WILL DEFINITELY TRY THIS OUT. THANKS!
>>
>> ________________________________
>> From: Leon Rosenberg <ro...@gmail.com>
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Sent: Friday, September 2, 2011 7:55 AM
>> Subject: Re: Tomcat Performance Turning.
>>
>> Hello Talha,
>>
>> well... the quickshots are through.
>> You should really create some threaddumps after each other (for
>> example with jstack) and try to find out which thread is slowing the
>> app down.
>> For starters you could try with code you changed for tomcat
>> adaptations if any. The problem seems to lie beneath the surface, so
>> you will have to start
>> performance monitoring and look into the inside.
>> However, time difference of 17 seconds must be something very very
>> obvious like a synchronized block in a valve/filter, db lock (have you
>> checked the db locks?), or an if (tomcat)
>> Thread.sleep(1000L*60*17);....
>>
>> regards
>> Leon
>>
>> On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tf...@credera.com> wrote:
>>> Hi Leon,
>>>
>>> Please see any answers in CAPS below.
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>>> Sent: Friday, September 02, 2011 9:06 AM
>>> To: Tomcat Users List
>>> Subject: Re: Tomcat Performance Turning.
>>>
>>> Hello Talha,
>>>
>>> seems that your tomcats are fine.
>>> Question, you mention that the tomcat do not perform, but you don't
>>> tell us how you come to this conclusion except for cpu load. ANSWER: LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
>>> What is the difference in response times between weblogic and tomcat? ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING 17000 ms ON AVERAGE.
>>> Maybe your tomcat just perform the job MUCH faster ;-)
>>> The thread dump you posted in another reply indicates that your
>>> tomcats are idling.
>>> Do you have manager installed? NO. With manager you could monitor active
>>> connections and requests.
>>> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS ARE IDLING IS WHAT WE OBSERVE.
>>> Finally, if you really want to profile your application, do you have
>>> moskito installed? :-) (the one i initially wrote
>>> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE WILL GIVE IT A TRY [TIME PERMITTING].
>>>
>>> In the last case I can offer you to provide support getting it running. WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
>>>
>>> regards
>>> Leon
>>>
>>> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
>>>> Plz. see my answers below in UPPERCAPS.
>>>>
>>>> Thanks!
>>>>
>>>> -----Original Message-----
>>>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>>>> Sent: Friday, September 02, 2011 8:35 AM
>>>> To: Tomcat Users List
>>>> Subject: Re: Tomcat Performance Turning.
>>>>
>>>> Hello Talha,
>>>>
>>>> from a quick glance at your post, do you have the same 30 max threads
>>>> limit in weblogic? YES.
>>>> Because sending 525 users through 2x30=60 max threads seems a little
>>>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>>>>
>>>> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>>>>
>>>> regards
>>>> Leon
>>>>
>>>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>>>>> Background:
>>>>>
>>>>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>>>>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>>>>
>>>>> The Issue - Load Testing:
>>>>>
>>>>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>>>>
>>>>> Setup for Load Testing:
>>>>>
>>>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>>>>
>>>>> Mentioned below is the version information:
>>>>> Apache Version 2.2.14 (with mod_jk module)
>>>>> Tomcat: 6.0.29
>>>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>>>>> Connection Pool: DBCP.
>>>>>
>>>>> Mentioned below are connector settings in conf/server.xml:
>>>>>
>>>>> <Connector
>>>>> address="stagingTCserver01"
>>>>> backlog="300"
>>>>> connectionTimeout="60000"
>>>>> enableLookups="false"
>>>>> maxPostSize="2097152"
>>>>> maxSpareThreads="10"
>>>>> maxThreads="30"
>>>>> minSpareThreads="5"
>>>>> port="8006"
>>>>> protocol="AJP/1.3"
>>>>> tcpNoDelay="true"
>>>>> />
>>>>>
>>>>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>>>>
>>>>> <Resource
>>>>> name="jdbc/onlinedb"
>>>>> auth="Container"
>>>>> type="javax.sql.DataSource"
>>>>> driverClassName="oracle.jdbc.OracleDriver"
>>>>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>>>>> initialSize="1"
>>>>> maxActive="30"
>>>>> minIdle="1"
>>>>> maxIdle="5"
>>>>> maxWait="300000"
>>>>> poolPreparedStatements="true"
>>>>> maxOpenPreparedStatements="300"
>>>>> validationQuery="SELECT 1 FROM BB_DUAL"
>>>>> testOnBorrow="true"
>>>>> validationInterval="10000"
>>>>> testWhileIdle="true"
>>>>> />
>>>>>
>>>>> JVM Parameters:
>>>>>
>>>>> -Xms512m -Xmx2048m
>>>>> -XX:PermSize=128m -XX:MaxPermSize=256m
>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>>>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>>>>
>>>>> It must be noted that Weblogic setup performs very well using similar settings.
>>>>>
>>>>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>>>>
>>>>> Any tips/pointers will be greatly appreciated.
>>>>>
>>>>> Talha.
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by Talha Fazal <tf...@credera.com>.
Q. Have you tried running directly against tomcat without apache inbetween?
A. Yes; this resolves the performance bottleneck.
Q. <holywar>from my experience there is no need for apache in your setup anyway </holywar>.
A. :-) The reasons for having apache in front: A) Create a DMZ and protect Tomcat from being exposed to outside attacks; B) Load balancing.
Thanks Leon!
-----Original Message-----
From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
Sent: Friday, September 02, 2011 3:19 PM
To: Tomcat Users List
Subject: Re: Tomcat Performance Turning.
Hello Talha,
On Fri, Sep 2, 2011 at 6:16 PM, Talha Fazal <tf...@credera.com> wrote:
> PLEASE SEE MY ANSWERS BELOW IN UPPER CAPS.
>
> ONE IMPORTANT OBSERVATION: Even though, tomcat's response time is decent, for some reason, apache is taking a lot longer to serve requests. We are using mod_jk with Tomcat using AJP1.3 protocol in the connector setting in server.xml. We plan to try mod_proxy_ajp and mod_proxy instead of mod_jk. Any thoughts here?
Interesting observation. Have you tried running directly against
tomcat without apache inbetween?
<holywar>from my experience there is no need for apache in your setup
anyway </holywar>.
In the past we used both mod_jk and mod_proxy_ajp, and even
mod_proxy_ajp is somewhat faster, the difference where never 17
seconds.
Your best guess would probably be to test without httpd to rule out if
tomcat or httpd is the bottleneck.
regards
Leon
>
> Thanks,
>
>
> -----Original Message-----
> From: R Batchelor [mailto:rsbatch8@yahoo.com]
> Sent: Friday, September 02, 2011 10:10 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Have you ruled out issues with db connection pooling? NO. IN FACT, THERE ARE KNOWN DBCP CONNECTION POOLING DEADLOCK ISSUES IN TOMCAT 6.0.26 [https://issues.apache.org/jira/browse/DBCP-270]. I AM LOBBYING TO UPGRADE TO TOMCAT 7.0.20 AND USE THE JDBC CONNECTION POOL INSTEAD OF DBCP CONNECTION POOL.
>
> You might consider setting your dbcp maxWait to 8000 or less and watch
> for "timeout waiting for idle object" exceptions. Also, you could
> monitor database connections/active-users on the DB side to see if your
> dbcp pools are max'd but all the database users/connections are idle.
> These can be symptoms of an application mis-managing its dbcp pools.
> I WILL DEFINITELY TRY THIS OUT. THANKS!
>
> ________________________________
> From: Leon Rosenberg <ro...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Friday, September 2, 2011 7:55 AM
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> well... the quickshots are through.
> You should really create some threaddumps after each other (for
> example with jstack) and try to find out which thread is slowing the
> app down.
> For starters you could try with code you changed for tomcat
> adaptations if any. The problem seems to lie beneath the surface, so
> you will have to start
> performance monitoring and look into the inside.
> However, time difference of 17 seconds must be something very very
> obvious like a synchronized block in a valve/filter, db lock (have you
> checked the db locks?), or an if (tomcat)
> Thread.sleep(1000L*60*17);....
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tf...@credera.com> wrote:
>> Hi Leon,
>>
>> Please see any answers in CAPS below.
>>
>>
>>
>> -----Original Message-----
>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>> Sent: Friday, September 02, 2011 9:06 AM
>> To: Tomcat Users List
>> Subject: Re: Tomcat Performance Turning.
>>
>> Hello Talha,
>>
>> seems that your tomcats are fine.
>> Question, you mention that the tomcat do not perform, but you don't
>> tell us how you come to this conclusion except for cpu load. ANSWER: LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
>> What is the difference in response times between weblogic and tomcat? ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING 17000 ms ON AVERAGE.
>> Maybe your tomcat just perform the job MUCH faster ;-)
>> The thread dump you posted in another reply indicates that your
>> tomcats are idling.
>> Do you have manager installed? NO. With manager you could monitor active
>> connections and requests.
>> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS ARE IDLING IS WHAT WE OBSERVE.
>> Finally, if you really want to profile your application, do you have
>> moskito installed? :-) (the one i initially wrote
>> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE WILL GIVE IT A TRY [TIME PERMITTING].
>>
>> In the last case I can offer you to provide support getting it running. WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
>>
>> regards
>> Leon
>>
>> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
>>> Plz. see my answers below in UPPERCAPS.
>>>
>>> Thanks!
>>>
>>> -----Original Message-----
>>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>>> Sent: Friday, September 02, 2011 8:35 AM
>>> To: Tomcat Users List
>>> Subject: Re: Tomcat Performance Turning.
>>>
>>> Hello Talha,
>>>
>>> from a quick glance at your post, do you have the same 30 max threads
>>> limit in weblogic? YES.
>>> Because sending 525 users through 2x30=60 max threads seems a little
>>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>>>
>>> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>>>
>>> regards
>>> Leon
>>>
>>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>>>> Background:
>>>>
>>>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>>>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>>>
>>>> The Issue - Load Testing:
>>>>
>>>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>>>
>>>> Setup for Load Testing:
>>>>
>>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>>>
>>>> Mentioned below is the version information:
>>>> Apache Version 2.2.14 (with mod_jk module)
>>>> Tomcat: 6.0.29
>>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>>>> Connection Pool: DBCP.
>>>>
>>>> Mentioned below are connector settings in conf/server.xml:
>>>>
>>>> <Connector
>>>> address="stagingTCserver01"
>>>> backlog="300"
>>>> connectionTimeout="60000"
>>>> enableLookups="false"
>>>> maxPostSize="2097152"
>>>> maxSpareThreads="10"
>>>> maxThreads="30"
>>>> minSpareThreads="5"
>>>> port="8006"
>>>> protocol="AJP/1.3"
>>>> tcpNoDelay="true"
>>>> />
>>>>
>>>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>>>
>>>> <Resource
>>>> name="jdbc/onlinedb"
>>>> auth="Container"
>>>> type="javax.sql.DataSource"
>>>> driverClassName="oracle.jdbc.OracleDriver"
>>>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>>>> initialSize="1"
>>>> maxActive="30"
>>>> minIdle="1"
>>>> maxIdle="5"
>>>> maxWait="300000"
>>>> poolPreparedStatements="true"
>>>> maxOpenPreparedStatements="300"
>>>> validationQuery="SELECT 1 FROM BB_DUAL"
>>>> testOnBorrow="true"
>>>> validationInterval="10000"
>>>> testWhileIdle="true"
>>>> />
>>>>
>>>> JVM Parameters:
>>>>
>>>> -Xms512m -Xmx2048m
>>>> -XX:PermSize=128m -XX:MaxPermSize=256m
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>>>
>>>> It must be noted that Weblogic setup performs very well using similar settings.
>>>>
>>>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>>>
>>>> Any tips/pointers will be greatly appreciated.
>>>>
>>>> Talha.
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Performance Turning.
Posted by Leon Rosenberg <ro...@gmail.com>.
Hello Christopher,
On Fri, Sep 2, 2011 at 11:52 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Leon,
>
> On 9/2/2011 4:19 PM, Leon Rosenberg wrote:
>> <holywar>from my experience there is no need for apache in your
>> setup anyway </holywar>.
>
> Uh, load-balancing?
Aeh... loadbalancer? There are both, soft and hardware solutions.
regards
Leon
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk5hUDgACgkQ9CaO5/Lv0PBfhQCdFwEDvm37L9QTe6FBns88RKLg
> 7ukAnikbIxI+y/oaskk/Zv/Y4aKPpYD9
> =Dc+M
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Performance Turning.
Posted by Igor Cicimov <ic...@gmail.com>.
Paste the workers.properties file and the mod_jk settings here. Are you
using prefork or mpm in Apache? Have you tuned the apache settings to match
the tomcat number of threads?
Igor
On Sat, Sep 3, 2011 at 7:52 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Leon,
>
> On 9/2/2011 4:19 PM, Leon Rosenberg wrote:
> > <holywar>from my experience there is no need for apache in your
> > setup anyway </holywar>.
>
> Uh, load-balancing?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk5hUDgACgkQ9CaO5/Lv0PBfhQCdFwEDvm37L9QTe6FBns88RKLg
> 7ukAnikbIxI+y/oaskk/Zv/Y4aKPpYD9
> =Dc+M
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat Performance Turning.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Leon,
On 9/2/2011 4:19 PM, Leon Rosenberg wrote:
> <holywar>from my experience there is no need for apache in your
> setup anyway </holywar>.
Uh, load-balancing?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk5hUDgACgkQ9CaO5/Lv0PBfhQCdFwEDvm37L9QTe6FBns88RKLg
7ukAnikbIxI+y/oaskk/Zv/Y4aKPpYD9
=Dc+M
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Performance Turning.
Posted by Leon Rosenberg <ro...@gmail.com>.
Hello Talha,
On Fri, Sep 2, 2011 at 6:16 PM, Talha Fazal <tf...@credera.com> wrote:
> PLEASE SEE MY ANSWERS BELOW IN UPPER CAPS.
>
> ONE IMPORTANT OBSERVATION: Even though, tomcat's response time is decent, for some reason, apache is taking a lot longer to serve requests. We are using mod_jk with Tomcat using AJP1.3 protocol in the connector setting in server.xml. We plan to try mod_proxy_ajp and mod_proxy instead of mod_jk. Any thoughts here?
Interesting observation. Have you tried running directly against
tomcat without apache inbetween?
<holywar>from my experience there is no need for apache in your setup
anyway </holywar>.
In the past we used both mod_jk and mod_proxy_ajp, and even
mod_proxy_ajp is somewhat faster, the difference where never 17
seconds.
Your best guess would probably be to test without httpd to rule out if
tomcat or httpd is the bottleneck.
regards
Leon
>
> Thanks,
>
>
> -----Original Message-----
> From: R Batchelor [mailto:rsbatch8@yahoo.com]
> Sent: Friday, September 02, 2011 10:10 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Have you ruled out issues with db connection pooling? NO. IN FACT, THERE ARE KNOWN DBCP CONNECTION POOLING DEADLOCK ISSUES IN TOMCAT 6.0.26 [https://issues.apache.org/jira/browse/DBCP-270]. I AM LOBBYING TO UPGRADE TO TOMCAT 7.0.20 AND USE THE JDBC CONNECTION POOL INSTEAD OF DBCP CONNECTION POOL.
>
> You might consider setting your dbcp maxWait to 8000 or less and watch
> for "timeout waiting for idle object" exceptions. Also, you could
> monitor database connections/active-users on the DB side to see if your
> dbcp pools are max'd but all the database users/connections are idle.
> These can be symptoms of an application mis-managing its dbcp pools.
> I WILL DEFINITELY TRY THIS OUT. THANKS!
>
> ________________________________
> From: Leon Rosenberg <ro...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Friday, September 2, 2011 7:55 AM
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> well... the quickshots are through.
> You should really create some threaddumps after each other (for
> example with jstack) and try to find out which thread is slowing the
> app down.
> For starters you could try with code you changed for tomcat
> adaptations if any. The problem seems to lie beneath the surface, so
> you will have to start
> performance monitoring and look into the inside.
> However, time difference of 17 seconds must be something very very
> obvious like a synchronized block in a valve/filter, db lock (have you
> checked the db locks?), or an if (tomcat)
> Thread.sleep(1000L*60*17);....
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tf...@credera.com> wrote:
>> Hi Leon,
>>
>> Please see any answers in CAPS below.
>>
>>
>>
>> -----Original Message-----
>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>> Sent: Friday, September 02, 2011 9:06 AM
>> To: Tomcat Users List
>> Subject: Re: Tomcat Performance Turning.
>>
>> Hello Talha,
>>
>> seems that your tomcats are fine.
>> Question, you mention that the tomcat do not perform, but you don't
>> tell us how you come to this conclusion except for cpu load. ANSWER: LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
>> What is the difference in response times between weblogic and tomcat? ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING 17000 ms ON AVERAGE.
>> Maybe your tomcat just perform the job MUCH faster ;-)
>> The thread dump you posted in another reply indicates that your
>> tomcats are idling.
>> Do you have manager installed? NO. With manager you could monitor active
>> connections and requests.
>> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS ARE IDLING IS WHAT WE OBSERVE.
>> Finally, if you really want to profile your application, do you have
>> moskito installed? :-) (the one i initially wrote
>> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE WILL GIVE IT A TRY [TIME PERMITTING].
>>
>> In the last case I can offer you to provide support getting it running. WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
>>
>> regards
>> Leon
>>
>> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
>>> Plz. see my answers below in UPPERCAPS.
>>>
>>> Thanks!
>>>
>>> -----Original Message-----
>>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>>> Sent: Friday, September 02, 2011 8:35 AM
>>> To: Tomcat Users List
>>> Subject: Re: Tomcat Performance Turning.
>>>
>>> Hello Talha,
>>>
>>> from a quick glance at your post, do you have the same 30 max threads
>>> limit in weblogic? YES.
>>> Because sending 525 users through 2x30=60 max threads seems a little
>>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>>>
>>> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>>>
>>> regards
>>> Leon
>>>
>>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>>>> Background:
>>>>
>>>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>>>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>>>
>>>> The Issue - Load Testing:
>>>>
>>>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>>>
>>>> Setup for Load Testing:
>>>>
>>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>>>
>>>> Mentioned below is the version information:
>>>> Apache Version 2.2.14 (with mod_jk module)
>>>> Tomcat: 6.0.29
>>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>>>> Connection Pool: DBCP.
>>>>
>>>> Mentioned below are connector settings in conf/server.xml:
>>>>
>>>> <Connector
>>>> address="stagingTCserver01"
>>>> backlog="300"
>>>> connectionTimeout="60000"
>>>> enableLookups="false"
>>>> maxPostSize="2097152"
>>>> maxSpareThreads="10"
>>>> maxThreads="30"
>>>> minSpareThreads="5"
>>>> port="8006"
>>>> protocol="AJP/1.3"
>>>> tcpNoDelay="true"
>>>> />
>>>>
>>>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>>>
>>>> <Resource
>>>> name="jdbc/onlinedb"
>>>> auth="Container"
>>>> type="javax.sql.DataSource"
>>>> driverClassName="oracle.jdbc.OracleDriver"
>>>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>>>> initialSize="1"
>>>> maxActive="30"
>>>> minIdle="1"
>>>> maxIdle="5"
>>>> maxWait="300000"
>>>> poolPreparedStatements="true"
>>>> maxOpenPreparedStatements="300"
>>>> validationQuery="SELECT 1 FROM BB_DUAL"
>>>> testOnBorrow="true"
>>>> validationInterval="10000"
>>>> testWhileIdle="true"
>>>> />
>>>>
>>>> JVM Parameters:
>>>>
>>>> -Xms512m -Xmx2048m
>>>> -XX:PermSize=128m -XX:MaxPermSize=256m
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>>>
>>>> It must be noted that Weblogic setup performs very well using similar settings.
>>>>
>>>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>>>
>>>> Any tips/pointers will be greatly appreciated.
>>>>
>>>> Talha.
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by Talha Fazal <tf...@credera.com>.
PLEASE SEE MY ANSWERS BELOW IN UPPER CAPS.
ONE IMPORTANT OBSERVATION: Even though, tomcat's response time is decent, for some reason, apache is taking a lot longer to serve requests. We are using mod_jk with Tomcat using AJP1.3 protocol in the connector setting in server.xml. We plan to try mod_proxy_ajp and mod_proxy instead of mod_jk. Any thoughts here?
Thanks,
-----Original Message-----
From: R Batchelor [mailto:rsbatch8@yahoo.com]
Sent: Friday, September 02, 2011 10:10 AM
To: Tomcat Users List
Subject: Re: Tomcat Performance Turning.
Have you ruled out issues with db connection pooling? NO. IN FACT, THERE ARE KNOWN DBCP CONNECTION POOLING DEADLOCK ISSUES IN TOMCAT 6.0.26 [https://issues.apache.org/jira/browse/DBCP-270]. I AM LOBBYING TO UPGRADE TO TOMCAT 7.0.20 AND USE THE JDBC CONNECTION POOL INSTEAD OF DBCP CONNECTION POOL.
You might consider setting your dbcp maxWait to 8000 or less and watch
for "timeout waiting for idle object" exceptions. Also, you could
monitor database connections/active-users on the DB side to see if your
dbcp pools are max'd but all the database users/connections are idle.
These can be symptoms of an application mis-managing its dbcp pools.
I WILL DEFINITELY TRY THIS OUT. THANKS!
________________________________
From: Leon Rosenberg <ro...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Friday, September 2, 2011 7:55 AM
Subject: Re: Tomcat Performance Turning.
Hello Talha,
well... the quickshots are through.
You should really create some threaddumps after each other (for
example with jstack) and try to find out which thread is slowing the
app down.
For starters you could try with code you changed for tomcat
adaptations if any. The problem seems to lie beneath the surface, so
you will have to start
performance monitoring and look into the inside.
However, time difference of 17 seconds must be something very very
obvious like a synchronized block in a valve/filter, db lock (have you
checked the db locks?), or an if (tomcat)
Thread.sleep(1000L*60*17);....
regards
Leon
On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tf...@credera.com> wrote:
> Hi Leon,
>
> Please see any answers in CAPS below.
>
>
>
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
> Sent: Friday, September 02, 2011 9:06 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> seems that your tomcats are fine.
> Question, you mention that the tomcat do not perform, but you don't
> tell us how you come to this conclusion except for cpu load. ANSWER: LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
> What is the difference in response times between weblogic and tomcat? ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING 17000 ms ON AVERAGE.
> Maybe your tomcat just perform the job MUCH faster ;-)
> The thread dump you posted in another reply indicates that your
> tomcats are idling.
> Do you have manager installed? NO. With manager you could monitor active
> connections and requests.
> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS ARE IDLING IS WHAT WE OBSERVE.
> Finally, if you really want to profile your application, do you have
> moskito installed? :-) (the one i initially wrote
> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE WILL GIVE IT A TRY [TIME PERMITTING].
>
> In the last case I can offer you to provide support getting it running. WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
>> Plz. see my answers below in UPPERCAPS.
>>
>> Thanks!
>>
>> -----Original Message-----
>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>> Sent: Friday, September 02, 2011 8:35 AM
>> To: Tomcat Users List
>> Subject: Re: Tomcat Performance Turning.
>>
>> Hello Talha,
>>
>> from a quick glance at your post, do you have the same 30 max threads
>> limit in weblogic? YES.
>> Because sending 525 users through 2x30=60 max threads seems a little
>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>>
>> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>>
>> regards
>> Leon
>>
>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>>> Background:
>>>
>>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>>
>>> The Issue - Load Testing:
>>>
>>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>>
>>> Setup for Load Testing:
>>>
>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>>
>>> Mentioned below is the version information:
>>> Apache Version 2.2.14 (with mod_jk module)
>>> Tomcat: 6.0.29
>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>>> Connection Pool: DBCP.
>>>
>>> Mentioned below are connector settings in conf/server.xml:
>>>
>>> <Connector
>>> address="stagingTCserver01"
>>> backlog="300"
>>> connectionTimeout="60000"
>>> enableLookups="false"
>>> maxPostSize="2097152"
>>> maxSpareThreads="10"
>>> maxThreads="30"
>>> minSpareThreads="5"
>>> port="8006"
>>> protocol="AJP/1.3"
>>> tcpNoDelay="true"
>>> />
>>>
>>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>>
>>> <Resource
>>> name="jdbc/onlinedb"
>>> auth="Container"
>>> type="javax.sql.DataSource"
>>> driverClassName="oracle.jdbc.OracleDriver"
>>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>>> initialSize="1"
>>> maxActive="30"
>>> minIdle="1"
>>> maxIdle="5"
>>> maxWait="300000"
>>> poolPreparedStatements="true"
>>> maxOpenPreparedStatements="300"
>>> validationQuery="SELECT 1 FROM BB_DUAL"
>>> testOnBorrow="true"
>>> validationInterval="10000"
>>> testWhileIdle="true"
>>> />
>>>
>>> JVM Parameters:
>>>
>>> -Xms512m -Xmx2048m
>>> -XX:PermSize=128m -XX:MaxPermSize=256m
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>>
>>> It must be noted that Weblogic setup performs very well using similar settings.
>>>
>>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>>
>>> Any tips/pointers will be greatly appreciated.
>>>
>>> Talha.
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Performance Turning.
Posted by R Batchelor <rs...@yahoo.com>.
Have you ruled out issues with db connection pooling?
You might consider setting your dbcp maxWait to 8000 or less and watch
for "timeout waiting for idle object" exceptions. Also, you could
monitor database connections/active-users on the DB side to see if your
dbcp pools are max'd but all the database users/connections are idle.
These can be symptoms of an application mis-managing its dbcp pools.
________________________________
From: Leon Rosenberg <ro...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Friday, September 2, 2011 7:55 AM
Subject: Re: Tomcat Performance Turning.
Hello Talha,
well... the quickshots are through.
You should really create some threaddumps after each other (for
example with jstack) and try to find out which thread is slowing the
app down.
For starters you could try with code you changed for tomcat
adaptations if any. The problem seems to lie beneath the surface, so
you will have to start
performance monitoring and look into the inside.
However, time difference of 17 seconds must be something very very
obvious like a synchronized block in a valve/filter, db lock (have you
checked the db locks?), or an if (tomcat)
Thread.sleep(1000L*60*17);....
regards
Leon
On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tf...@credera.com> wrote:
> Hi Leon,
>
> Please see any answers in CAPS below.
>
>
>
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
> Sent: Friday, September 02, 2011 9:06 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> seems that your tomcats are fine.
> Question, you mention that the tomcat do not perform, but you don't
> tell us how you come to this conclusion except for cpu load. ANSWER: LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
> What is the difference in response times between weblogic and tomcat? ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING 17000 ms ON AVERAGE.
> Maybe your tomcat just perform the job MUCH faster ;-)
> The thread dump you posted in another reply indicates that your
> tomcats are idling.
> Do you have manager installed? NO. With manager you could monitor active
> connections and requests.
> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS ARE IDLING IS WHAT WE OBSERVE.
> Finally, if you really want to profile your application, do you have
> moskito installed? :-) (the one i initially wrote
> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE WILL GIVE IT A TRY [TIME PERMITTING].
>
> In the last case I can offer you to provide support getting it running. WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
>> Plz. see my answers below in UPPERCAPS.
>>
>> Thanks!
>>
>> -----Original Message-----
>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>> Sent: Friday, September 02, 2011 8:35 AM
>> To: Tomcat Users List
>> Subject: Re: Tomcat Performance Turning.
>>
>> Hello Talha,
>>
>> from a quick glance at your post, do you have the same 30 max threads
>> limit in weblogic? YES.
>> Because sending 525 users through 2x30=60 max threads seems a little
>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>>
>> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>>
>> regards
>> Leon
>>
>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>>> Background:
>>>
>>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>>
>>> The Issue - Load Testing:
>>>
>>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>>
>>> Setup for Load Testing:
>>>
>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>>
>>> Mentioned below is the version information:
>>> Apache Version 2.2.14 (with mod_jk module)
>>> Tomcat: 6.0.29
>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>>> Connection Pool: DBCP.
>>>
>>> Mentioned below are connector settings in conf/server.xml:
>>>
>>> <Connector
>>> address="stagingTCserver01"
>>> backlog="300"
>>> connectionTimeout="60000"
>>> enableLookups="false"
>>> maxPostSize="2097152"
>>> maxSpareThreads="10"
>>> maxThreads="30"
>>> minSpareThreads="5"
>>> port="8006"
>>> protocol="AJP/1.3"
>>> tcpNoDelay="true"
>>> />
>>>
>>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>>
>>> <Resource
>>> name="jdbc/onlinedb"
>>> auth="Container"
>>> type="javax.sql.DataSource"
>>> driverClassName="oracle.jdbc.OracleDriver"
>>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>>> initialSize="1"
>>> maxActive="30"
>>> minIdle="1"
>>> maxIdle="5"
>>> maxWait="300000"
>>> poolPreparedStatements="true"
>>> maxOpenPreparedStatements="300"
>>> validationQuery="SELECT 1 FROM BB_DUAL"
>>> testOnBorrow="true"
>>> validationInterval="10000"
>>> testWhileIdle="true"
>>> />
>>>
>>> JVM Parameters:
>>>
>>> -Xms512m -Xmx2048m
>>> -XX:PermSize=128m -XX:MaxPermSize=256m
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>>
>>> It must be noted that Weblogic setup performs very well using similar settings.
>>>
>>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>>
>>> Any tips/pointers will be greatly appreciated.
>>>
>>> Talha.
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Performance Turning.
Posted by Leon Rosenberg <ro...@gmail.com>.
Hello Talha,
well... the quickshots are through.
You should really create some threaddumps after each other (for
example with jstack) and try to find out which thread is slowing the
app down.
For starters you could try with code you changed for tomcat
adaptations if any. The problem seems to lie beneath the surface, so
you will have to start
performance monitoring and look into the inside.
However, time difference of 17 seconds must be something very very
obvious like a synchronized block in a valve/filter, db lock (have you
checked the db locks?), or an if (tomcat)
Thread.sleep(1000L*60*17);....
regards
Leon
On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tf...@credera.com> wrote:
> Hi Leon,
>
> Please see any answers in CAPS below.
>
>
>
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
> Sent: Friday, September 02, 2011 9:06 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> seems that your tomcats are fine.
> Question, you mention that the tomcat do not perform, but you don't
> tell us how you come to this conclusion except for cpu load. ANSWER: LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
> What is the difference in response times between weblogic and tomcat? ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING 17000 ms ON AVERAGE.
> Maybe your tomcat just perform the job MUCH faster ;-)
> The thread dump you posted in another reply indicates that your
> tomcats are idling.
> Do you have manager installed? NO. With manager you could monitor active
> connections and requests.
> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS ARE IDLING IS WHAT WE OBSERVE.
> Finally, if you really want to profile your application, do you have
> moskito installed? :-) (the one i initially wrote
> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE WILL GIVE IT A TRY [TIME PERMITTING].
>
> In the last case I can offer you to provide support getting it running. WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
>> Plz. see my answers below in UPPERCAPS.
>>
>> Thanks!
>>
>> -----Original Message-----
>> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
>> Sent: Friday, September 02, 2011 8:35 AM
>> To: Tomcat Users List
>> Subject: Re: Tomcat Performance Turning.
>>
>> Hello Talha,
>>
>> from a quick glance at your post, do you have the same 30 max threads
>> limit in weblogic? YES.
>> Because sending 525 users through 2x30=60 max threads seems a little
>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>>
>> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>>
>> regards
>> Leon
>>
>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>>> Background:
>>>
>>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>>
>>> The Issue - Load Testing:
>>>
>>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>>
>>> Setup for Load Testing:
>>>
>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>>
>>> Mentioned below is the version information:
>>> Apache Version 2.2.14 (with mod_jk module)
>>> Tomcat: 6.0.29
>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>>> Connection Pool: DBCP.
>>>
>>> Mentioned below are connector settings in conf/server.xml:
>>>
>>> <Connector
>>> address="stagingTCserver01"
>>> backlog="300"
>>> connectionTimeout="60000"
>>> enableLookups="false"
>>> maxPostSize="2097152"
>>> maxSpareThreads="10"
>>> maxThreads="30"
>>> minSpareThreads="5"
>>> port="8006"
>>> protocol="AJP/1.3"
>>> tcpNoDelay="true"
>>> />
>>>
>>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>>
>>> <Resource
>>> name="jdbc/onlinedb"
>>> auth="Container"
>>> type="javax.sql.DataSource"
>>> driverClassName="oracle.jdbc.OracleDriver"
>>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>>> initialSize="1"
>>> maxActive="30"
>>> minIdle="1"
>>> maxIdle="5"
>>> maxWait="300000"
>>> poolPreparedStatements="true"
>>> maxOpenPreparedStatements="300"
>>> validationQuery="SELECT 1 FROM BB_DUAL"
>>> testOnBorrow="true"
>>> validationInterval="10000"
>>> testWhileIdle="true"
>>> />
>>>
>>> JVM Parameters:
>>>
>>> -Xms512m -Xmx2048m
>>> -XX:PermSize=128m -XX:MaxPermSize=256m
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>>
>>> It must be noted that Weblogic setup performs very well using similar settings.
>>>
>>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>>
>>> Any tips/pointers will be greatly appreciated.
>>>
>>> Talha.
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by Talha Fazal <tf...@credera.com>.
Hi Leon,
Please see any answers in CAPS below.
-----Original Message-----
From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
Sent: Friday, September 02, 2011 9:06 AM
To: Tomcat Users List
Subject: Re: Tomcat Performance Turning.
Hello Talha,
seems that your tomcats are fine.
Question, you mention that the tomcat do not perform, but you don't
tell us how you come to this conclusion except for cpu load. ANSWER: LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE).
What is the difference in response times between weblogic and tomcat? ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS TAKING 17000 ms ON AVERAGE.
Maybe your tomcat just perform the job MUCH faster ;-)
The thread dump you posted in another reply indicates that your
tomcats are idling.
Do you have manager installed? NO. With manager you could monitor active
connections and requests.
Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS ARE IDLING IS WHAT WE OBSERVE.
Finally, if you really want to profile your application, do you have
moskito installed? :-) (the one i initially wrote
-moskito.anotheria.net, therefore the shameless advertisement). :-) SURE WILL GIVE IT A TRY [TIME PERMITTING].
In the last case I can offer you to provide support getting it running. WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS.
regards
Leon
On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
> Plz. see my answers below in UPPERCAPS.
>
> Thanks!
>
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
> Sent: Friday, September 02, 2011 8:35 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> from a quick glance at your post, do you have the same 30 max threads
> limit in weblogic? YES.
> Because sending 525 users through 2x30=60 max threads seems a little
> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>
> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>> Background:
>>
>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>
>> The Issue - Load Testing:
>>
>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>
>> Setup for Load Testing:
>>
>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>
>> Mentioned below is the version information:
>> Apache Version 2.2.14 (with mod_jk module)
>> Tomcat: 6.0.29
>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>> Connection Pool: DBCP.
>>
>> Mentioned below are connector settings in conf/server.xml:
>>
>> <Connector
>> address="stagingTCserver01"
>> backlog="300"
>> connectionTimeout="60000"
>> enableLookups="false"
>> maxPostSize="2097152"
>> maxSpareThreads="10"
>> maxThreads="30"
>> minSpareThreads="5"
>> port="8006"
>> protocol="AJP/1.3"
>> tcpNoDelay="true"
>> />
>>
>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>
>> <Resource
>> name="jdbc/onlinedb"
>> auth="Container"
>> type="javax.sql.DataSource"
>> driverClassName="oracle.jdbc.OracleDriver"
>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>> initialSize="1"
>> maxActive="30"
>> minIdle="1"
>> maxIdle="5"
>> maxWait="300000"
>> poolPreparedStatements="true"
>> maxOpenPreparedStatements="300"
>> validationQuery="SELECT 1 FROM BB_DUAL"
>> testOnBorrow="true"
>> validationInterval="10000"
>> testWhileIdle="true"
>> />
>>
>> JVM Parameters:
>>
>> -Xms512m -Xmx2048m
>> -XX:PermSize=128m -XX:MaxPermSize=256m
>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>
>> It must be noted that Weblogic setup performs very well using similar settings.
>>
>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>
>> Any tips/pointers will be greatly appreciated.
>>
>> Talha.
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Performance Turning.
Posted by Leon Rosenberg <ro...@gmail.com>.
Hello Talha,
seems that your tomcats are fine.
Question, you mention that the tomcat do not perform, but you don't
tell us how you come to this conclusion except for cpu load.
What is the difference in response times between weblogic and tomcat?
Maybe your tomcat just perform the job MUCH faster ;-)
The thread dump you posted in another reply indicates that your
tomcats are idling.
Do you have manager installed? With manager you could monitor active
connections and requests.
Do you have probe installed?
Finally, if you really want to profile your application, do you have
moskito installed? :-) (the one i initially wrote
-moskito.anotheria.net, therefore the shameless advertisement).
In the last case I can offer you to provide support getting it running.
regards
Leon
On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tf...@credera.com> wrote:
> Plz. see my answers below in UPPERCAPS.
>
> Thanks!
>
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
> Sent: Friday, September 02, 2011 8:35 AM
> To: Tomcat Users List
> Subject: Re: Tomcat Performance Turning.
>
> Hello Talha,
>
> from a quick glance at your post, do you have the same 30 max threads
> limit in weblogic? YES.
> Because sending 525 users through 2x30=60 max threads seems a little
> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
>
> Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
>
> regards
> Leon
>
> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
>> Background:
>>
>> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
>> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>>
>> The Issue - Load Testing:
>>
>> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>>
>> Setup for Load Testing:
>>
>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>>
>> Mentioned below is the version information:
>> Apache Version 2.2.14 (with mod_jk module)
>> Tomcat: 6.0.29
>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
>> Connection Pool: DBCP.
>>
>> Mentioned below are connector settings in conf/server.xml:
>>
>> <Connector
>> address="stagingTCserver01"
>> backlog="300"
>> connectionTimeout="60000"
>> enableLookups="false"
>> maxPostSize="2097152"
>> maxSpareThreads="10"
>> maxThreads="30"
>> minSpareThreads="5"
>> port="8006"
>> protocol="AJP/1.3"
>> tcpNoDelay="true"
>> />
>>
>> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>>
>> <Resource
>> name="jdbc/onlinedb"
>> auth="Container"
>> type="javax.sql.DataSource"
>> driverClassName="oracle.jdbc.OracleDriver"
>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
>> initialSize="1"
>> maxActive="30"
>> minIdle="1"
>> maxIdle="5"
>> maxWait="300000"
>> poolPreparedStatements="true"
>> maxOpenPreparedStatements="300"
>> validationQuery="SELECT 1 FROM BB_DUAL"
>> testOnBorrow="true"
>> validationInterval="10000"
>> testWhileIdle="true"
>> />
>>
>> JVM Parameters:
>>
>> -Xms512m -Xmx2048m
>> -XX:PermSize=128m -XX:MaxPermSize=256m
>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>>
>> It must be noted that Weblogic setup performs very well using similar settings.
>>
>> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>>
>> Any tips/pointers will be greatly appreciated.
>>
>> Talha.
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by Talha Fazal <tf...@credera.com>.
Plz. see my answers below in UPPERCAPS.
Thanks!
-----Original Message-----
From: Leon Rosenberg [mailto:rosenberg.leon@gmail.com]
Sent: Friday, September 02, 2011 8:35 AM
To: Tomcat Users List
Subject: Re: Tomcat Performance Turning.
Hello Talha,
from a quick glance at your post, do you have the same 30 max threads
limit in weblogic? YES.
Because sending 525 users through 2x30=60 max threads seems a little
bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU USAGE INCREASED FROM 7% to 10%.
Which software are you using to produce the load? LOADRUNNER. Does it keepalive the connections? YES.
regards
Leon
On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
> Background:
>
> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>
> The Issue - Load Testing:
>
> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>
> Setup for Load Testing:
>
> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>
> Mentioned below is the version information:
> Apache Version 2.2.14 (with mod_jk module)
> Tomcat: 6.0.29
> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
> Connection Pool: DBCP.
>
> Mentioned below are connector settings in conf/server.xml:
>
> <Connector
> address="stagingTCserver01"
> backlog="300"
> connectionTimeout="60000"
> enableLookups="false"
> maxPostSize="2097152"
> maxSpareThreads="10"
> maxThreads="30"
> minSpareThreads="5"
> port="8006"
> protocol="AJP/1.3"
> tcpNoDelay="true"
> />
>
> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>
> <Resource
> name="jdbc/onlinedb"
> auth="Container"
> type="javax.sql.DataSource"
> driverClassName="oracle.jdbc.OracleDriver"
> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
> initialSize="1"
> maxActive="30"
> minIdle="1"
> maxIdle="5"
> maxWait="300000"
> poolPreparedStatements="true"
> maxOpenPreparedStatements="300"
> validationQuery="SELECT 1 FROM BB_DUAL"
> testOnBorrow="true"
> validationInterval="10000"
> testWhileIdle="true"
> />
>
> JVM Parameters:
>
> -Xms512m -Xmx2048m
> -XX:PermSize=128m -XX:MaxPermSize=256m
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>
> It must be noted that Weblogic setup performs very well using similar settings.
>
> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>
> Any tips/pointers will be greatly appreciated.
>
> Talha.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Performance Turning.
Posted by Leon Rosenberg <ro...@gmail.com>.
Hello Talha,
from a quick glance at your post, do you have the same 30 max threads
limit in weblogic?
Because sending 525 users through 2x30=60 max threads seems a little
bottlenecky.
Which software are you using to produce the load? Does it keepalive
the connections?
regards
Leon
On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tf...@credera.com> wrote:
> Background:
>
> We have a moderately high traffic web application (between 8 to 21 million hits/day) running Apache to serve static content (also to load balance and create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web application).
> We are trying to replace Weblogic with Tomcat and we have ported our code to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING environment as long as there is no real load.
>
> The Issue - Load Testing:
>
> In our staging environment for load testing, when we run the load test using 525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is also between 4 and 5%.
>
> Setup for Load Testing:
>
> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
>
> Mentioned below is the version information:
> Apache Version 2.2.14 (with mod_jk module)
> Tomcat: 6.0.29
> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
> Connection Pool: DBCP.
>
> Mentioned below are connector settings in conf/server.xml:
>
> <Connector
> address="stagingTCserver01"
> backlog="300"
> connectionTimeout="60000"
> enableLookups="false"
> maxPostSize="2097152"
> maxSpareThreads="10"
> maxThreads="30"
> minSpareThreads="5"
> port="8006"
> protocol="AJP/1.3"
> tcpNoDelay="true"
> />
>
> Mentioned below are the settings for JNDI resource configured in conf/context.xml:
>
> <Resource
> name="jdbc/onlinedb"
> auth="Container"
> type="javax.sql.DataSource"
> driverClassName="oracle.jdbc.OracleDriver"
> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
> initialSize="1"
> maxActive="30"
> minIdle="1"
> maxIdle="5"
> maxWait="300000"
> poolPreparedStatements="true"
> maxOpenPreparedStatements="300"
> validationQuery="SELECT 1 FROM BB_DUAL"
> testOnBorrow="true"
> validationInterval="10000"
> testWhileIdle="true"
> />
>
> JVM Parameters:
>
> -Xms512m -Xmx2048m
> -XX:PermSize=128m -XX:MaxPermSize=256m
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
>
> It must be noted that Weblogic setup performs very well using similar settings.
>
> Garbage Collection: While the load test is running, Garbage collection works just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a second. Full GC occurs every hour and takes a little over 2 seconds.
>
> Any tips/pointers will be greatly appreciated.
>
> Talha.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by Talha Fazal <tf...@credera.com>.
WE GENERATE REPORT ON NETWORK BACKLOG USING ADVANCED TOOLS, WHICH INDICATE NEGLIGIBLE NETWORK DELAY. AT ONE POINT TO TIME THIS WAS AN ISSUE. WE INCREASED THE BANDWIDTH FROM 45 MBPS TO 100 MBPS WHICH RESOLVED THE ISSUE.
THANKS, TALHA.
-----Original Message-----
From: Caldarale, Charles R [mailto:Chuck.Caldarale@unisys.com]
Sent: Friday, September 02, 2011 9:16 AM
To: Tomcat Users List
Subject: RE: Tomcat Performance Turning.
> From: Talha Fazal [mailto:tfazal@credera.com]
> Subject: RE: Tomcat Performance Turning.
> We did take a thread dump and we found a lot of threads locked.
> "http-8014-9" daemon prio=10 tid=0x0000000060965c00 nid=0x6c83 in Object.wait()
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
Often, a throughput problem is not the many, but just one that is holding all of the others up. The example you cite above is simply a thread waiting for something to do.
Do you perhaps have a network problem, where requests are simply not getting delivered to httpd or Tomcat in a timely fashion?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Talha Fazal [mailto:tfazal@credera.com]
> Subject: RE: Tomcat Performance Turning.
> We did take a thread dump and we found a lot of threads locked.
> "http-8014-9" daemon prio=10 tid=0x0000000060965c00 nid=0x6c83 in Object.wait()
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
Often, a throughput problem is not the many, but just one that is holding all of the others up. The example you cite above is simply a thread waiting for something to do.
Do you perhaps have a network problem, where requests are simply not getting delivered to httpd or Tomcat in a timely fashion?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by Talha Fazal <tf...@credera.com>.
Thx Charles! We did take a thread dump and we found a lot of threads locked. Please see a short sample below:
"http-8014-9" daemon prio=10 tid=0x0000000060965c00 nid=0x6c83 in Object.wait() [0x000000004c688000..0x000000004c688c90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
- locked <0x00002aaae10403b0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
at java.lang.Thread.run(Thread.java:619)
Sincerely, Talha.
-----Original Message-----
From: Caldarale, Charles R [mailto:Chuck.Caldarale@unisys.com]
Sent: Friday, September 02, 2011 8:42 AM
To: Tomcat Users List
Subject: RE: Tomcat Performance Turning.
> From: Talha Fazal [mailto:tfazal@credera.com]
> Subject: Tomcat Performance Turning.
> In our staging environment for load testing, when we run the load
> test using 525 concurrent users, the app doesn't perform at all.
> The CPU usage (on Apache and Tomcat Servers) hovers between 7% to
> 8%. The database server CPU usage is also between 4 and 5%.
Since your CPU usage is low, your threads must be waiting for something. Take several thread dumps and find out what.
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
(I'll guess maxThreads and data base connections; a max of 30 seems a bit low for 525 concurrent requests.)
> validationQuery="SELECT 1 FROM BB_DUAL"
Most DBs provide a very simple ping-like request capability; that would be preferable to a select.
> -Xms512m -Xmx2048m
In a server environment, one normally sets Xms and Xmx to the same value to avoid heap thrashing. However, this is unrelated to your current problem.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Performance Turning.
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Talha Fazal [mailto:tfazal@credera.com]
> Subject: Tomcat Performance Turning.
> In our staging environment for load testing, when we run the load
> test using 525 concurrent users, the app doesn't perform at all.
> The CPU usage (on Apache and Tomcat Servers) hovers between 7% to
> 8%. The database server CPU usage is also between 4 and 5%.
Since your CPU usage is low, your threads must be waiting for something. Take several thread dumps and find out what.
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
(I'll guess maxThreads and data base connections; a max of 30 seems a bit low for 525 concurrent requests.)
> validationQuery="SELECT 1 FROM BB_DUAL"
Most DBs provide a very simple ping-like request capability; that would be preferable to a select.
> -Xms512m -Xmx2048m
In a server environment, one normally sets Xms and Xmx to the same value to avoid heap thrashing. However, this is unrelated to your current problem.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org