You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by McAfe <ne...@yahoo.com> on 2011/05/24 09:01:42 UTC
Performance & *SpareThreads
Hi,
I got few question would like ask hope someone answer my question, below is
the server information
Windows 2003 server
Tomcat 6.0 (configured java initial / maximum memory pool: 1024 MB)
Apache 2.2 (using mod_jk)
Java 1.6
The testing result I've use Jmeter (configured 100 thread per-second) it
will die when thread(Jmeter) between 200 ~ 300
>From the research get to know minSpareThreads and maxSpareThreads, after set
the configure it can handle more to 400 ~ 600
(1st) What is the minSpareThreads and maxSpareThreads is? [Unable to find
the description on tomcat 6.0 but able find for tomcat 5.5 as
http://tomcat.apache.org/tomcat-5.5-doc/config/http.html]
(2nd) From read 1 by 1 at the forums about Performance tuning - tomcat, some
mention keep hit > 20k per seconds, how their do it?
I'm new to the tomcat / apache, hope someone can point me a link or useful
information to help / improve the performance
Thank You very much
- McAfe
--
View this message in context: http://old.nabble.com/Performance---*SpareThreads-tp31687863p31687863.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance & *SpareThreads
Posted by McAfe <ne...@yahoo.com>.
Hi Chris,
Noted and thank you for your feedback (let me try 1st) as I no idea which
mod_* is using on the setting
- McAfe
--
View this message in context: http://old.nabble.com/Performance---*SpareThreads-tp31687863p31713191.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance & *SpareThreads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
McAfe,
On 5/26/2011 12:00 AM, McAfe wrote:
> And I've changed to following (the result did not improve or any better
> sign)
> MaxClients 256
>
> [snip]
>
> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> maxThreads="150" minSpareThreads="25" maxSpareThreads="75"/>
You're going to want MaxClients = maxThreads, so set maxThreads="256", here.
> <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
> enableLookups="false" disableUploadTimeout="true"
> connectionTimeout="20000"
> redirectPort="8443" />
Are you sure you're using mod_jk? HTTP/1.1 over port 8080 does not smell
like a mod_jk configuration.
> The scenario testing in Jmeter, before that in the tomcat I've 3 different
> webapps (The Jmeter testing have 3 HTTP request and each point to each
> webapps in the tomcat)
>
> 1) Number of thread = 200 and ramp-up period=1
> Result: Test 10 / Pass 10
> 2) Number of thread = 300 and ramp-up period=1
> Overall result: Test 10 each test will get 3%~5% error (but the tomcat /
> apache still working)
> * The error is java.net.ConnectException: Connection refused: connect
With 256 connections available and 300 trying to connect, it's not
surprising that you are getting errors at this stage.
> 3) Number of thread = 500 and ramp-up period=1
> Overall result: Test 10 each test will get 20%~50% error (but the tomcat
> / apache still working)
> * The error is java.net.ConnectException: Connection refused: connect
Same issue here: you just don't have enough connections on the server to
handle the load. This is what your clients will experience if you were
to experience this much load in production.
You have three options:
1. Increase the number of connections you allow. That's easy: just raise
MaxClients and maxThreads. Note that if you run out of server
resources (threads/processes, CPU time, memory, I/O throughput),
this won't help at all.... you will have just hit a wall.
2. Increase the acceptCount on the <Connector> (if you are using Tomcat
only) or ListenBacklog (if you are using Apache httpd). This will
allow the OS's TCP stack to accept more connections and queue them
while waiting for your webapp's connections to become free.
Note that if you are handling sustained load, this also doesn't
help at all, because your accept queue will always be full.
3. Improve your web application. Try consulting a profiler or, if you
are heavily relying on a database, consult a DBA. You may just have
to increase the number of connections that your /database/ can
handle. If you want to serve 1000 simultaneous requests, but your
DB connection pool only has 10 connections, then 990 of those
requests will be sitting there waiting for the database to be
available.
4. Improve your infrastructure. Buy more/better servers, use
load-balancing, etc.
> My plan is to target Number of thread = 1000 and ramp-up period =1 (I no
> idea this can it be done) and the result is 100% if possible (so I can have
> the benchmarks)
>
> In the same time I will look on database connection, thank you as yesterday
> you have giving the tips of IO part and worry when grab data from database
> it may delay.
Performance can be improved by looking at the types of queries you are
running, considering indexing, caching, etc. Only you know what can
benefit your application. I'm sure there are lots of folks on the list
(myself included) who would be happy to get paid to help you evaluate
your own webapp and make specific suggestions.
If it's not directly related to Tomcat (and in this case, it's almost
certainly not), interest on this list for completely free help is
somewhat limited. :)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3eWs4ACgkQ9CaO5/Lv0PBq1wCffNbOpYD7GJx8Oxr+OE0PQgNW
OXYAn3jJ4RxWzoez1GuVRF7Rm0QhG2u2
=LtaO
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance & *SpareThreads
Posted by McAfe <ne...@yahoo.com>.
Hi Chris,
The Apache MPM I was using the default configuration (believe is prefork
MPM)
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
And I've changed to following (the result did not improve or any better
sign)
StartServers 64
MinSpareServers 64
MaxSpareServers 128
MaxClients 256
MaxRequestsPerChild 0
For tomcat executor this my setting, as I mention early
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"/>
<Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
enableLookups="false" disableUploadTimeout="true"
connectionTimeout="20000"
redirectPort="8443" />
The scenario testing in Jmeter, before that in the tomcat I've 3 different
webapps (The Jmeter testing have 3 HTTP request and each point to each
webapps in the tomcat)
1) Number of thread = 200 and ramp-up period=1
Result: Test 10 / Pass 10
2) Number of thread = 300 and ramp-up period=1
Overall result: Test 10 each test will get 3%~5% error (but the tomcat /
apache still working)
* The error is java.net.ConnectException: Connection refused: connect
3) Number of thread = 500 and ramp-up period=1
Overall result: Test 10 each test will get 20%~50% error (but the tomcat
/ apache still working)
* The error is java.net.ConnectException: Connection refused: connect
My plan is to target Number of thread = 1000 and ramp-up period =1 (I no
idea this can it be done) and the result is 100% if possible (so I can have
the benchmarks)
In the same time I will look on database connection, thank you as yesterday
you have giving the tips of IO part and worry when grab data from database
it may delay.
Thank You for your time and knowledge :-)
- McAfe
--
View this message in context: http://old.nabble.com/Performance---*SpareThreads-tp31687863p31704618.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance & *SpareThreads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
McAfe,
On 5/25/2011 2:24 AM, McAfe wrote:
> Christopher Schultz-2 wrote:
>> AJP expects to have persistent connections. What is your Apache httpd
>> configuration for StartServers/MinSpareServers/MaxSpareServers (for
>> prefork MPM) or startServers/MinSpareThreads/MaxSpareThreads (for worker
>> MPM)?
>
> Mcafe[20110523]: Did not configure Apache so far target on Tomcat (as for
> research from google and test the result in Apache not much different)
If Apache httpd is in it's default configuration, then you likely have
either the worker or prefork MPM with a maximum simultaneous client
MaxClients=256.
>> What are you settings on the Tomcat side for your AJP <Connector>?
>
> Mcafe[20110523]: the follows is my setting for AJP <Connector>
>
> <Connector port="8080" protocol="HTTP/1.1"
> maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
> enableLookups="false" disableUploadTimeout="true"
> connectionTimeout="20000"
> redirectPort="8443" />
The above <Connector> is for HTTP/1.1 and not AJP. Are you sure you are
using Apache httpd in your setup?
If you *are* using AJP on another similarly-configured <Connector>,
since Apache httpd can handle 256 connections (you'll have to check your
MaxClients setting) but Tomcat can only handle 150 connections, you have
the possibility of httpd trying to forward a request to Tomcat that will
fail because all of your AJP connections are busy. Since AJP connections
are expected to be persistent, queuing the request in the TCP/IP stack
is pointless: the request will never be handled. Instead, mod_jk fails
fast and you'll probably get errors on the client (usually 500 or
something similar). Check your mod_jk log (or configure one if you don't
have one) and check to see if you are getting connection errors.
>>>> From the research get to know minSpareThreads and maxSpareThreads, after
>>>> set
>>>> the configure it can handle more to 400 ~ 600
>
>> minSpareThreads and maxSpareThreads shouldn't be used when using AJP:
>> you should be using the exact number configured on the httpd side.
>> Otherwise, you risk AJP connections churning.
>
> Mcafe[20110523]: Sorry no idea about this, as I'm new and wanted to learn
> more. Fine tuning is not simple thing as just develop, hope you can give
> me more example
Use an executor:
<Executor name="thread-pool" maxThreads="..." minSpareThreads="..." />
<Connector port="8080" protocol="HTTP/1.1" executor="thread-pool" />
> Mcafe[20110523]: Yup saw the default setting in the server.xml but unable
> to find any useful example for the setting tried and the result not much
> different (from the testing)
>
> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> maxThreads="150" minSpareThreads="4"/>
That should work. Remember to set the "executor" attribute in your
<Connector> to link them together.
>>>> (2nd) From read 1 by 1 at the forums about Performance tuning - tomcat,
>>>> some
>>>> mention keep hit > 20k per seconds, how their do it?
>>
>> That depends a lot on what kind of load it was under. Serving static
>> files is fast. Running database transactions is slow. Also, sizing the
>> various pools can have an effect: you can max-out the throughput of a
>> single client but still have plenty of CPU time and I/O time available
>> on the server. If that's the case, make your connection pools bigger.
>
> Mcafe[20110523]: Yup, the webapps have to grab data from database and
> return according. I'm interesting with "you can max-out the throughput of
> a
> single client but still have plenty of CPU time and I/O time available
> on the server. If that's the case, make your connection pools bigger."
>
> Can you explain more?
When you are making lots of requests, what does the CPU time and I/O
throughput look like? Use a tool like sar to take samples and maybe
graph them. Try taking samples during a load factor of 1 (just a single
request-at-a-time) and then a load factor of 10 (10 requests at a time)
and then a load factor of maybe 50 or 100. Compare the graphs to see if
it looks like you are hitting a throughput maximum. If not, you can
certainly handle more connections.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3dBB8ACgkQ9CaO5/Lv0PCMgQCdHbe573mrNjBux0fgnT9XLcwJ
rqIAn1Fv5G+MbEouM/QD4dxxsS92e5xC
=JS6Z
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance & *SpareThreads
Posted by McAfe <ne...@yahoo.com>.
Hi Chris,
My reply as follows Mcafe[20110523] and thanks for the reply
- McAfe
Christopher Schultz-2 wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> McAfe,
>
> On 5/24/2011 3:01 AM, McAfe wrote:
>> Apache 2.2 (using mod_jk)
>>
>> The testing result I've use Jmeter (configured 100 thread per-second) it
>> will die when thread(Jmeter) between 200 ~ 300
>
> AJP expects to have persistent connections. What is your Apache httpd
> configuration for StartServers/MinSpareServers/MaxSpareServers (for
> prefork MPM) or startServers/MinSpareThreads/MaxSpareThreads (for worker
> MPM)?
>
> Mcafe[20110523]: Did not configure Apache so far target on Tomcat (as for
> research from google and test the result in Apache not much different)
>
> What are you settings on the Tomcat side for your AJP <Connector>?
>
> Mcafe[20110523]: the follows is my setting for AJP <Connector>
>
> <Connector port="8080" protocol="HTTP/1.1"
> maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
> enableLookups="false" disableUploadTimeout="true"
> connectionTimeout="20000"
> redirectPort="8443" />
>
>> From the research get to know minSpareThreads and maxSpareThreads, after
>> set
>> the configure it can handle more to 400 ~ 600
>
> minSpareThreads and maxSpareThreads shouldn't be used when using AJP:
> you should be using the exact number configured on the httpd side.
> Otherwise, you risk AJP connections churning.
>
> Mcafe[20110523]: Sorry no idea about this, as I'm new and wanted to learn
> more. Fine tuning is not simple thing as just develop, hope you can give
> me more example
>
>> (1st) What is the minSpareThreads and maxSpareThreads is? [Unable to find
>> the description on tomcat 6.0 but able find for tomcat 5.5 as
>> http://tomcat.apache.org/tomcat-5.5-doc/config/http.html]
>
> Tomcat 6 and 7 no longer use these settings. If you want a
> dynamically-sized thread pool, use an <Executor> instead.
> http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html
>
> Mcafe[20110523]: Yup saw the default setting in the server.xml but unable
> to find any useful example for the setting tried and the result not much
> different (from the testing)
>
> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> maxThreads="150" minSpareThreads="4"/>
>
>> (2nd) From read 1 by 1 at the forums about Performance tuning - tomcat,
>> some
>> mention keep hit > 20k per seconds, how their do it?
>
> That depends a lot on what kind of load it was under. Serving static
> files is fast. Running database transactions is slow. Also, sizing the
> various pools can have an effect: you can max-out the throughput of a
> single client but still have plenty of CPU time and I/O time available
> on the server. If that's the case, make your connection pools bigger.
>
> Mcafe[20110523]: Yup, the webapps have to grab data from database and
> return according. I'm interesting with "you can max-out the throughput of
> a
> single client but still have plenty of CPU time and I/O time available
> on the server. If that's the case, make your connection pools bigger."
>
> Can you explain more?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk3b86gACgkQ9CaO5/Lv0PD/BgCgn4GRuqON8mS9Y0IO9j8ra0Ou
> TTgAn08U0AgGtpi6JE+nnk1OLNLlhzDK
> =zahv
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://old.nabble.com/Performance---*SpareThreads-tp31687863p31696321.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance & *SpareThreads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
McAfe,
On 5/24/2011 3:01 AM, McAfe wrote:
> Apache 2.2 (using mod_jk)
>
> The testing result I've use Jmeter (configured 100 thread per-second) it
> will die when thread(Jmeter) between 200 ~ 300
AJP expects to have persistent connections. What is your Apache httpd
configuration for StartServers/MinSpareServers/MaxSpareServers (for
prefork MPM) or startServers/MinSpareThreads/MaxSpareThreads (for worker
MPM)?
What are you settings on the Tomcat side for your AJP <Connector>?
> From the research get to know minSpareThreads and maxSpareThreads, after set
> the configure it can handle more to 400 ~ 600
minSpareThreads and maxSpareThreads shouldn't be used when using AJP:
you should be using the exact number configured on the httpd side.
Otherwise, you risk AJP connections churning.
> (1st) What is the minSpareThreads and maxSpareThreads is? [Unable to find
> the description on tomcat 6.0 but able find for tomcat 5.5 as
> http://tomcat.apache.org/tomcat-5.5-doc/config/http.html]
Tomcat 6 and 7 no longer use these settings. If you want a
dynamically-sized thread pool, use an <Executor> instead.
http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html
> (2nd) From read 1 by 1 at the forums about Performance tuning - tomcat, some
> mention keep hit > 20k per seconds, how their do it?
That depends a lot on what kind of load it was under. Serving static
files is fast. Running database transactions is slow. Also, sizing the
various pools can have an effect: you can max-out the throughput of a
single client but still have plenty of CPU time and I/O time available
on the server. If that's the case, make your connection pools bigger.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3b86gACgkQ9CaO5/Lv0PD/BgCgn4GRuqON8mS9Y0IO9j8ra0Ou
TTgAn08U0AgGtpi6JE+nnk1OLNLlhzDK
=zahv
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org