You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Rainer Jung <ra...@kippdata.de> on 2008/10/01 14:00:44 UTC
Re: JK and IIS - troubles?
br1 wrote:
> Dear all,
>
> I am experiencing a weird problem happening between IIS and Tomcat: it seems
> that the JK connector is hanging IIS.
>
> A badly behaving Tomcat app is filling up AJP connections due to some DB
> connection problems. Nothing strange so far, we had these kinds of problems
> already, but this time this is causing IIS to hang.
> Simply restarting the failing app's Tomcat instance causes IIS to start
> again like it was simply frozen.
> Instead, restarting IIS makes obviously the whole site restart, except for
> the failing application.
>
> I struggled with this problem for the past months on two servers, and just
> configured an additional IIS on our dedicated Tomcat machine with the same
> results. Configuration is as follows:
> IIS 5
> JK 1.2.26 (/maybe some other 1.2.2x version but cannot be sure)
You and we need to be sure. Set you JK log level to at least info. All
relatively new JK versions will log their version in the log file during
startup.
> Tomcat 5.5.9 / 5.5.27
> JDK 1.5.0_12 / 1.5.0_16 / 1.6.0_07
> About 40 workers and Tomcats
>
> Since we use Tomcat since version 3 and Jk since before year 2k - sorry
> Linux guys - on Windows platforms without a single serious problem, I am a
> bit sorry not to be able to solve this one by myself now.
> Clearly the application needs patching, and the affected Tomcat is
> restarting hourly now, but to me the JK connector should be fixed if it is
> causing this problem.
>
> Kindly let me know what should I provide in order to investigate about
> this.
1) Finding the real reason: Assuming some requests send to your Tomcat
simply get stuck and do not return fast enough, you'll need to take
thread dumps of Tomcat and look at those. Take the dumps when the
problem is there and take more than one, e.g. 3 dumps each 3 seconds
apart from the previous one.
2) Making IIS/mod_jk more robust against the problem: Use timeouts as
described on page
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
Those will not make your app work, when in fact requests in Tomcat get
stuck and block all available connector threads, but it will at least
keep iis/mod_jk running.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: JK and IIS - troubles?
Posted by Martin Gainty <mg...@hotmail.com>.
as mt mentioned in his article
http://people.apache.org/~mturk/docs/article/ftwai.html
non server window products has an upper limit of 10 connections
read the article..and tune the connection_pool_size parameters according to the parameters your product supports..
Martin
______________________________________________
Disclaimer and confidentiality note
Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission.
> Date: Sat, 4 Oct 2008 16:09:23 -0700
> From: myrealbruno@myrealbox.com
> To: users@tomcat.apache.org
> Subject: Re: JK and IIS - troubles?
>
>
> Mturk,
>
> Thanks for your reply.
>
>
> mturk wrote:
> >
> > Nope, it won't.
> > The reason is that you are using IIS6+ that has worker processes,
> >
> IIS 5
>
>
> mturk wrote:
> >
> > and ISAPI API provides no way of knowing what that number is.
> > The entire calculation is up to you, because depending on both
> > the IIS version and how you setup the worker processes and their
> > number the connection_pool_size is either MaxClients or
> > ThreadsPerChild (from httpd)
> >
> > There is no magic wand for every possible use case.
> > Like with anything you'll need to switch on the brain and
> > do some tuning by hand (setting the workers connection_pool_size is
> > all you need to do) that will match your topology.
> >
> My objection was not exactly technical.
> As I said I already did the tuning you are suggesting, when the default was
> too few connections, not too many.
> Still, I would rather have a too low default value on JK and get a
> corresponding error message on Tomcat asking to raise maxThreads, which was
> happening in the past.
> A default 250 connections on the JK side on IIS does not exactly force
> anyone to fine tune anything, and if too many connections are the reason of
> my problem I would still suggest to lower this number.
>
> Though, so far it is not clear at all that this is the cause. :-)
>
> Thanks,
> br1
>
> --
> View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19818936.html
> Sent from the Tomcat - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
_________________________________________________________________
See how Windows connects the people, information, and fun that are part of your life.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Mturk,
Thanks for your reply.
mturk wrote:
>
> Nope, it won't.
> The reason is that you are using IIS6+ that has worker processes,
>
IIS 5
mturk wrote:
>
> and ISAPI API provides no way of knowing what that number is.
> The entire calculation is up to you, because depending on both
> the IIS version and how you setup the worker processes and their
> number the connection_pool_size is either MaxClients or
> ThreadsPerChild (from httpd)
>
> There is no magic wand for every possible use case.
> Like with anything you'll need to switch on the brain and
> do some tuning by hand (setting the workers connection_pool_size is
> all you need to do) that will match your topology.
>
My objection was not exactly technical.
As I said I already did the tuning you are suggesting, when the default was
too few connections, not too many.
Still, I would rather have a too low default value on JK and get a
corresponding error message on Tomcat asking to raise maxThreads, which was
happening in the past.
A default 250 connections on the JK side on IIS does not exactly force
anyone to fine tune anything, and if too many connections are the reason of
my problem I would still suggest to lower this number.
Though, so far it is not clear at all that this is the cause. :-)
Thanks,
br1
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19818936.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Mladen Turk <mt...@apache.org>.
br1 wrote:
> Rainer,
> So perhaps.. upgrading from older JKs the default changed from 10
> connections to 250. With a lot of workers (for instance, 40) the total
> number of maximum threads went from 400 to 10000. I will have to fix this
> and check what happens.
>
> Though I understand the perfect default value can vary depending on the site
> load etc., to me 10 is too low and 250 is far too high.. maybe a 50 would
> fit for most installations and be a better default number in this case. :-)
>
Nope, it won't.
The reason is that you are using IIS6+ that has worker processes,
and ISAPI API provides no way of knowing what that number is.
The entire calculation is up to you, because depending on both
the IIS version and how you setup the worker processes and their
number the connection_pool_size is either MaxClients or
ThreadsPerChild (from httpd)
There is no magic wand for every possible use case.
Like with anything you'll need to switch on the brain and
do some tuning by hand (setting the workers connection_pool_size is
all you need to do) that will match your topology.
Regards
--
^(TM)
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Rainer Jung <ra...@kippdata.de>.
br1 wrote:
> Apologies,
>
> This one is much better, netstat shows 50 connections
> I don't know enough of Tomcat to understand if anything in this log could
> cause this issue..
Much better: You have a synchronization issue in your database
connection pool. It seems you are using the c3p0 pool, which can be
found at:
http://c3p0.cvs.sourceforge.net/
Your thread dumps (all three of them) show, that 52 threads are waiting
to get a connection from the pool:
java.lang.Object.wait(Native Method)
- waiting on <0x07c3cc50> (a com.mchange.v2.resourcepool.BasicResourcePool)
com.mchange.v2.resourcepool.BasicResourcePool.awaitAcquire(BasicResourcePool.java:968)
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:208)
- locked <0x07c3cc50> (a com.mchange.v2.resourcepool.BasicResourcePool)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:260)
com.mchange.v2.c3p0.PoolBackedDataSource.getConnection(PoolBackedDataSource.java:94)
com.mchange.v2.c3p0.ComboPooledDataSource.getConnection(ComboPooledDataSource.java:521)
org.theorg.thebranch.util.Sql.GetMySqlJDBCConnFromPool(Sql.java:61)
Those calls are coming from three JSPs:
Count JSP
47 jinclude.pager_jsp
3 ag_005ffigures_jsp
2 ag_005falpha_jsp
but those are not the problems. The problem seems to be either in the
pool, or the application doesn't return connections correctly to the
pool after use. The c3p0 web site has a couple of bug entries, which at
first site look related.
One Thread TP-Processor number 8 is working on some request in all three
dumps, returning content via the default servlet.
In this situation further requests would simply get stuck in the same
stack, eating up IIS threads and connections.
You can free the IIS threads by using a reply_timeout for the worker,
but you can't break the waiting for the pool inside tomcat. If you
consider using a reply_timeout, use version 1.2.26 of the redirector and
don't set the timeout to low. Also consider using max_reply_timeouts. If
the reply_timeout fires (and you got more than max_reply_timeouts of
them), then the redirector will disable the whole worker (Tomcat) for
some time. In the observed situation this is the correct way of handling
it, but you don't want to set the timeout let's say to 5 seconds and
then notice, that every now and then a Tomcat gets disabled because one
request took longer than 5 seconds. Have a look at the Timeouts docs page.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Apologies,
This one is much better, netstat shows 50 connections
I don't know enough of Tomcat to understand if anything in this log could
cause this issue..
Thanks!
Br1
http://www.nabble.com/file/p19920236/tomcat%2Bthread%2Bdump%2B4%2B-%2Bfiltered.zip
tomcat+thread+dump+4+-+filtered.zip
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19920236.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Rainer Jung <ra...@kippdata.de>.
br1 wrote:
> I managed to schedule a few thread dumps (3, with a 5 seconds interval) and
> a Tomcat restart in case of troubles, and just implemented some more
> logging. I kept the faulty Tomcat app running all day.
> There were three failures today, but thanks to the logging I just added I
> can be only sure about the last one, and it was blocking the whole site. As
> usual, a simple Tomcat restart cured the whole site hang.
>
> The HTTP error the application gives, as well as the error of the site root
> (that BTW is a static html page) simply is: The connection with the server
> was terminated abnormally
> Very strangely, an (automated) netstat in this last event shows just four
> (4) ESTABLISHED connections on the Tomcat server from the IIS server when
> everything was failing.
Can you provide the netstat (because of having a look at other
connection stati aprt from ESTABLISHED).
> I am enclosing the last thread dump here, hoping it could shed some light.
Unfortunately the thread dumps show that Tomcat is basically idle. Three
threads are connected to the redirector and wait for the next request,
and one thread actually works on a request (ag_005fclassifschemes JSP).
A few additional threads are sitting idle in the pool, waiting for
handling future connections.
The picture is the same in all 3 dumps.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Hello again Rainer,
Rainer Jung-3 wrote:
>
> But if you are really actively using 300 connections, it means there are
> 300 requests processed in parallel inside Tomcat. So you should first
> check with the thread dump, what those are actually doing, and whether
> it's a good idea to send even more requests there.
>
Definitely it is not a good idea. Details and a thread dump below.
Rainer Jung-3 wrote:
>
> If the web server
> can't acquire another connection from the pool, further requests should
> fail, not block.
>
Totally agreed.
Please note, if I was not clear enough before, that my goal is to avoid that
any failing Tomcat application hangs the whole web server (and site).
This should not happen at all, whatever settings in the JK connector.. I am
not going to solve this particular application's problem before the JK
connection works properly - though I will send thread dumps as appropriate
to the application developer, and drink a beer with him when this is all
over.. but this is another story.
I changed the settings just a bit, so now the IIS side JK connections are
limited to 50 instead of the previous 250 and 300.
I managed to schedule a few thread dumps (3, with a 5 seconds interval) and
a Tomcat restart in case of troubles, and just implemented some more
logging. I kept the faulty Tomcat app running all day.
There were three failures today, but thanks to the logging I just added I
can be only sure about the last one, and it was blocking the whole site. As
usual, a simple Tomcat restart cured the whole site hang.
The HTTP error the application gives, as well as the error of the site root
(that BTW is a static html page) simply is: The connection with the server
was terminated abnormally
Very strangely, an (automated) netstat in this last event shows just four
(4) ESTABLISHED connections on the Tomcat server from the IIS server when
everything was failing.
I am enclosing the last thread dump here, hoping it could shed some light.
Please let me know - and forgive me - if I forgot to provide additional
information, configuration files, etc.
Some of my colleagues use to tell me: "Someone, somewhere in the world, had
some problem using something. Can you please solve this."
Thanks for your help,
Br1
http://www.nabble.com/file/p19907446/tomcat%2Bthread%2Bdump%2B3%2B-%2Bfiltered.zip
tomcat+thread+dump+3+-+filtered.zip
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19907446.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Rainer Jung <ra...@kippdata.de>.
br1 schrieb:
> Rainer Jung-3 wrote:
>> Yes, if the notion of worker is an IIS worker and not an isapi plugin
>> worker. A plugin worker in the sense of a worker configuration item in
>> workers.properties is 1:1 with a connection pool, and a connection pool
>> doesn't create threads. It will grow at most as the lower of thread
>> number and configured pool size.
>>
> Well, no, maybe we need to go some steps back.
> I am talking about JK workers (isapi plugin workers), and I have no IIS
> workers: the problem is on IIS 5.
> As I said I have a connection_pool_size currently set to 300 on the JK
> worker (IIS side) and a matching maxThreads set to 300 on the Tomcat side.
>
> What you are telling me is that I don't have 300 threads on this IIS/JK
> worker side, but simply (a maximum of) 300 tcp connections to the Tomcat
> instance, is this right? So maybe I am just out of connections..
But if you are really actively using 300 connections, it means there are
300 requests processed in parallel inside Tomcat. So you should first
check with the thread dump, what those are actually doing, and whether
it's a good idea to send even more requests there. If the web server
can't acquire another connection from the pool, further requests should
fail, not block.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Rainer,
Rainer Jung-3 wrote:
>
> If you like, we are always keen on helpful documentation. Maybe a page
> about sizing connection pool and the relation between connections, web
> server processes, IIS workers and Tomcat threads, measuring your needs
> and the dangers arising from going to far.
>
Sure, as soon as I understand the problem completely, which I did not so
far.
Rainer Jung-3 wrote:
>
> I see, yes the default changed end of 2006 between 1.2.19 and 1.2.20.
> Usually we don't change defaults, but the 10 was really to low.
>
Though I keep thinking it is probably better to choose a small default than
a large one in this case.
Rainer Jung-3 wrote:
>
> Yes, if the notion of worker is an IIS worker and not an isapi plugin
> worker. A plugin worker in the sense of a worker configuration item in
> workers.properties is 1:1 with a connection pool, and a connection pool
> doesn't create threads. It will grow at most as the lower of thread
> number and configured pool size.
>
Well, no, maybe we need to go some steps back.
I am talking about JK workers (isapi plugin workers), and I have no IIS
workers: the problem is on IIS 5.
As I said I have a connection_pool_size currently set to 300 on the JK
worker (IIS side) and a matching maxThreads set to 300 on the Tomcat side.
What you are telling me is that I don't have 300 threads on this IIS/JK
worker side, but simply (a maximum of) 300 tcp connections to the Tomcat
instance, is this right? So maybe I am just out of connections..
Rainer Jung-3 wrote:
>
> Done.
>
Though it would fit better on "connections" page more than on a "timeouts"
page. :-)
Thanks for the help!
br1
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19818811.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Rainer Jung <ra...@kippdata.de>.
br1 schrieb:
> Rainer Jung-3 wrote:
>> That might be the reason for trouble. If things start to get slow, the
>> web server gets filled by all thenew requests still coming in without
>> answering fast enough the existing ones.
>>
> This starts to make sense now.
> I have a few suggestions inline, I would like to contribute to document a
> case like this, if possible.
If you like, we are always keen on helpful documentation. Maybe a page
about sizing connection pool and the relation between connections, web
server processes, IIS workers and Tomcat threads, measuring your needs
and the dangers arising from going to far.
> Rainer Jung-3 wrote:
>> People often try to work around
>> this by increasing the number of threads available for processing the
>> requests,
>>
> Yes, some of us might still be used to old JK connectors defaulting to 10
> connections.
> More likely to cause confusion is the Tomcat message "increase the threads",
> while it might help more if it was "fine tune the threads" instead.
I see, yes the default changed end of 2006 between 1.2.19 and 1.2.20.
Usually we don't change defaults, but the 10 was really to low.
> Rainer Jung-3 wrote:
>> but if the backend is stuck or too slow, this will not help
>> and instead also trash the frontend.
>>
> So perhaps.. upgrading from older JKs the default changed from 10
> connections to 250. With a lot of workers (for instance, 40) the total
> number of maximum threads went from 400 to 10000. I will have to fix this
> and check what happens.
Yes, if the notion of worker is an IIS worker and not an isapi plugin
worker. A plugin worker in the sense of a worker configuration item in
workers.properties is 1:1 with a connection pool, and a connection pool
doesn't create threads. It will grow at most as the lower of thread
number and configured pool size.
For iis workers see Mladen's response.
> Rainer Jung-3 wrote:
>> You should measure how many connections you need during peak hours
>> without performance problems, and then add some percentage depending on
>> your growth rate etc. Finally you need to make sure, that your web
>> server itself uses at least as many threads, as you configured as the
>> pool size.
>>
> This has been measured already. Same as above, when the problem were too few
> connections, too many seemed not to be a problem. :-)
> I would suggest to add your above sentence right after "We recommend
> adjusting this value for IIS and the Sun Web Server" at
> http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html and
> maybe also in http://tomcat.apache.org/connectors-doc/reference/workers.html
> (And maybe *strongly* recommend it )
Done.
> Rainer Jung-3 wrote:
>> For Apache with mod_jk we can automatically detect the number of threads
>> and by default size our pool to the same number. For IIS you have to
>> size the pool yourself (or live with the big default of 250).
>>
> It seems we are dying with the big default. :-)
>
> I have read somewhere that the number of threads is not likely to cause
> these problems on modern multiple cpu with multiple cores servers. I wonder
> if there is some kind of formula to know what total should not be exceeded
> in this particular case, of if the good old document by Mladen Turk is still
> valid fixing this number to 2000
> (http://people.apache.org/~mturk/docs/article/ftwai.html)
I do think so, although I didn't actually test recently.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Rainer,
Rainer Jung-3 wrote:
>
> That might be the reason for trouble. If things start to get slow, the
> web server gets filled by all thenew requests still coming in without
> answering fast enough the existing ones.
>
This starts to make sense now.
I have a few suggestions inline, I would like to contribute to document a
case like this, if possible.
Rainer Jung-3 wrote:
> People often try to work around
> this by increasing the number of threads available for processing the
> requests,
>
Yes, some of us might still be used to old JK connectors defaulting to 10
connections.
More likely to cause confusion is the Tomcat message "increase the threads",
while it might help more if it was "fine tune the threads" instead.
Rainer Jung-3 wrote:
>
> but if the backend is stuck or too slow, this will not help
> and instead also trash the frontend.
>
So perhaps.. upgrading from older JKs the default changed from 10
connections to 250. With a lot of workers (for instance, 40) the total
number of maximum threads went from 400 to 10000. I will have to fix this
and check what happens.
Though I understand the perfect default value can vary depending on the site
load etc., to me 10 is too low and 250 is far too high.. maybe a 50 would
fit for most installations and be a better default number in this case. :-)
Rainer Jung-3 wrote:
>
> You should measure how many connections you need during peak hours
> without performance problems, and then add some percentage depending on
> your growth rate etc. Finally you need to make sure, that your web
> server itself uses at least as many threads, as you configured as the
> pool size.
>
This has been measured already. Same as above, when the problem were too few
connections, too many seemed not to be a problem. :-)
I would suggest to add your above sentence right after "We recommend
adjusting this value for IIS and the Sun Web Server" at
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html and
maybe also in http://tomcat.apache.org/connectors-doc/reference/workers.html
(And maybe *strongly* recommend it )
Rainer Jung-3 wrote:
>
> For Apache with mod_jk we can automatically detect the number of threads
> and by default size our pool to the same number. For IIS you have to
> size the pool yourself (or live with the big default of 250).
>
It seems we are dying with the big default. :-)
I have read somewhere that the number of threads is not likely to cause
these problems on modern multiple cpu with multiple cores servers. I wonder
if there is some kind of formula to know what total should not be exceeded
in this particular case, of if the good old document by Mladen Turk is still
valid fixing this number to 2000
(http://people.apache.org/~mturk/docs/article/ftwai.html)
Thanks a lot for your help,
br1.
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19787259.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Rainer Jung <ra...@kippdata.de>.
br1 wrote:
> Rainer Jung-3 wrote:
>> If so, what is your connection pool size? The default is 250, which is
>> quite high.
>>
> I did not go too far, it was the default, after some time I set it to 300
> but this did not change anything.
That might be the reason for trouble. If things start to get slow, the
web server gets filled by all thenew requests still coming in without
answering fast enough the existing ones. People often try to work around
this by increasing the number of threads available for processing the
requests, but if the backend is stuck or too slow, this will not help
and instead also trash the frontend.
You should measure how many connections you need during peak hours
without performance problems, and then add some percentage depending on
your growth rate etc. Finally you need to make sure, that your web
server itself uses at least as many threads, as you configured as the
pool size.
For Apache with mod_jk we can automatically detect the number of threads
and by default size our pool to the same number. For IIS you have to
size the pool yourself (or live with the big default of 250).
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Hi Rainer,
Rainer Jung-3 wrote:
>
> Yes it does limit it that way. But when a timeout fires in mod_jk,
> Tomcat is not aware of it and will still proceed working on the request.
> Your web server though receives new rquests and forwards them, so Tomcat
> could run out of memory.
>
If there was a way to set the maximum request processing time on Tomcat..
you can do it in ASP.. :-)
Rainer Jung-3 wrote:
>
> Are you saying, that you can't get any content from IIS, even not
> content, that does not involve Tomcat to be returned?
>
Exactly, the whole site hangs. Everything stops working, with no exceptions.
Local static and dynamic content, all remote Tomcat workers, other remote
stuff in php we have, everything.
No error in the event log, the IIS processes just stay there idle.
Rainer Jung-3 wrote:
>
> If so, what is your connection pool size? The default is 250, which is
> quite high.
>
I did not go too far, it was the default, after some time I set it to 300
but this did not change anything.
Rainer Jung-3 wrote:
>
> Thread Dumps only halt the JVM for a couple of milliseconds. No need to
> be afraid of them, even in serious production. Don't mix the notions of
> thread dump and memory dump though. The latter takes quite long, but is
> not what you need here.
>
Yes, but I am just afraid of my site hanging as I was explaining above. :-)
Thank you,
br1.
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19775890.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Rainer Jung <ra...@kippdata.de>.
> Rainer Jung-3 wrote:
>>> My main question here is: is it normal that that a faulty application
>>> takes
>>> down the whole site?
>> Yes, unfortunately that's a common source for trouble. If an app doesn't
>> finish processinf a request, e.g. because it wais without timeout for
>> some other service, or runs in a loop, or has a deadlock, then each
>> broken request will block a thread until finally - maybe after weeks -
>> all threads are blocked in the buggy requests of the single app.
>>
> Again, not so common for me, nor - sorry - for what I found on the whole
> list so far.
> So, one should not think that setting a worker's connection_pool_size to x
> would limit the web server to use no more than x threads for that worker?
> This was happening with peaks of every two hours or so, but I guess it
> depends on the site traffic.
Yes it does limit it that way. But when a timeout fires in mod_jk,
Tomcat is not aware of it and will still proceed working on the request.
Your web server though receives new rquests and forwards them, so Tomcat
could run out of memory.
Are you saying, that you can't get any content from IIS, even not
content, that does not involve Tomcat to be returned?
If so, what is your connection pool size? The default is 250, which is
quite high.
> Rainer Jung-3 wrote:
>> Beware, that the timeouts will shield IIS, but if the Tomcat can't
>> handle more requests, the timeouts in the IIS plugin will not heal Tomcat.
>>
> Thank you for telling me. It sounds fair some way.. and it's definitely
> what is needed to take a thread dump without killing everything..
Thread Dumps only halt the JVM for a couple of milliseconds. No need to
be afraid of them, even in serious production. Don't mix the notions of
thread dump and memory dump though. The latter takes quite long, but is
not what you need here.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Rainer,
Good evening.
Rainer Jung-3 wrote:
>
>> My main question here is: is it normal that that a faulty application
>> takes
>> down the whole site?
>
> Yes, unfortunately that's a common source for trouble. If an app doesn't
> finish processinf a request, e.g. because it wais without timeout for
> some other service, or runs in a loop, or has a deadlock, then each
> broken request will block a thread until finally - maybe after weeks -
> all threads are blocked in the buggy requests of the single app.
>
Again, not so common for me, nor - sorry - for what I found on the whole
list so far.
So, one should not think that setting a worker's connection_pool_size to x
would limit the web server to use no more than x threads for that worker?
This was happening with peaks of every two hours or so, but I guess it
depends on the site traffic.
Rainer Jung-3 wrote:
>
> The Thread Dumps are an exciting way of bridging the gap between the
> operations people and the developers. Operations people can provide it,
> and developers can read it. Actually it is not that hard to read for
> operations people too.
>
Thanks. I will let you know how the specific developer and operation person
will react (though I doubt we will ever give a Thread Dump party), but again
this is not the main priority at the moment. Also, I would not be happy to
let the site hang again to take a thread dump - though I could fix the
timeouts first maybe as you say below, and avoid another downtime.
Rainer Jung-3 wrote:
>
> Beware, that the timeouts will shield IIS, but if the Tomcat can't
> handle more requests, the timeouts in the IIS plugin will not heal Tomcat.
>
Thank you for telling me. It sounds fair some way.. and it's definitely
what is needed to take a thread dump without killing everything..
Rainer Jung-3 wrote:
>
>> BTW, thanks for writing the timeouts page, it was really needed. :-)
>
> Yes it was, sorry that it took that long.
>
It's okay, just a wrong timeout.. it happens to the most of us here. :-)
Thanks!
Br1
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19770072.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by Rainer Jung <ra...@kippdata.de>.
br1 schrieb:
> Rainer,
>
> Thank you for your reply.
>
> My main question here is: is it normal that that a faulty application takes
> down the whole site?
Yes, unfortunately that's a common source for trouble. If an app doesn't
finish processinf a request, e.g. because it wais without timeout for
some other service, or runs in a loop, or has a deadlock, then each
broken request will block a thread until finally - maybe after weeks -
all threads are blocked in the buggy requests of the single app.
>> 1) Finding the real reason: Assuming some requests send to your Tomcat
>> simply get stuck and do not return fast enough, you'll need to take
>> thread dumps of Tomcat and look at those. Take the dumps when the
>> problem is there and take more than one, e.g. 3 dumps each 3 seconds
>> apart from the previous one.
>>
>
> Thanks. I will give this a try, though my main priority is to have the site
> working, not the single application. I suspect there are some problem
> releasing the db connections, but I am not the developer nor a
> modern-Java-with-tons-of-frameworks developer. :-)
The Thread Dumps are an exciting way of bridging the gap between the
operations people and the developers. Operations people can provide it,
and developers can read it. Actually it is not that hard to read for
operations people too.
>> 2) Making IIS/mod_jk more robust against the problem: Use timeouts as
>> described on page
>>
>> http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
>>
>> Those will not make your app work, when in fact requests in Tomcat get
>> stuck and block all available connector threads, but it will at least
>> keep iis/mod_jk running.
>>
>
> Actually, I was about to try this when I found that restarting Tomcat cured
> the problem.
> I will try with the suggested timeouts as soon as everyone stops watching.
>
> As I was writing above, I am stumped to see a single app stopping the web
> server, because we have a zero timeout on all Tomcats and apps since years,
> and no problems like this.
> I myself thought this was just not possible, so it took long time to figure
> this out.
Beware, that the timeouts will shield IIS, but if the Tomcat can't
handle more requests, the timeouts in the IIS plugin will not heal Tomcat.
> BTW, thanks for writing the timeouts page, it was really needed. :-)
Yes it was, sorry that it took that long.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK and IIS - troubles?
Posted by br1 <my...@myrealbox.com>.
Rainer,
Thank you for your reply.
My main question here is: is it normal that that a faulty application takes
down the whole site?
I have seen applications with this problem several times, and all the 40ish
Tomcats we have are configured with connectionTimeout set to zero - and
connection_pool_timeout as well on the JK side - but this never hung the
whole site so far (and I am talking about eight or nine years, if not more).
Rainer Jung-3 wrote:
>
>> JK 1.2.26 (/maybe some other 1.2.2x version but cannot be sure)
>
> You and we need to be sure. Set you JK log level to at least info. All
>
Apologies, I was definitely not clear.
The thing is happening with JK version 1.2.26, and I remember having a
problem with some 1.2.2x version - but I cannot tell for sure now if it was
the same problem because I rapidly upgraded.
> Tomcat 5.5.9 / 5.5.27
> JDK 1.5.0_12 / 1.5.0_16 / 1.6.0_07
And here, that I experienced the same with the above versions and with the
same application. It is currently running with 5.5.27 on 1.6.0_07.
Rainer Jung-3 wrote:
>
> 1) Finding the real reason: Assuming some requests send to your Tomcat
> simply get stuck and do not return fast enough, you'll need to take
> thread dumps of Tomcat and look at those. Take the dumps when the
> problem is there and take more than one, e.g. 3 dumps each 3 seconds
> apart from the previous one.
>
Thanks. I will give this a try, though my main priority is to have the site
working, not the single application. I suspect there are some problem
releasing the db connections, but I am not the developer nor a
modern-Java-with-tons-of-frameworks developer. :-)
Rainer Jung-3 wrote:
>
> 2) Making IIS/mod_jk more robust against the problem: Use timeouts as
> described on page
>
> http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
>
> Those will not make your app work, when in fact requests in Tomcat get
> stuck and block all available connector threads, but it will at least
> keep iis/mod_jk running.
>
Actually, I was about to try this when I found that restarting Tomcat cured
the problem.
I will try with the suggested timeouts as soon as everyone stops watching.
As I was writing above, I am stumped to see a single app stopping the web
server, because we have a zero timeout on all Tomcats and apps since years,
and no problems like this.
I myself thought this was just not possible, so it took long time to figure
this out.
BTW, thanks for writing the timeouts page, it was really needed. :-)
Br1.
--
View this message in context: http://www.nabble.com/JK-and-IIS---troubles--tp19750760p19768252.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org