You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kobi Biton <ko...@comns.co.il> on 2011/01/06 17:26:59 UTC
Tomcat 6 busy threads issue
Hello Experts,
Recently I have exposed my tomcat 6 instances via JMX for monitoring I
noticed that 1 of the servers is using twice the bust thread count then
all others , then I noticed a configuration diff the server had 300 as
maxThreads while all other had 1000 as MaxThreads After changing it all
came back to normal can someone explain me what is the correlation
between maxThreads count to busyThreads ?
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 busy threads issue
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Kobi,
I can't really help if you don't give me more information
On 1/8/2011 11:36 AM, Kobi Biton wrote:
> On Thu, 2011-01-06 at 12:23 -0500, Christopher Schultz wrote:
>> Okay, that will mean that the second server will handle, at most, 300
>> simultaneous connections. Do you have any per-backend-server connection
>> limits set at the load balancer?
??
>> So, let's do an example so I can get my head around it:
>>
>> N = 3
>>
>> Server 1: maxThreads=1000, busy=100
>> Server 2: maxThreads=1000, busy=100
>> Server 3: maxThreads=300, busy=200
>>
>> Is that roughly what you observed?
> - Yes 100% Right
Good.
>> Was that consistent over a long period of time?
> - Yep consistent since tomcat start
Ok.
>> Did you observe what kinds of requests were being served
>> on server 3? If some of them were long-running, you may just be
>> observing a natural build-up of work on a server that reliably gets 33%
>> of incoming requests but has some older requests that are still processing.
??
>> What kind of lb are you using?
> Linux ipvsadm basically LVS
Are you sure it's configured properly? Posting the details of your
configuration would go a long way towards solving your problem.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk0rWrUACgkQ9CaO5/Lv0PBs6QCcCejBSgmF2iyyiGaZsefJB94e
snIAnAqiKJjZt7ktEagyaZl3FfYsHWc2
=ixry
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 busy threads issue
Posted by Kobi Biton <ko...@comns.co.il>.
Christopher,
On Thu, 2011-01-06 at 12:23 -0500, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Kobi,
>
> On 1/6/2011 12:14 PM, Kobi Biton wrote:
> > -Tomcat 6.0.22
>
> Great. Any opportunities to upgrade? Your version was never released and
> is over a year old.
- Yep I agree and I believe that I will upgrade soon.
>
> > - yes they are load balanced simple Round Robin Linux LVS , (LB
> > distributes the requests evenly)
>
> So, you expect each server to serve 1/N of the incoming requests per
> unit time. Any session stickiness?
>
> > - All the servers (But 1) had the following thread pool confguration
> > prior the change:
> >
> > MaxThreads=1000 , minSpareThread=25,maxSpateThreads=75
> >
> > 1 server had:
> >
> > MaxThreads=300 , minSpareThread=25,maxSpateThreads=75
>
> Okay, that will mean that the second server will handle, at most, 300
> simultaneous connections. Do you have any per-backend-server connection
> limits set at the load balancer?
>
> > For some reason the server that had MaxThreads=300 report x2
> > threadCount / threadBusy Via JMX polling , only after changing its
> > maxThreads to 1000 the threadCount/threadBusy count was decreased x2 and
> > matched all the other servers
>
> So, let's do an example so I can get my head around it:
>
> N = 3
>
> Server 1: maxThreads=1000, busy=100
> Server 2: maxThreads=1000, busy=100
> Server 3: maxThreads=300, busy=200
>
> Is that roughly what you observed?
- Yes 100% Right
> Was that consistent over a long
> period of time?
- Yep consistent since tomcat start
> Did you observe what kinds of requests were being served
> on server 3? If some of them were long-running, you may just be
> observing a natural build-up of work on a server that reliably gets 33%
> of incoming requests but has some older requests that are still processing.
>
> > - Indeed busyThreads <= maxThreads
>
> Good. If that weren't true, something would be terribly wrong :)
>
> > Just trying to figure out why did the maxThreads was Impacting the
> > thread busy/count ...
>
> I can't imagine why it would, other than putting a cap on the number of
> busy threads (because busyThreads <= maxThreads).
>
> What kind of lb are you using?
Linux ipvsadm basically LVS
>
> - -chris
Kobi.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk0l+pYACgkQ9CaO5/Lv0PBuogCgitvm9D01NSi84oO0tVO+itdn
> ByMAmwfOVpq59BhVIlUtx/EswxsBm3lJ
> =AzZU
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Kobi Biton
Com N S Ltd.
Mobile: +972 (54) 8017668
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 busy threads issue
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Kobi,
On 1/6/2011 12:14 PM, Kobi Biton wrote:
> -Tomcat 6.0.22
Great. Any opportunities to upgrade? Your version was never released and
is over a year old.
> - yes they are load balanced simple Round Robin Linux LVS , (LB
> distributes the requests evenly)
So, you expect each server to serve 1/N of the incoming requests per
unit time. Any session stickiness?
> - All the servers (But 1) had the following thread pool confguration
> prior the change:
>
> MaxThreads=1000 , minSpareThread=25,maxSpateThreads=75
>
> 1 server had:
>
> MaxThreads=300 , minSpareThread=25,maxSpateThreads=75
Okay, that will mean that the second server will handle, at most, 300
simultaneous connections. Do you have any per-backend-server connection
limits set at the load balancer?
> For some reason the server that had MaxThreads=300 report x2
> threadCount / threadBusy Via JMX polling , only after changing its
> maxThreads to 1000 the threadCount/threadBusy count was decreased x2 and
> matched all the other servers
So, let's do an example so I can get my head around it:
N = 3
Server 1: maxThreads=1000, busy=100
Server 2: maxThreads=1000, busy=100
Server 3: maxThreads=300, busy=200
Is that roughly what you observed? Was that consistent over a long
period of time? Did you observe what kinds of requests were being served
on server 3? If some of them were long-running, you may just be
observing a natural build-up of work on a server that reliably gets 33%
of incoming requests but has some older requests that are still processing.
> - Indeed busyThreads <= maxThreads
Good. If that weren't true, something would be terribly wrong :)
> Just trying to figure out why did the maxThreads was Impacting the
> thread busy/count ...
I can't imagine why it would, other than putting a cap on the number of
busy threads (because busyThreads <= maxThreads).
What kind of lb are you using?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk0l+pYACgkQ9CaO5/Lv0PBuogCgitvm9D01NSi84oO0tVO+itdn
ByMAmwfOVpq59BhVIlUtx/EswxsBm3lJ
=AzZU
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 busy threads issue
Posted by Kobi Biton <ko...@comns.co.il>.
Hi Christopher thanks for the reply.
-Tomcat 6.0.22
- yes they are load balanced simple Round Robin Linux LVS , (LB
distributes the requests evenly)
- All the servers (But 1) had the following thread pool confguration
prior the change:
MaxThreads=1000 , minSpareThread=25,maxSpateThreads=75
1 server had:
MaxThreads=300 , minSpareThread=25,maxSpateThreads=75
For some reason the server that had MaxThreads=300 report x2
threadCount / threadBusy Via JMX polling , only after changing its
maxThreads to 1000 the threadCount/threadBusy count was decreased x2 and
matched all the other servers
- Indeed busyThreads <= maxThreads
Just trying to figure out why did the maxThreads was Impacting the
threasd busy/count ...
Kobi
On Thu, 2011-01-06 at 12:05 -0500, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Kobi,
>
> On 1/6/2011 11:26 AM, Kobi Biton wrote:
> > Recently I have exposed my tomcat 6
>
> 6.what?
>
> > instances via JMX for monitoring I noticed that 1 of the servers is
> > using twice the bust thread count then all others
>
> Are they load-balanced in any way? That's good information to have. You
> meant "busy" threads, right?
>
> > then I noticed a configuration diff the server had 300 as maxThreads
> > while all other had 1000 as MaxThreads After changing it all came
> > back to normal
>
> What did you change and what is "normal"?
>
> > can someone explain me what is the correlation between maxThreads
> > count to busyThreads ?
>
> Presumably, busyThreads <= maxThreads.
>
> Anything else?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk0l9m4ACgkQ9CaO5/Lv0PDz5ACfZhVT+MFoIAIgbxVkpIiXpyPI
> lIYAn0BBA4K6uK32/0JjuuTGR5/M3pUl
> =sFxS
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Kobi Biton
Com N S Ltd.
Mobile: +972 (54) 8017668
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6 busy threads issue
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Kobi,
On 1/6/2011 11:26 AM, Kobi Biton wrote:
> Recently I have exposed my tomcat 6
6.what?
> instances via JMX for monitoring I noticed that 1 of the servers is
> using twice the bust thread count then all others
Are they load-balanced in any way? That's good information to have. You
meant "busy" threads, right?
> then I noticed a configuration diff the server had 300 as maxThreads
> while all other had 1000 as MaxThreads After changing it all came
> back to normal
What did you change and what is "normal"?
> can someone explain me what is the correlation between maxThreads
> count to busyThreads ?
Presumably, busyThreads <= maxThreads.
Anything else?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk0l9m4ACgkQ9CaO5/Lv0PDz5ACfZhVT+MFoIAIgbxVkpIiXpyPI
lIYAn0BBA4K6uK32/0JjuuTGR5/M3pUl
=sFxS
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org