You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Suvendu Sekhar Mondal <su...@gmail.com> on 2020/11/11 11:40:52 UTC
HTTP thread pool size remained high after database failover
Hello Everyone,
During database failover resiliency testing we noticed something
unusual which we still cannot explain. So, reaching out to the
community for help.
Application Setup:
AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016
Observations:
After database failover, we noticed that HTTP thread pool size jumped
from 10 to 50. No. of busy threads jumped to 40 for a minute and then
came down to normal range(1-5). But thread pool size remained 50 for
the next one hour. When we collected the heap dump after one hour,
thread pool size jumped again to ~90. Thread dump analysis is showing
most of the HTTP threads were just waiting to get some work[1]. That
looked surprising to me. Few of them are runnable and doing some work
- which matches "busy thread" count. During all this timeframe load on
the system remained the same.
We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
do not have any keepAliveTimeout set in Tomcat. App uses session
affinity. We were looking for some connector properties which deal
with idle threads but found none[3].
We are not seeing any "problem" in our app after DB failover but this
is something also newer to us and we are trying to explain it.
Tomcat's memory usage increased by a few MB as new threads were
spawned. My expectation was that if threads in the thread pool are not
getting used and idle, after sometime thread pool size will reduce to
"minSpareThreads" size which is 10 in our case. Is that expectation
wrong? Can you please share your thoughts?
[1]
"http-apr-18110-exec-63" #2170 daemon prio=5 os_prio=0
tid=0x0000000045f03000 nid=0x30d4 waiting on condition
[0x000000005a4cf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006a5400178> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
[2]
<Connector port="18110" protocol="HTTP/1.1" maxThreads="500"
acceptCount="300" URIEncoding="UTF-8" redirectPort="18113"
connectionTimeout="30000"/>
<Connector port="18113" threadPriority="6" compression="force"
compressionMinSize="2048"
compressableMimeTypes="text/html,text/xml,text/plain,application/json"
sslProtocol="TLS" secure="true" scheme="https" protocol="HTTP/1.1"
maxThreads="150" maxSavePostSize="-1" maxHttpHeaderSize="8192"
keystoreFile="conf/.keystore" enableLookups="false"
disableUploadTimeout="true" clientAuth="false" acceptCount="100"
SSLPassword="" SSLEnabled="true" SSLCertificateKeyFile="\app\my.key"
SSLCertificateFile="\app\mycert.cer"/>
[3]
https://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Thanks & Regards,
Suvendu
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hi Michael,
On Wed, Nov 11, 2020 at 6:03 PM Michael Osipov <mi...@apache.org> wrote:
>
> Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal:
> > Hello Everyone,
> >
> > During database failover resiliency testing we noticed something
> > unusual which we still cannot explain. So, reaching out to the
> > community for help.
> >
> > Application Setup:
> > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016
>
> Almost of them them are outdated, you should upgrade first.
I know. We will be upgrading them in the next release. For this
release we have to use aforementioned softwares.
For the last three releases we used the same softwares but had not
noticed this issue - maybe we were not looking that closely. Now, we
are seeing this and can't let it go without explaining it. :)
So, any thoughts on this?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Posted by Michael Osipov <mi...@apache.org>.
Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal:
> Hello Everyone,
>
> During database failover resiliency testing we noticed something
> unusual which we still cannot explain. So, reaching out to the
> community for help.
>
> Application Setup:
> AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016
Almost of them them are outdated, you should upgrade first.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Thank you Chris for your response.
On Wed, Nov 11, 2020 at 10:02 PM Christopher Schultz
<ch...@christopherschultz.net> wrote:
>
> Suvendu,
>
> On 11/11/20 06:40, Suvendu Sekhar Mondal wrote:
> > Application Setup:
> > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016
>
> As Michael pointed-out, these all need to be upgraded, but they are not
> affecting your experience, here.
>
> > Observations:
> > After database failover, we noticed that HTTP thread pool size jumped
> > from 10 to 50. No. of busy threads jumped to 40 for a minute and then
> > came down to normal range(1-5). But thread pool size remained 50 for
> > the next one hour. When we collected the heap dump after one hour,
> > thread pool size jumped again to ~90. Thread dump analysis is showing
> > most of the HTTP threads were just waiting to get some work[1]. That
> > looked surprising to me. Few of them are runnable and doing some work
> > - which matches "busy thread" count. During all this timeframe load on
> > the system remained the same.
> >
> > We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
> > do not have any keepAliveTimeout set in Tomcat. App uses session
> > affinity. We were looking for some connector properties which deal
> > with idle threads but found none[3].
>
> You won't find them.
>
Okay.
> > We are not seeing any "problem" in our app after DB failover but this
> > is something also newer to us and we are trying to explain it.
>
> What are you trying to explain?
>
We are simply trying to understand this behavior as this is something
new we noticed.
> > Tomcat's memory usage increased by a few MB as new threads were
> > spawned. My expectation was that if threads in the thread pool are not
> > getting used and idle, after sometime thread pool size will reduce to
> > "minSpareThreads" size which is 10 in our case. Is that expectation
> > wrong? Can you please share your thoughts?
>
> This is a common misconception. If you do not configure an "executor"
> then Tomcat will create one for you and it will NOT remove idle threads
> from the thread pool.
>
> The solution is to manually-specify an Executor:
>
> <Executor name="myExecutor" namePrefix="myExecutor-" maxThreads="X"
> minSpareThreads="Y" />
>
> <Connector executor="myExecutor" ... />
>
> With this configuration, you should see your thread count stay between Y
> and X, but when X are not necessary, your thread count should drop
> towards Y as appropriate.
>
Thanks for the explanation. This is the kind of information I was
looking for in Tomcat docs but was not able to locate it. We will try
this approach and let you know the outcome.
> Hope that helps,
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Suvendu,
On 11/11/20 06:40, Suvendu Sekhar Mondal wrote:
> Application Setup:
> AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016
As Michael pointed-out, these all need to be upgraded, but they are not
affecting your experience, here.
> Observations:
> After database failover, we noticed that HTTP thread pool size jumped
> from 10 to 50. No. of busy threads jumped to 40 for a minute and then
> came down to normal range(1-5). But thread pool size remained 50 for
> the next one hour. When we collected the heap dump after one hour,
> thread pool size jumped again to ~90. Thread dump analysis is showing
> most of the HTTP threads were just waiting to get some work[1]. That
> looked surprising to me. Few of them are runnable and doing some work
> - which matches "busy thread" count. During all this timeframe load on
> the system remained the same.
>
> We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
> do not have any keepAliveTimeout set in Tomcat. App uses session
> affinity. We were looking for some connector properties which deal
> with idle threads but found none[3].
You won't find them.
> We are not seeing any "problem" in our app after DB failover but this
> is something also newer to us and we are trying to explain it.
What are you trying to explain?
> Tomcat's memory usage increased by a few MB as new threads were
> spawned. My expectation was that if threads in the thread pool are not
> getting used and idle, after sometime thread pool size will reduce to
> "minSpareThreads" size which is 10 in our case. Is that expectation
> wrong? Can you please share your thoughts?
This is a common misconception. If you do not configure an "executor"
then Tomcat will create one for you and it will NOT remove idle threads
from the thread pool.
The solution is to manually-specify an Executor:
<Executor name="myExecutor" namePrefix="myExecutor-" maxThreads="X"
minSpareThreads="Y" />
<Connector executor="myExecutor" ... />
With this configuration, you should see your thread count stay between Y
and X, but when X are not necessary, your thread count should drop
towards Y as appropriate.
Hope that helps,
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org