You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by William Crowell <WC...@perforce.com.INVALID> on 2025/03/24 18:56:39 UTC
NIO Thread Madness
Hi,
I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
I have my thread pool defined as follows in server.xml:
…
<Executor name="tomcatThreadPool"
namePrefix="catalina-exec-"
maxThreads="1000"
minSpareThreads="50"
maxIdleTime="60000"
maxQueueSize="1000" />
<Connector port="8080"
protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="130000"
redirectPort="8443"
disableUploadTimeout="false"
acceptCount="1000"
maxConnections="1000"
executor="tomcatThreadPool" />
…
Are there any logs I can enable to find out why the application server stops accepting connections?
Regards,
William Crowell
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Christopher Schultz <ch...@christopherschultz.net>.
William,
On 3/31/25 3:59 PM, William Crowell wrote:
> Oracle’s ojdbc8.jar version 19.25.0.0.0 which should support JDBC 4.2:
They I don't think you are experiencing what was reported in DBCP-599.
-chris
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Monday, March 31, 2025 at 3:50 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/31/25 2:31 PM, William Crowell wrote:
>> Question related to this. I found issue DBCP-599 which was released
>> in DBCP 2.13.0 as part of Apache Tomcat 9.0.98 release. The
>> characteristics of this major bug appear very similar to the issue I am
>> having.
>
> Are you using an old driver that does not support JDBC 4.2?
>
>> In Tomcat’s lib directory there is a tomcat-dbcp.jar. Is that just
>> DBCP2 repackaged under Tomcat or is that an interface from Tomcat
>> into Apache Commons DBCP2?
>
> It's re-packaged.
>
> -chris
>
>> From: William Crowell <WC...@perforce.com.INVALID>
>> Date: Friday, March 28, 2025 at 12:22 PM
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> Very good idea Chriis. Thank you.
>>
>> Regards,
>>
>> William Crowell
>>
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Date: Friday, March 28, 2025 at 12:05 PM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> William,
>>
>> On 3/26/25 7:06 PM, William Crowell wrote:
>>> That maxTotal was a typo due to trying to copy the config from a
>>> screenshot.
>>>
>>> I agree with your assessment. I think there are 2 situations going
>>> on. One is that the code may not be properly closing connections,
>>> and the connection pool is not properly configured.
>> I wrote this a looong time ago, but it's very helpful to show to
>> programmers who haven't been careful.
>>
>> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303279804%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n52OlWTJYXuUvy%2FPepZNxRtHjWOQbQlmo6zskRW28JU%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303299823%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fIYe4E6WYeBXonauF%2B3mUeQs9EnIG5NeB8ovsiXZais%3D&reserved=0><https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303310799%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=h6TJBfYtU3ZA4MY5cyPE%2BOl4KitCpIPCUhQo9P0VJDM%3D&reserved=0><https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/>
>>
>> For your long-transactions, I think you are going to want to use a
>> separate pool.
>>
>> -chris
>>
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Date: Wednesday, March 26, 2025 at 6:57 PM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> William,
>>>
>>> On 3/25/25 2:51 PM, William Crowell wrote:
>>>> Mark,
>>>>
>>>> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>>>>
>>>> …
>>>> accessToUnderlyingConnectionAllowed=”true”
>>>> maxIdle=”100”
>>>> maxWaitMillis=”10000”
>>>> minIdle=”25”
>>>> validationQuery=”select 1 from dual”
>>>> url=”blah blah blah”
>>>> maxTotal=”true”
>>>
>>> ^ This is typically a number, not a true/false value.
>>>
>>>> logAbandoned=”true”
>>>> removeAbandonedOnBorrow=”true”
>>>> removeAbandonedTimeout=”900”
>>>> removeAbandonedOnMaintenance=”true”
>>>> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
>>>> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>>>>
>>>> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>>>>
>>>> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
>>>
>>> When the database pool dries up, your application will basically stop.
>>> Setting removeAbandoned as you have is good, but with those long long
>>> timeouts, they aren't doing any good.
>>>
>>> I might set up a connection pool for "short queries" (like logging-in,
>>> etc.) and then set up a separate pool for much longer queries.
>>>
>>> Also, you might want to review your use of connections, etc. to ensure
>>> that you are always closing everything in finally blocks. Resource
>>> management with JDBC can be tedious and if you don't do it right, you
>>> can leak connections from your pool very easily.
>>>
>>> -chris
>>>
>>>> From: Mark Thomas <ma...@apache.org>
>>>> Date: Tuesday, March 25, 2025 at 1:13 PM
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: NIO Thread Madness
>>>> On 25/03/2025 12:33, William Crowell wrote:
>>>>> Mark,
>>>>>
>>>>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>>>>
>>>> Hmm. Both the Acceptor thread and the Poller thread are running and
>>>> appear to be in states that would enable new requests to be processed.
>>>>
>>>> There isn't logging in those two components for normal operations. I
>>>> assume because of performance concerns.
>>>>
>>>> Do you have any thread dumps from when you have clients attempting new
>>>> requests that are hanging? I'm wondering if processing is hanging
>>>> somewhere in Tomcat / the application during request processing.
>>>>
>>>> I would probably be thinking about enabling remote debugging and
>>>> connecting a debugger the next time it goes wrong and tracing the
>>>> progress of a new request. But I accept that may not be practical for a
>>>> production system.
>>>>
>>>> It seems unlikely that the new requests are hanging before they reach
>>>> the Tomcat code but that seems unlikely.
>>>>
>>>> Other things to try:
>>>>
>>>> netstat may shed some light on the current state of Tomcat and database
>>>> connections.
>>>>
>>>> Wireshark or similar on the client side might also provide some clues.
>>>>
>>>> I am wondering if Tomcat / Windows is responding at all, dropping the
>>>> connection, sending a reset, etc.
>>>>
>>>> There might be some clues in the exceptions the exceptions that occur
>>>> just before the problem starts. Can you share any of them?
>>>>
>>>> Mark
>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> William Crowell
>>>>>
>>>>> From: Mark Thomas <ma...@apache.org>
>>>>> Date: Tuesday, March 25, 2025 at 8:27 AM
>>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>>> Subject: Re: NIO Thread Madness
>>>>> On 25/03/2025 11:24, William Crowell wrote:
>>>>>> Chris,
>>>>>>
>>>>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>>>>
>>>>> Looking that the thread dump you sent me privately now.
>>>>>
>>>>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>>>>
>>>>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>>>>
>>>>> Mark
>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> William Crowell
>>>>>>
>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>>>> Subject: Re: NIO Thread Madness
>>>>>> William,
>>>>>>
>>>>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>>>>
>>>>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>>>>
>>>>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>>>>
>>>>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>>>>
>>>>>> Are you able to use JMX or similar to see how many used database
>>>>>> connections you have?
>>>>>>
>>>>>> When you say that Tomcat does not accept requests, do you mean that you
>>>>>> make a request and it never returns, or do you mean that you get a
>>>>>> "connection refused" or something similar?
>>>>>>
>>>>>>> I have my thread pool defined as follows in server.xml:
>>>>>>>
>>>>>>> …
>>>>>>> <Executor name="tomcatThreadPool"
>>>>>>> namePrefix="catalina-exec-"
>>>>>>> maxThreads="1000"
>>>>>>> minSpareThreads="50"
>>>>>>> maxIdleTime="60000"
>>>>>>> maxQueueSize="1000" />
>>>>>>>
>>>>>>> <Connector port="8080"
>>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>>> connectionTimeout="130000"
>>>>>>> redirectPort="8443"
>>>>>>> disableUploadTimeout="false"
>>>>>>> acceptCount="1000"
>>>>>>> maxConnections="1000"
>>>>>>> executor="tomcatThreadPool" />
>>>>>>
>>>>>> Have you tried enabling any of the "abandoned"-related settings? I
>>>>>> highly recommend logAbandoned="true" and possibly others.
>>>>>>
>>>>>> I would imagine you'd see threads waiting on database connections if you
>>>>>> had exhausted your pool. Just remember that threads can be stuck on
>>>>>> things without being BLOCKED.
>>>>>>
>>>>>> > Are there any logs I can enable to find out why the application
>>>>>> server stops accepting connections?
>>>>>>
>>>>>> -chris
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>>>
>>>>>>
>>>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>>
>>>>>
>>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>
>>>>
>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
> Т���������������������������������������������������������������������ХF�V�7V'67&�&R�R���âW6W'2�V�7V'67&�&TF��6B�6�R��&pФf�"FF�F����6����G2�R���âW6W'2ֆV�F��6B�6�R��&pР
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Chris,
Oracle’s ojdbc8.jar version 19.25.0.0.0 which should support JDBC 4.2:
https://www.oracle.com/database/technologies/maven-central-guide.html
Regards,
William Crowell
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Monday, March 31, 2025 at 3:50 PM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
William,
On 3/31/25 2:31 PM, William Crowell wrote:
> Question related to this. I found issue DBCP-599 which was released
> in DBCP 2.13.0 as part of Apache Tomcat 9.0.98 release. The
> characteristics of this major bug appear very similar to the issue I am
> having.
Are you using an old driver that does not support JDBC 4.2?
> In Tomcat’s lib directory there is a tomcat-dbcp.jar. Is that just
> DBCP2 repackaged under Tomcat or is that an interface from Tomcat
> into Apache Commons DBCP2?
It's re-packaged.
-chris
> From: William Crowell <WC...@perforce.com.INVALID>
> Date: Friday, March 28, 2025 at 12:22 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> Very good idea Chriis. Thank you.
>
> Regards,
>
> William Crowell
>
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Friday, March 28, 2025 at 12:05 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/26/25 7:06 PM, William Crowell wrote:
>> That maxTotal was a typo due to trying to copy the config from a
>> screenshot.
>>
>> I agree with your assessment. I think there are 2 situations going
>> on. One is that the code may not be properly closing connections,
>> and the connection pool is not properly configured.
> I wrote this a looong time ago, but it's very helpful to show to
> programmers who haven't been careful.
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303279804%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n52OlWTJYXuUvy%2FPepZNxRtHjWOQbQlmo6zskRW28JU%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303299823%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fIYe4E6WYeBXonauF%2B3mUeQs9EnIG5NeB8ovsiXZais%3D&reserved=0><https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303310799%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=h6TJBfYtU3ZA4MY5cyPE%2BOl4KitCpIPCUhQo9P0VJDM%3D&reserved=0><https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/>
>
> For your long-transactions, I think you are going to want to use a
> separate pool.
>
> -chris
>
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Date: Wednesday, March 26, 2025 at 6:57 PM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> William,
>>
>> On 3/25/25 2:51 PM, William Crowell wrote:
>>> Mark,
>>>
>>> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>>>
>>> …
>>> accessToUnderlyingConnectionAllowed=”true”
>>> maxIdle=”100”
>>> maxWaitMillis=”10000”
>>> minIdle=”25”
>>> validationQuery=”select 1 from dual”
>>> url=”blah blah blah”
>>> maxTotal=”true”
>>
>> ^ This is typically a number, not a true/false value.
>>
>>> logAbandoned=”true”
>>> removeAbandonedOnBorrow=”true”
>>> removeAbandonedTimeout=”900”
>>> removeAbandonedOnMaintenance=”true”
>>> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
>>> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>>>
>>> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>>>
>>> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
>>
>> When the database pool dries up, your application will basically stop.
>> Setting removeAbandoned as you have is good, but with those long long
>> timeouts, they aren't doing any good.
>>
>> I might set up a connection pool for "short queries" (like logging-in,
>> etc.) and then set up a separate pool for much longer queries.
>>
>> Also, you might want to review your use of connections, etc. to ensure
>> that you are always closing everything in finally blocks. Resource
>> management with JDBC can be tedious and if you don't do it right, you
>> can leak connections from your pool very easily.
>>
>> -chris
>>
>>> From: Mark Thomas <ma...@apache.org>
>>> Date: Tuesday, March 25, 2025 at 1:13 PM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> On 25/03/2025 12:33, William Crowell wrote:
>>>> Mark,
>>>>
>>>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>>>
>>> Hmm. Both the Acceptor thread and the Poller thread are running and
>>> appear to be in states that would enable new requests to be processed.
>>>
>>> There isn't logging in those two components for normal operations. I
>>> assume because of performance concerns.
>>>
>>> Do you have any thread dumps from when you have clients attempting new
>>> requests that are hanging? I'm wondering if processing is hanging
>>> somewhere in Tomcat / the application during request processing.
>>>
>>> I would probably be thinking about enabling remote debugging and
>>> connecting a debugger the next time it goes wrong and tracing the
>>> progress of a new request. But I accept that may not be practical for a
>>> production system.
>>>
>>> It seems unlikely that the new requests are hanging before they reach
>>> the Tomcat code but that seems unlikely.
>>>
>>> Other things to try:
>>>
>>> netstat may shed some light on the current state of Tomcat and database
>>> connections.
>>>
>>> Wireshark or similar on the client side might also provide some clues.
>>>
>>> I am wondering if Tomcat / Windows is responding at all, dropping the
>>> connection, sending a reset, etc.
>>>
>>> There might be some clues in the exceptions the exceptions that occur
>>> just before the problem starts. Can you share any of them?
>>>
>>> Mark
>>>
>>>>
>>>> Regards,
>>>>
>>>> William Crowell
>>>>
>>>> From: Mark Thomas <ma...@apache.org>
>>>> Date: Tuesday, March 25, 2025 at 8:27 AM
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: NIO Thread Madness
>>>> On 25/03/2025 11:24, William Crowell wrote:
>>>>> Chris,
>>>>>
>>>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>>>
>>>> Looking that the thread dump you sent me privately now.
>>>>
>>>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>>>
>>>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>>>
>>>> Mark
>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> William Crowell
>>>>>
>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>>> Subject: Re: NIO Thread Madness
>>>>> William,
>>>>>
>>>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>>>
>>>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>>>
>>>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>>>
>>>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>>>
>>>>> Are you able to use JMX or similar to see how many used database
>>>>> connections you have?
>>>>>
>>>>> When you say that Tomcat does not accept requests, do you mean that you
>>>>> make a request and it never returns, or do you mean that you get a
>>>>> "connection refused" or something similar?
>>>>>
>>>>>> I have my thread pool defined as follows in server.xml:
>>>>>>
>>>>>> …
>>>>>> <Executor name="tomcatThreadPool"
>>>>>> namePrefix="catalina-exec-"
>>>>>> maxThreads="1000"
>>>>>> minSpareThreads="50"
>>>>>> maxIdleTime="60000"
>>>>>> maxQueueSize="1000" />
>>>>>>
>>>>>> <Connector port="8080"
>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>> connectionTimeout="130000"
>>>>>> redirectPort="8443"
>>>>>> disableUploadTimeout="false"
>>>>>> acceptCount="1000"
>>>>>> maxConnections="1000"
>>>>>> executor="tomcatThreadPool" />
>>>>>
>>>>> Have you tried enabling any of the "abandoned"-related settings? I
>>>>> highly recommend logAbandoned="true" and possibly others.
>>>>>
>>>>> I would imagine you'd see threads waiting on database connections if you
>>>>> had exhausted your pool. Just remember that threads can be stuck on
>>>>> things without being BLOCKED.
>>>>>
>>>>> > Are there any logs I can enable to find out why the application
>>>>> server stops accepting connections?
>>>>>
>>>>> -chris
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>>
>>>>>
>>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>
>>>>
>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
Т���������������������������������������������������������������������ХF�V�7V'67&�&R�R���âW6W'2�V�7V'67&�&TF��6B�6�R��&pФf�"FF�F����6����G2�R���âW6W'2ֆV�F��6B�6�R��&pР
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Christopher Schultz <ch...@christopherschultz.net>.
William,
On 3/31/25 2:31 PM, William Crowell wrote:
> Question related to this. I found issue DBCP-599 which was released
> in DBCP 2.13.0 as part of Apache Tomcat 9.0.98 release. The
> characteristics of this major bug appear very similar to the issue I am
> having.
Are you using an old driver that does not support JDBC 4.2?
> In Tomcat’s lib directory there is a tomcat-dbcp.jar. Is that just
> DBCP2 repackaged under Tomcat or is that an interface from Tomcat
> into Apache Commons DBCP2?
It's re-packaged.
-chris
> From: William Crowell <WC...@perforce.com.INVALID>
> Date: Friday, March 28, 2025 at 12:22 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> Very good idea Chriis. Thank you.
>
> Regards,
>
> William Crowell
>
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Friday, March 28, 2025 at 12:05 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/26/25 7:06 PM, William Crowell wrote:
>> That maxTotal was a typo due to trying to copy the config from a
>> screenshot.
>>
>> I agree with your assessment. I think there are 2 situations going
>> on. One is that the code may not be properly closing connections,
>> and the connection pool is not properly configured.
> I wrote this a looong time ago, but it's very helpful to show to
> programmers who haven't been careful.
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C092af9879f8b47b68ff508dd6e14c1b7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638787757657843607%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=F65r%2FaEPBooSj5KfFrbD1UYPNPyZefLbDxM%2BDoGIgDI%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C092af9879f8b47b68ff508dd6e14c1b7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638787757657875920%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=FD%2FcM30z3BqgZktYzYa9b6oU4WbRmpEUlL%2ByxHtp4eU%3D&reserved=0><https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/>
>
> For your long-transactions, I think you are going to want to use a
> separate pool.
>
> -chris
>
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Date: Wednesday, March 26, 2025 at 6:57 PM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> William,
>>
>> On 3/25/25 2:51 PM, William Crowell wrote:
>>> Mark,
>>>
>>> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>>>
>>> …
>>> accessToUnderlyingConnectionAllowed=”true”
>>> maxIdle=”100”
>>> maxWaitMillis=”10000”
>>> minIdle=”25”
>>> validationQuery=”select 1 from dual”
>>> url=”blah blah blah”
>>> maxTotal=”true”
>>
>> ^ This is typically a number, not a true/false value.
>>
>>> logAbandoned=”true”
>>> removeAbandonedOnBorrow=”true”
>>> removeAbandonedTimeout=”900”
>>> removeAbandonedOnMaintenance=”true”
>>> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
>>> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>>>
>>> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>>>
>>> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
>>
>> When the database pool dries up, your application will basically stop.
>> Setting removeAbandoned as you have is good, but with those long long
>> timeouts, they aren't doing any good.
>>
>> I might set up a connection pool for "short queries" (like logging-in,
>> etc.) and then set up a separate pool for much longer queries.
>>
>> Also, you might want to review your use of connections, etc. to ensure
>> that you are always closing everything in finally blocks. Resource
>> management with JDBC can be tedious and if you don't do it right, you
>> can leak connections from your pool very easily.
>>
>> -chris
>>
>>> From: Mark Thomas <ma...@apache.org>
>>> Date: Tuesday, March 25, 2025 at 1:13 PM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> On 25/03/2025 12:33, William Crowell wrote:
>>>> Mark,
>>>>
>>>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>>>
>>> Hmm. Both the Acceptor thread and the Poller thread are running and
>>> appear to be in states that would enable new requests to be processed.
>>>
>>> There isn't logging in those two components for normal operations. I
>>> assume because of performance concerns.
>>>
>>> Do you have any thread dumps from when you have clients attempting new
>>> requests that are hanging? I'm wondering if processing is hanging
>>> somewhere in Tomcat / the application during request processing.
>>>
>>> I would probably be thinking about enabling remote debugging and
>>> connecting a debugger the next time it goes wrong and tracing the
>>> progress of a new request. But I accept that may not be practical for a
>>> production system.
>>>
>>> It seems unlikely that the new requests are hanging before they reach
>>> the Tomcat code but that seems unlikely.
>>>
>>> Other things to try:
>>>
>>> netstat may shed some light on the current state of Tomcat and database
>>> connections.
>>>
>>> Wireshark or similar on the client side might also provide some clues.
>>>
>>> I am wondering if Tomcat / Windows is responding at all, dropping the
>>> connection, sending a reset, etc.
>>>
>>> There might be some clues in the exceptions the exceptions that occur
>>> just before the problem starts. Can you share any of them?
>>>
>>> Mark
>>>
>>>>
>>>> Regards,
>>>>
>>>> William Crowell
>>>>
>>>> From: Mark Thomas <ma...@apache.org>
>>>> Date: Tuesday, March 25, 2025 at 8:27 AM
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: NIO Thread Madness
>>>> On 25/03/2025 11:24, William Crowell wrote:
>>>>> Chris,
>>>>>
>>>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>>>
>>>> Looking that the thread dump you sent me privately now.
>>>>
>>>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>>>
>>>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>>>
>>>> Mark
>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> William Crowell
>>>>>
>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>>> Subject: Re: NIO Thread Madness
>>>>> William,
>>>>>
>>>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>>>
>>>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>>>
>>>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>>>
>>>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>>>
>>>>> Are you able to use JMX or similar to see how many used database
>>>>> connections you have?
>>>>>
>>>>> When you say that Tomcat does not accept requests, do you mean that you
>>>>> make a request and it never returns, or do you mean that you get a
>>>>> "connection refused" or something similar?
>>>>>
>>>>>> I have my thread pool defined as follows in server.xml:
>>>>>>
>>>>>> …
>>>>>> <Executor name="tomcatThreadPool"
>>>>>> namePrefix="catalina-exec-"
>>>>>> maxThreads="1000"
>>>>>> minSpareThreads="50"
>>>>>> maxIdleTime="60000"
>>>>>> maxQueueSize="1000" />
>>>>>>
>>>>>> <Connector port="8080"
>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>> connectionTimeout="130000"
>>>>>> redirectPort="8443"
>>>>>> disableUploadTimeout="false"
>>>>>> acceptCount="1000"
>>>>>> maxConnections="1000"
>>>>>> executor="tomcatThreadPool" />
>>>>>
>>>>> Have you tried enabling any of the "abandoned"-related settings? I
>>>>> highly recommend logAbandoned="true" and possibly others.
>>>>>
>>>>> I would imagine you'd see threads waiting on database connections if you
>>>>> had exhausted your pool. Just remember that threads can be stuck on
>>>>> things without being BLOCKED.
>>>>>
>>>>> > Are there any logs I can enable to find out why the application
>>>>> server stops accepting connections?
>>>>>
>>>>> -chris
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>>
>>>>>
>>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>
>>>>
>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Question related to this. I found issue DBCP-599 which was released in DBCP 2.13.0 as part of Apache Tomcat 9.0.98 release. The characteristics of this major bug appear very similar to the issue I am having.
In Tomcat’s lib directory there is a tomcat-dbcp.jar. Is that just DBCP2 repackaged under Tomcat or is that an interface from Tomcat into Apache Commons DBCP2?
Regards,
William Crowell
From: William Crowell <WC...@perforce.com.INVALID>
Date: Friday, March 28, 2025 at 12:22 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
Very good idea Chriis. Thank you.
Regards,
William Crowell
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Friday, March 28, 2025 at 12:05 PM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
William,
On 3/26/25 7:06 PM, William Crowell wrote:
> That maxTotal was a typo due to trying to copy the config from a
> screenshot.
>
> I agree with your assessment. I think there are 2 situations going
> on. One is that the code may not be properly closing connections,
> and the connection pool is not properly configured.
I wrote this a looong time ago, but it's very helpful to show to
programmers who haven't been careful.
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C092af9879f8b47b68ff508dd6e14c1b7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638787757657843607%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=F65r%2FaEPBooSj5KfFrbD1UYPNPyZefLbDxM%2BDoGIgDI%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C092af9879f8b47b68ff508dd6e14c1b7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638787757657875920%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=FD%2FcM30z3BqgZktYzYa9b6oU4WbRmpEUlL%2ByxHtp4eU%3D&reserved=0><https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/>
For your long-transactions, I think you are going to want to use a
separate pool.
-chris
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Wednesday, March 26, 2025 at 6:57 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/25/25 2:51 PM, William Crowell wrote:
>> Mark,
>>
>> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>>
>> …
>> accessToUnderlyingConnectionAllowed=”true”
>> maxIdle=”100”
>> maxWaitMillis=”10000”
>> minIdle=”25”
>> validationQuery=”select 1 from dual”
>> url=”blah blah blah”
>> maxTotal=”true”
>
> ^ This is typically a number, not a true/false value.
>
>> logAbandoned=”true”
>> removeAbandonedOnBorrow=”true”
>> removeAbandonedTimeout=”900”
>> removeAbandonedOnMaintenance=”true”
>> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
>> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>>
>> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>>
>> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
>
> When the database pool dries up, your application will basically stop.
> Setting removeAbandoned as you have is good, but with those long long
> timeouts, they aren't doing any good.
>
> I might set up a connection pool for "short queries" (like logging-in,
> etc.) and then set up a separate pool for much longer queries.
>
> Also, you might want to review your use of connections, etc. to ensure
> that you are always closing everything in finally blocks. Resource
> management with JDBC can be tedious and if you don't do it right, you
> can leak connections from your pool very easily.
>
> -chris
>
>> From: Mark Thomas <ma...@apache.org>
>> Date: Tuesday, March 25, 2025 at 1:13 PM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> On 25/03/2025 12:33, William Crowell wrote:
>>> Mark,
>>>
>>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>>
>> Hmm. Both the Acceptor thread and the Poller thread are running and
>> appear to be in states that would enable new requests to be processed.
>>
>> There isn't logging in those two components for normal operations. I
>> assume because of performance concerns.
>>
>> Do you have any thread dumps from when you have clients attempting new
>> requests that are hanging? I'm wondering if processing is hanging
>> somewhere in Tomcat / the application during request processing.
>>
>> I would probably be thinking about enabling remote debugging and
>> connecting a debugger the next time it goes wrong and tracing the
>> progress of a new request. But I accept that may not be practical for a
>> production system.
>>
>> It seems unlikely that the new requests are hanging before they reach
>> the Tomcat code but that seems unlikely.
>>
>> Other things to try:
>>
>> netstat may shed some light on the current state of Tomcat and database
>> connections.
>>
>> Wireshark or similar on the client side might also provide some clues.
>>
>> I am wondering if Tomcat / Windows is responding at all, dropping the
>> connection, sending a reset, etc.
>>
>> There might be some clues in the exceptions the exceptions that occur
>> just before the problem starts. Can you share any of them?
>>
>> Mark
>>
>>>
>>> Regards,
>>>
>>> William Crowell
>>>
>>> From: Mark Thomas <ma...@apache.org>
>>> Date: Tuesday, March 25, 2025 at 8:27 AM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> On 25/03/2025 11:24, William Crowell wrote:
>>>> Chris,
>>>>
>>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>>
>>> Looking that the thread dump you sent me privately now.
>>>
>>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>>
>>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>>
>>> Mark
>>>
>>>>
>>>> Regards,
>>>>
>>>> William Crowell
>>>>
>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: NIO Thread Madness
>>>> William,
>>>>
>>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>>
>>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>>
>>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>>
>>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>>
>>>> Are you able to use JMX or similar to see how many used database
>>>> connections you have?
>>>>
>>>> When you say that Tomcat does not accept requests, do you mean that you
>>>> make a request and it never returns, or do you mean that you get a
>>>> "connection refused" or something similar?
>>>>
>>>>> I have my thread pool defined as follows in server.xml:
>>>>>
>>>>> …
>>>>> <Executor name="tomcatThreadPool"
>>>>> namePrefix="catalina-exec-"
>>>>> maxThreads="1000"
>>>>> minSpareThreads="50"
>>>>> maxIdleTime="60000"
>>>>> maxQueueSize="1000" />
>>>>>
>>>>> <Connector port="8080"
>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>> connectionTimeout="130000"
>>>>> redirectPort="8443"
>>>>> disableUploadTimeout="false"
>>>>> acceptCount="1000"
>>>>> maxConnections="1000"
>>>>> executor="tomcatThreadPool" />
>>>>
>>>> Have you tried enabling any of the "abandoned"-related settings? I
>>>> highly recommend logAbandoned="true" and possibly others.
>>>>
>>>> I would imagine you'd see threads waiting on database connections if you
>>>> had exhausted your pool. Just remember that threads can be stuck on
>>>> things without being BLOCKED.
>>>>
>>>> > Are there any logs I can enable to find out why the application
>>>> server stops accepting connections?
>>>>
>>>> -chris
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>
>>>>
>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Very good idea Chriis. Thank you.
Regards,
William Crowell
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Friday, March 28, 2025 at 12:05 PM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
William,
On 3/26/25 7:06 PM, William Crowell wrote:
> That maxTotal was a typo due to trying to copy the config from a
> screenshot.
>
> I agree with your assessment. I think there are 2 situations going
> on. One is that the code may not be properly closing connections,
> and the connection pool is not properly configured.
I wrote this a looong time ago, but it's very helpful to show to
programmers who haven't been careful.
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C94d759d31b034232774108dd6e1255ee%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638787747219563730%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=IjI0vAoxrQ53I%2FTByJRoQ4mRVYmG746pnENm8JUlq6Q%3D&reserved=0<https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/>
For your long-transactions, I think you are going to want to use a
separate pool.
-chris
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Wednesday, March 26, 2025 at 6:57 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/25/25 2:51 PM, William Crowell wrote:
>> Mark,
>>
>> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>>
>> …
>> accessToUnderlyingConnectionAllowed=”true”
>> maxIdle=”100”
>> maxWaitMillis=”10000”
>> minIdle=”25”
>> validationQuery=”select 1 from dual”
>> url=”blah blah blah”
>> maxTotal=”true”
>
> ^ This is typically a number, not a true/false value.
>
>> logAbandoned=”true”
>> removeAbandonedOnBorrow=”true”
>> removeAbandonedTimeout=”900”
>> removeAbandonedOnMaintenance=”true”
>> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
>> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>>
>> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>>
>> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
>
> When the database pool dries up, your application will basically stop.
> Setting removeAbandoned as you have is good, but with those long long
> timeouts, they aren't doing any good.
>
> I might set up a connection pool for "short queries" (like logging-in,
> etc.) and then set up a separate pool for much longer queries.
>
> Also, you might want to review your use of connections, etc. to ensure
> that you are always closing everything in finally blocks. Resource
> management with JDBC can be tedious and if you don't do it right, you
> can leak connections from your pool very easily.
>
> -chris
>
>> From: Mark Thomas <ma...@apache.org>
>> Date: Tuesday, March 25, 2025 at 1:13 PM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> On 25/03/2025 12:33, William Crowell wrote:
>>> Mark,
>>>
>>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>>
>> Hmm. Both the Acceptor thread and the Poller thread are running and
>> appear to be in states that would enable new requests to be processed.
>>
>> There isn't logging in those two components for normal operations. I
>> assume because of performance concerns.
>>
>> Do you have any thread dumps from when you have clients attempting new
>> requests that are hanging? I'm wondering if processing is hanging
>> somewhere in Tomcat / the application during request processing.
>>
>> I would probably be thinking about enabling remote debugging and
>> connecting a debugger the next time it goes wrong and tracing the
>> progress of a new request. But I accept that may not be practical for a
>> production system.
>>
>> It seems unlikely that the new requests are hanging before they reach
>> the Tomcat code but that seems unlikely.
>>
>> Other things to try:
>>
>> netstat may shed some light on the current state of Tomcat and database
>> connections.
>>
>> Wireshark or similar on the client side might also provide some clues.
>>
>> I am wondering if Tomcat / Windows is responding at all, dropping the
>> connection, sending a reset, etc.
>>
>> There might be some clues in the exceptions the exceptions that occur
>> just before the problem starts. Can you share any of them?
>>
>> Mark
>>
>>>
>>> Regards,
>>>
>>> William Crowell
>>>
>>> From: Mark Thomas <ma...@apache.org>
>>> Date: Tuesday, March 25, 2025 at 8:27 AM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> On 25/03/2025 11:24, William Crowell wrote:
>>>> Chris,
>>>>
>>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>>
>>> Looking that the thread dump you sent me privately now.
>>>
>>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>>
>>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>>
>>> Mark
>>>
>>>>
>>>> Regards,
>>>>
>>>> William Crowell
>>>>
>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: NIO Thread Madness
>>>> William,
>>>>
>>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>>
>>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>>
>>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>>
>>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>>
>>>> Are you able to use JMX or similar to see how many used database
>>>> connections you have?
>>>>
>>>> When you say that Tomcat does not accept requests, do you mean that you
>>>> make a request and it never returns, or do you mean that you get a
>>>> "connection refused" or something similar?
>>>>
>>>>> I have my thread pool defined as follows in server.xml:
>>>>>
>>>>> …
>>>>> <Executor name="tomcatThreadPool"
>>>>> namePrefix="catalina-exec-"
>>>>> maxThreads="1000"
>>>>> minSpareThreads="50"
>>>>> maxIdleTime="60000"
>>>>> maxQueueSize="1000" />
>>>>>
>>>>> <Connector port="8080"
>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>> connectionTimeout="130000"
>>>>> redirectPort="8443"
>>>>> disableUploadTimeout="false"
>>>>> acceptCount="1000"
>>>>> maxConnections="1000"
>>>>> executor="tomcatThreadPool" />
>>>>
>>>> Have you tried enabling any of the "abandoned"-related settings? I
>>>> highly recommend logAbandoned="true" and possibly others.
>>>>
>>>> I would imagine you'd see threads waiting on database connections if you
>>>> had exhausted your pool. Just remember that threads can be stuck on
>>>> things without being BLOCKED.
>>>>
>>>> > Are there any logs I can enable to find out why the application
>>>> server stops accepting connections?
>>>>
>>>> -chris
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>
>>>>
>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Christopher Schultz <ch...@christopherschultz.net>.
William,
On 3/26/25 7:06 PM, William Crowell wrote:
> That maxTotal was a typo due to trying to copy the config from a
> screenshot.
>
> I agree with your assessment. I think there are 2 situations going
> on. One is that the code may not be properly closing connections,
> and the connection pool is not properly configured.
I wrote this a looong time ago, but it's very helpful to show to
programmers who haven't been careful.
https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/
For your long-transactions, I think you are going to want to use a
separate pool.
-chris
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Wednesday, March 26, 2025 at 6:57 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/25/25 2:51 PM, William Crowell wrote:
>> Mark,
>>
>> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>>
>> …
>> accessToUnderlyingConnectionAllowed=”true”
>> maxIdle=”100”
>> maxWaitMillis=”10000”
>> minIdle=”25”
>> validationQuery=”select 1 from dual”
>> url=”blah blah blah”
>> maxTotal=”true”
>
> ^ This is typically a number, not a true/false value.
>
>> logAbandoned=”true”
>> removeAbandonedOnBorrow=”true”
>> removeAbandonedTimeout=”900”
>> removeAbandonedOnMaintenance=”true”
>> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
>> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>>
>> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>>
>> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
>
> When the database pool dries up, your application will basically stop.
> Setting removeAbandoned as you have is good, but with those long long
> timeouts, they aren't doing any good.
>
> I might set up a connection pool for "short queries" (like logging-in,
> etc.) and then set up a separate pool for much longer queries.
>
> Also, you might want to review your use of connections, etc. to ensure
> that you are always closing everything in finally blocks. Resource
> management with JDBC can be tedious and if you don't do it right, you
> can leak connections from your pool very easily.
>
> -chris
>
>> From: Mark Thomas <ma...@apache.org>
>> Date: Tuesday, March 25, 2025 at 1:13 PM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> On 25/03/2025 12:33, William Crowell wrote:
>>> Mark,
>>>
>>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>>
>> Hmm. Both the Acceptor thread and the Poller thread are running and
>> appear to be in states that would enable new requests to be processed.
>>
>> There isn't logging in those two components for normal operations. I
>> assume because of performance concerns.
>>
>> Do you have any thread dumps from when you have clients attempting new
>> requests that are hanging? I'm wondering if processing is hanging
>> somewhere in Tomcat / the application during request processing.
>>
>> I would probably be thinking about enabling remote debugging and
>> connecting a debugger the next time it goes wrong and tracing the
>> progress of a new request. But I accept that may not be practical for a
>> production system.
>>
>> It seems unlikely that the new requests are hanging before they reach
>> the Tomcat code but that seems unlikely.
>>
>> Other things to try:
>>
>> netstat may shed some light on the current state of Tomcat and database
>> connections.
>>
>> Wireshark or similar on the client side might also provide some clues.
>>
>> I am wondering if Tomcat / Windows is responding at all, dropping the
>> connection, sending a reset, etc.
>>
>> There might be some clues in the exceptions the exceptions that occur
>> just before the problem starts. Can you share any of them?
>>
>> Mark
>>
>>>
>>> Regards,
>>>
>>> William Crowell
>>>
>>> From: Mark Thomas <ma...@apache.org>
>>> Date: Tuesday, March 25, 2025 at 8:27 AM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> On 25/03/2025 11:24, William Crowell wrote:
>>>> Chris,
>>>>
>>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>>
>>> Looking that the thread dump you sent me privately now.
>>>
>>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>>
>>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>>
>>> Mark
>>>
>>>>
>>>> Regards,
>>>>
>>>> William Crowell
>>>>
>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>>> Subject: Re: NIO Thread Madness
>>>> William,
>>>>
>>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>>
>>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>>
>>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>>
>>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>>
>>>> Are you able to use JMX or similar to see how many used database
>>>> connections you have?
>>>>
>>>> When you say that Tomcat does not accept requests, do you mean that you
>>>> make a request and it never returns, or do you mean that you get a
>>>> "connection refused" or something similar?
>>>>
>>>>> I have my thread pool defined as follows in server.xml:
>>>>>
>>>>> …
>>>>> <Executor name="tomcatThreadPool"
>>>>> namePrefix="catalina-exec-"
>>>>> maxThreads="1000"
>>>>> minSpareThreads="50"
>>>>> maxIdleTime="60000"
>>>>> maxQueueSize="1000" />
>>>>>
>>>>> <Connector port="8080"
>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>> connectionTimeout="130000"
>>>>> redirectPort="8443"
>>>>> disableUploadTimeout="false"
>>>>> acceptCount="1000"
>>>>> maxConnections="1000"
>>>>> executor="tomcatThreadPool" />
>>>>
>>>> Have you tried enabling any of the "abandoned"-related settings? I
>>>> highly recommend logAbandoned="true" and possibly others.
>>>>
>>>> I would imagine you'd see threads waiting on database connections if you
>>>> had exhausted your pool. Just remember that threads can be stuck on
>>>> things without being BLOCKED.
>>>>
>>>> > Are there any logs I can enable to find out why the application
>>>> server stops accepting connections?
>>>>
>>>> -chris
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>>
>>>>
>>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Chris,
That maxTotal was a typo due to trying to copy the config from a screenshot.
I agree with your assessment. I think there are 2 situations going on. One is that the code may not be properly closing connections, and the connection pool is not properly configured.
Regards,
William Crowell
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Wednesday, March 26, 2025 at 6:57 PM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
William,
On 3/25/25 2:51 PM, William Crowell wrote:
> Mark,
>
> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>
> …
> accessToUnderlyingConnectionAllowed=”true”
> maxIdle=”100”
> maxWaitMillis=”10000”
> minIdle=”25”
> validationQuery=”select 1 from dual”
> url=”blah blah blah”
> maxTotal=”true”
^ This is typically a number, not a true/false value.
> logAbandoned=”true”
> removeAbandonedOnBorrow=”true”
> removeAbandonedTimeout=”900”
> removeAbandonedOnMaintenance=”true”
> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>
> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>
> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
When the database pool dries up, your application will basically stop.
Setting removeAbandoned as you have is good, but with those long long
timeouts, they aren't doing any good.
I might set up a connection pool for "short queries" (like logging-in,
etc.) and then set up a separate pool for much longer queries.
Also, you might want to review your use of connections, etc. to ensure
that you are always closing everything in finally blocks. Resource
management with JDBC can be tedious and if you don't do it right, you
can leak connections from your pool very easily.
-chris
> From: Mark Thomas <ma...@apache.org>
> Date: Tuesday, March 25, 2025 at 1:13 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> On 25/03/2025 12:33, William Crowell wrote:
>> Mark,
>>
>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>
> Hmm. Both the Acceptor thread and the Poller thread are running and
> appear to be in states that would enable new requests to be processed.
>
> There isn't logging in those two components for normal operations. I
> assume because of performance concerns.
>
> Do you have any thread dumps from when you have clients attempting new
> requests that are hanging? I'm wondering if processing is hanging
> somewhere in Tomcat / the application during request processing.
>
> I would probably be thinking about enabling remote debugging and
> connecting a debugger the next time it goes wrong and tracing the
> progress of a new request. But I accept that may not be practical for a
> production system.
>
> It seems unlikely that the new requests are hanging before they reach
> the Tomcat code but that seems unlikely.
>
> Other things to try:
>
> netstat may shed some light on the current state of Tomcat and database
> connections.
>
> Wireshark or similar on the client side might also provide some clues.
>
> I am wondering if Tomcat / Windows is responding at all, dropping the
> connection, sending a reset, etc.
>
> There might be some clues in the exceptions the exceptions that occur
> just before the problem starts. Can you share any of them?
>
> Mark
>
>>
>> Regards,
>>
>> William Crowell
>>
>> From: Mark Thomas <ma...@apache.org>
>> Date: Tuesday, March 25, 2025 at 8:27 AM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> On 25/03/2025 11:24, William Crowell wrote:
>>> Chris,
>>>
>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>
>> Looking that the thread dump you sent me privately now.
>>
>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>
>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>
>> Mark
>>
>>>
>>> Regards,
>>>
>>> William Crowell
>>>
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> William,
>>>
>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>
>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>
>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>
>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>
>>> Are you able to use JMX or similar to see how many used database
>>> connections you have?
>>>
>>> When you say that Tomcat does not accept requests, do you mean that you
>>> make a request and it never returns, or do you mean that you get a
>>> "connection refused" or something similar?
>>>
>>>> I have my thread pool defined as follows in server.xml:
>>>>
>>>> …
>>>> <Executor name="tomcatThreadPool"
>>>> namePrefix="catalina-exec-"
>>>> maxThreads="1000"
>>>> minSpareThreads="50"
>>>> maxIdleTime="60000"
>>>> maxQueueSize="1000" />
>>>>
>>>> <Connector port="8080"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>> connectionTimeout="130000"
>>>> redirectPort="8443"
>>>> disableUploadTimeout="false"
>>>> acceptCount="1000"
>>>> maxConnections="1000"
>>>> executor="tomcatThreadPool" />
>>>
>>> Have you tried enabling any of the "abandoned"-related settings? I
>>> highly recommend logAbandoned="true" and possibly others.
>>>
>>> I would imagine you'd see threads waiting on database connections if you
>>> had exhausted your pool. Just remember that threads can be stuck on
>>> things without being BLOCKED.
>>>
>>> > Are there any logs I can enable to find out why the application
>>> server stops accepting connections?
>>>
>>> -chris
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Christopher Schultz <ch...@christopherschultz.net>.
William,
On 3/25/25 2:51 PM, William Crowell wrote:
> Mark,
>
> I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
>
> …
> accessToUnderlyingConnectionAllowed=”true”
> maxIdle=”100”
> maxWaitMillis=”10000”
> minIdle=”25”
> validationQuery=”select 1 from dual”
> url=”blah blah blah”
> maxTotal=”true”
^ This is typically a number, not a true/false value.
> logAbandoned=”true”
> removeAbandonedOnBorrow=”true”
> removeAbandonedTimeout=”900”
> removeAbandonedOnMaintenance=”true”
> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
> minEvictableIdleTimeMillis=”900000” <!—15 minutes •
>
> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
>
> Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
When the database pool dries up, your application will basically stop.
Setting removeAbandoned as you have is good, but with those long long
timeouts, they aren't doing any good.
I might set up a connection pool for "short queries" (like logging-in,
etc.) and then set up a separate pool for much longer queries.
Also, you might want to review your use of connections, etc. to ensure
that you are always closing everything in finally blocks. Resource
management with JDBC can be tedious and if you don't do it right, you
can leak connections from your pool very easily.
-chris
> From: Mark Thomas <ma...@apache.org>
> Date: Tuesday, March 25, 2025 at 1:13 PM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> On 25/03/2025 12:33, William Crowell wrote:
>> Mark,
>>
>> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
>
> Hmm. Both the Acceptor thread and the Poller thread are running and
> appear to be in states that would enable new requests to be processed.
>
> There isn't logging in those two components for normal operations. I
> assume because of performance concerns.
>
> Do you have any thread dumps from when you have clients attempting new
> requests that are hanging? I'm wondering if processing is hanging
> somewhere in Tomcat / the application during request processing.
>
> I would probably be thinking about enabling remote debugging and
> connecting a debugger the next time it goes wrong and tracing the
> progress of a new request. But I accept that may not be practical for a
> production system.
>
> It seems unlikely that the new requests are hanging before they reach
> the Tomcat code but that seems unlikely.
>
> Other things to try:
>
> netstat may shed some light on the current state of Tomcat and database
> connections.
>
> Wireshark or similar on the client side might also provide some clues.
>
> I am wondering if Tomcat / Windows is responding at all, dropping the
> connection, sending a reset, etc.
>
> There might be some clues in the exceptions the exceptions that occur
> just before the problem starts. Can you share any of them?
>
> Mark
>
>>
>> Regards,
>>
>> William Crowell
>>
>> From: Mark Thomas <ma...@apache.org>
>> Date: Tuesday, March 25, 2025 at 8:27 AM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> On 25/03/2025 11:24, William Crowell wrote:
>>> Chris,
>>>
>>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>>
>> Looking that the thread dump you sent me privately now.
>>
>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>>
>> Are you connecting directly to Tomcat or is there a proxy involved at all?
>>
>> Mark
>>
>>>
>>> Regards,
>>>
>>> William Crowell
>>>
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Date: Tuesday, March 25, 2025 at 7:20 AM
>>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>>> Subject: Re: NIO Thread Madness
>>> William,
>>>
>>> On 3/24/25 2:56 PM, William Crowell wrote:
>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>>
>>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>>
>>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>>
>>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>>
>>> Are you able to use JMX or similar to see how many used database
>>> connections you have?
>>>
>>> When you say that Tomcat does not accept requests, do you mean that you
>>> make a request and it never returns, or do you mean that you get a
>>> "connection refused" or something similar?
>>>
>>>> I have my thread pool defined as follows in server.xml:
>>>>
>>>> …
>>>> <Executor name="tomcatThreadPool"
>>>> namePrefix="catalina-exec-"
>>>> maxThreads="1000"
>>>> minSpareThreads="50"
>>>> maxIdleTime="60000"
>>>> maxQueueSize="1000" />
>>>>
>>>> <Connector port="8080"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>> connectionTimeout="130000"
>>>> redirectPort="8443"
>>>> disableUploadTimeout="false"
>>>> acceptCount="1000"
>>>> maxConnections="1000"
>>>> executor="tomcatThreadPool" />
>>>
>>> Have you tried enabling any of the "abandoned"-related settings? I
>>> highly recommend logAbandoned="true" and possibly others.
>>>
>>> I would imagine you'd see threads waiting on database connections if you
>>> had exhausted your pool. Just remember that threads can be stuck on
>>> things without being BLOCKED.
>>>
>>> > Are there any logs I can enable to find out why the application
>>> server stops accepting connections?
>>>
>>> -chris
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Mark,
I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
…
accessToUnderlyingConnectionAllowed=”true”
maxIdle=”100”
maxWaitMillis=”10000”
minIdle=”25”
validationQuery=”select 1 from dual”
url=”blah blah blah”
maxTotal=”true”
logAbandoned=”true”
removeAbandonedOnBorrow=”true”
removeAbandonedTimeout=”900”
removeAbandonedOnMaintenance=”true”
timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
minEvictableIdleTimeMillis=”900000” <!—15 minutes •
The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously.
Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that.
Regards,
William Crowell
From: Mark Thomas <ma...@apache.org>
Date: Tuesday, March 25, 2025 at 1:13 PM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
On 25/03/2025 12:33, William Crowell wrote:
> Mark,
>
> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
Hmm. Both the Acceptor thread and the Poller thread are running and
appear to be in states that would enable new requests to be processed.
There isn't logging in those two components for normal operations. I
assume because of performance concerns.
Do you have any thread dumps from when you have clients attempting new
requests that are hanging? I'm wondering if processing is hanging
somewhere in Tomcat / the application during request processing.
I would probably be thinking about enabling remote debugging and
connecting a debugger the next time it goes wrong and tracing the
progress of a new request. But I accept that may not be practical for a
production system.
It seems unlikely that the new requests are hanging before they reach
the Tomcat code but that seems unlikely.
Other things to try:
netstat may shed some light on the current state of Tomcat and database
connections.
Wireshark or similar on the client side might also provide some clues.
I am wondering if Tomcat / Windows is responding at all, dropping the
connection, sending a reset, etc.
There might be some clues in the exceptions the exceptions that occur
just before the problem starts. Can you share any of them?
Mark
>
> Regards,
>
> William Crowell
>
> From: Mark Thomas <ma...@apache.org>
> Date: Tuesday, March 25, 2025 at 8:27 AM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> On 25/03/2025 11:24, William Crowell wrote:
>> Chris,
>>
>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>
> Looking that the thread dump you sent me privately now.
>
> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>
> Are you connecting directly to Tomcat or is there a proxy involved at all?
>
> Mark
>
>>
>> Regards,
>>
>> William Crowell
>>
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Date: Tuesday, March 25, 2025 at 7:20 AM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> William,
>>
>> On 3/24/25 2:56 PM, William Crowell wrote:
>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>
>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>
>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>
>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>
>> Are you able to use JMX or similar to see how many used database
>> connections you have?
>>
>> When you say that Tomcat does not accept requests, do you mean that you
>> make a request and it never returns, or do you mean that you get a
>> "connection refused" or something similar?
>>
>>> I have my thread pool defined as follows in server.xml:
>>>
>>> …
>>> <Executor name="tomcatThreadPool"
>>> namePrefix="catalina-exec-"
>>> maxThreads="1000"
>>> minSpareThreads="50"
>>> maxIdleTime="60000"
>>> maxQueueSize="1000" />
>>>
>>> <Connector port="8080"
>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>> connectionTimeout="130000"
>>> redirectPort="8443"
>>> disableUploadTimeout="false"
>>> acceptCount="1000"
>>> maxConnections="1000"
>>> executor="tomcatThreadPool" />
>>
>> Have you tried enabling any of the "abandoned"-related settings? I
>> highly recommend logAbandoned="true" and possibly others.
>>
>> I would imagine you'd see threads waiting on database connections if you
>> had exhausted your pool. Just remember that threads can be stuck on
>> things without being BLOCKED.
>>
>> > Are there any logs I can enable to find out why the application
>> server stops accepting connections?
>>
>> -chris
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Mark Thomas <ma...@apache.org>.
On 25/03/2025 12:33, William Crowell wrote:
> Mark,
>
> I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
Hmm. Both the Acceptor thread and the Poller thread are running and
appear to be in states that would enable new requests to be processed.
There isn't logging in those two components for normal operations. I
assume because of performance concerns.
Do you have any thread dumps from when you have clients attempting new
requests that are hanging? I'm wondering if processing is hanging
somewhere in Tomcat / the application during request processing.
I would probably be thinking about enabling remote debugging and
connecting a debugger the next time it goes wrong and tracing the
progress of a new request. But I accept that may not be practical for a
production system.
It seems unlikely that the new requests are hanging before they reach
the Tomcat code but that seems unlikely.
Other things to try:
netstat may shed some light on the current state of Tomcat and database
connections.
Wireshark or similar on the client side might also provide some clues.
I am wondering if Tomcat / Windows is responding at all, dropping the
connection, sending a reset, etc.
There might be some clues in the exceptions the exceptions that occur
just before the problem starts. Can you share any of them?
Mark
>
> Regards,
>
> William Crowell
>
> From: Mark Thomas <ma...@apache.org>
> Date: Tuesday, March 25, 2025 at 8:27 AM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> On 25/03/2025 11:24, William Crowell wrote:
>> Chris,
>>
>> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
>
> Looking that the thread dump you sent me privately now.
>
> Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
>
> Are you connecting directly to Tomcat or is there a proxy involved at all?
>
> Mark
>
>>
>> Regards,
>>
>> William Crowell
>>
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Date: Tuesday, March 25, 2025 at 7:20 AM
>> To: users@tomcat.apache.org <us...@tomcat.apache.org>
>> Subject: Re: NIO Thread Madness
>> William,
>>
>> On 3/24/25 2:56 PM, William Crowell wrote:
>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>>
>>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>>
>>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>>
>>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>>
>> Are you able to use JMX or similar to see how many used database
>> connections you have?
>>
>> When you say that Tomcat does not accept requests, do you mean that you
>> make a request and it never returns, or do you mean that you get a
>> "connection refused" or something similar?
>>
>>> I have my thread pool defined as follows in server.xml:
>>>
>>> …
>>> <Executor name="tomcatThreadPool"
>>> namePrefix="catalina-exec-"
>>> maxThreads="1000"
>>> minSpareThreads="50"
>>> maxIdleTime="60000"
>>> maxQueueSize="1000" />
>>>
>>> <Connector port="8080"
>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>> connectionTimeout="130000"
>>> redirectPort="8443"
>>> disableUploadTimeout="false"
>>> acceptCount="1000"
>>> maxConnections="1000"
>>> executor="tomcatThreadPool" />
>>
>> Have you tried enabling any of the "abandoned"-related settings? I
>> highly recommend logAbandoned="true" and possibly others.
>>
>> I would imagine you'd see threads waiting on database connections if you
>> had exhausted your pool. Just remember that threads can be stuck on
>> things without being BLOCKED.
>>
>> > Are there any logs I can enable to find out why the application
>> server stops accepting connections?
>>
>> -chris
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Mark,
I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung.
Regards,
William Crowell
From: Mark Thomas <ma...@apache.org>
Date: Tuesday, March 25, 2025 at 8:27 AM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
On 25/03/2025 11:24, William Crowell wrote:
> Chris,
>
> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
Looking that the thread dump you sent me privately now.
Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
Are you connecting directly to Tomcat or is there a proxy involved at all?
Mark
>
> Regards,
>
> William Crowell
>
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Tuesday, March 25, 2025 at 7:20 AM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/24/25 2:56 PM, William Crowell wrote:
>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>
>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>
>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>
>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>
> Are you able to use JMX or similar to see how many used database
> connections you have?
>
> When you say that Tomcat does not accept requests, do you mean that you
> make a request and it never returns, or do you mean that you get a
> "connection refused" or something similar?
>
>> I have my thread pool defined as follows in server.xml:
>>
>> …
>> <Executor name="tomcatThreadPool"
>> namePrefix="catalina-exec-"
>> maxThreads="1000"
>> minSpareThreads="50"
>> maxIdleTime="60000"
>> maxQueueSize="1000" />
>>
>> <Connector port="8080"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> connectionTimeout="130000"
>> redirectPort="8443"
>> disableUploadTimeout="false"
>> acceptCount="1000"
>> maxConnections="1000"
>> executor="tomcatThreadPool" />
>
> Have you tried enabling any of the "abandoned"-related settings? I
> highly recommend logAbandoned="true" and possibly others.
>
> I would imagine you'd see threads waiting on database connections if you
> had exhausted your pool. Just remember that threads can be stuck on
> things without being BLOCKED.
>
> > Are there any logs I can enable to find out why the application
> server stops accepting connections?
>
> -chris
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Mark Thomas <ma...@apache.org>.
On 25/03/2025 11:24, William Crowell wrote:
> Chris,
>
> Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
Looking that the thread dump you sent me privately now.
Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
Are you connecting directly to Tomcat or is there a proxy involved at all?
Mark
>
> Regards,
>
> William Crowell
>
> From: Christopher Schultz <ch...@christopherschultz.net>
> Date: Tuesday, March 25, 2025 at 7:20 AM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
> William,
>
> On 3/24/25 2:56 PM, William Crowell wrote:
>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>>
>> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>>
>> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>>
>> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
>
> Are you able to use JMX or similar to see how many used database
> connections you have?
>
> When you say that Tomcat does not accept requests, do you mean that you
> make a request and it never returns, or do you mean that you get a
> "connection refused" or something similar?
>
>> I have my thread pool defined as follows in server.xml:
>>
>> …
>> <Executor name="tomcatThreadPool"
>> namePrefix="catalina-exec-"
>> maxThreads="1000"
>> minSpareThreads="50"
>> maxIdleTime="60000"
>> maxQueueSize="1000" />
>>
>> <Connector port="8080"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> connectionTimeout="130000"
>> redirectPort="8443"
>> disableUploadTimeout="false"
>> acceptCount="1000"
>> maxConnections="1000"
>> executor="tomcatThreadPool" />
>
> Have you tried enabling any of the "abandoned"-related settings? I
> highly recommend logAbandoned="true" and possibly others.
>
> I would imagine you'd see threads waiting on database connections if you
> had exhausted your pool. Just remember that threads can be stuck on
> things without being BLOCKED.
>
> > Are there any logs I can enable to find out why the application
> server stops accepting connections?
>
> -chris
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Chris,
Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs.
Regards,
William Crowell
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Tuesday, March 25, 2025 at 7:20 AM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
William,
On 3/24/25 2:56 PM, William Crowell wrote:
> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>
> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>
> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>
> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
Are you able to use JMX or similar to see how many used database
connections you have?
When you say that Tomcat does not accept requests, do you mean that you
make a request and it never returns, or do you mean that you get a
"connection refused" or something similar?
> I have my thread pool defined as follows in server.xml:
>
> …
> <Executor name="tomcatThreadPool"
> namePrefix="catalina-exec-"
> maxThreads="1000"
> minSpareThreads="50"
> maxIdleTime="60000"
> maxQueueSize="1000" />
>
> <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="130000"
> redirectPort="8443"
> disableUploadTimeout="false"
> acceptCount="1000"
> maxConnections="1000"
> executor="tomcatThreadPool" />
Have you tried enabling any of the "abandoned"-related settings? I
highly recommend logAbandoned="true" and possibly others.
I would imagine you'd see threads waiting on database connections if you
had exhausted your pool. Just remember that threads can be stuck on
things without being BLOCKED.
> Are there any logs I can enable to find out why the application
server stops accepting connections?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Christopher Schultz <ch...@christopherschultz.net>.
William,
On 3/24/25 2:56 PM, William Crowell wrote:
> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
>
> I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests.
>
> I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced.
>
> I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many).
Are you able to use JMX or similar to see how many used database
connections you have?
When you say that Tomcat does not accept requests, do you mean that you
make a request and it never returns, or do you mean that you get a
"connection refused" or something similar?
> I have my thread pool defined as follows in server.xml:
>
> …
> <Executor name="tomcatThreadPool"
> namePrefix="catalina-exec-"
> maxThreads="1000"
> minSpareThreads="50"
> maxIdleTime="60000"
> maxQueueSize="1000" />
>
> <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="130000"
> redirectPort="8443"
> disableUploadTimeout="false"
> acceptCount="1000"
> maxConnections="1000"
> executor="tomcatThreadPool" />
Have you tried enabling any of the "abandoned"-related settings? I
highly recommend logAbandoned="true" and possibly others.
I would imagine you'd see threads waiting on database connections if you
had exhausted your pool. Just remember that threads can be stuck on
things without being BLOCKED.
> Are there any logs I can enable to find out why the application
server stops accepting connections?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Thread Madness
Posted by Mark Thomas <ma...@apache.org>.
Can you provide those after the issue thread dumps? Privately is fine if
you don't want to share them publicly.
Tomcat not responding to any requests is something that shouldn't
happen. I'd like to understand what is going on.
Mark
On 25/03/2025 10:22, William Crowell wrote:
> Mark,
>
> I did take 3 thread dumps, but it was after the issue occurred. The thread dumps just show threads in a waiting for work state. They are not doing anything. They are not blocked. When I attach JProfiler to the JVM then it points out that the threads were blocked during that short period of time due to a database connectivity issue. Then Tomcat stops servicing requests. The JVM is up though and has a valid process ID. I would have thought the threads would be in a blocked state for a long period of time in this situation.
>
> Regards,
>
> William Crowell
>
>
>
> ________________________________
> From: Mark Thomas <ma...@apache.org>
> Sent: Tuesday, March 25, 2025 5:09:20 AM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: NIO Thread Madness
>
> On 24/03/2025 18:56, William Crowell wrote:
>
> <snip/>
>
>> Are there any logs I can enable to find out why the application server stops accepting connections?
>
> I'd suggest taking 3 thread dumps approx 5s apart when this happens.
> Hopefully you'll see a bunch of threads waiting on the database and
> where they are waiting will (hopefully) tell you why they are waiting
> and point you towards a solution.
>
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Mark,
I did take 3 thread dumps, but it was after the issue occurred. The thread dumps just show threads in a waiting for work state. They are not doing anything. They are not blocked. When I attach JProfiler to the JVM then it points out that the threads were blocked during that short period of time due to a database connectivity issue. Then Tomcat stops servicing requests. The JVM is up though and has a valid process ID. I would have thought the threads would be in a blocked state for a long period of time in this situation.
Regards,
William Crowell
________________________________
From: Mark Thomas <ma...@apache.org>
Sent: Tuesday, March 25, 2025 5:09:20 AM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: NIO Thread Madness
On 24/03/2025 18:56, William Crowell wrote:
<snip/>
> Are there any logs I can enable to find out why the application server stops accepting connections?
I'd suggest taking 3 thread dumps approx 5s apart when this happens.
Hopefully you'll see a bunch of threads waiting on the database and
where they are waiting will (hopefully) tell you why they are waiting
and point you towards a solution.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: NIO Thread Madness
Posted by Mark Thomas <ma...@apache.org>.
On 24/03/2025 18:56, William Crowell wrote:
<snip/>
> Are there any logs I can enable to find out why the application server stops accepting connections?
I'd suggest taking 3 thread dumps approx 5s apart when this happens.
Hopefully you'll see a bunch of threads waiting on the database and
where they are waiting will (hopefully) tell you why they are waiting
and point you towards a solution.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] NIO Thread Madness
Posted by William Crowell <WC...@perforce.com.INVALID>.
Chris,
That’s a very good point, and that is something we will do.
Regards,
William Crowell
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Tuesday, March 25, 2025 at 7:16 AM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: [OT] NIO Thread Madness
William,
On 3/24/25 2:56 PM, William Crowell wrote:
> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
Might I suggest that you stick with both min and max heap sizes set to
16GiB? If you expect to run for a long time, you should expect to use
that 16GiB heap. If you are going to use it, you may as well allocate it
all during startup and use it, rather than re-sizing the heap as you go.
Re-sizing the heap is very expensive, so you should probably avoid it.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: [OT] NIO Thread Madness
Posted by Christopher Schultz <ch...@christopherschultz.net>.
William,
On 3/24/25 2:56 PM, William Crowell wrote:
> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
Might I suggest that you stick with both min and max heap sizes set to
16GiB? If you expect to run for a long time, you should expect to use
that 16GiB heap. If you are going to use it, you may as well allocate it
all during startup and use it, rather than re-sizing the heap as you go.
Re-sizing the heap is very expensive, so you should probably avoid it.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org