You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Chris Brown <Ch...@ncci.com> on 2016/03/04 20:13:10 UTC
Tomcat 7 Web Service Shutdown Can't Stop Threads
Tomcat version: 7.0.42
Server OS: Sun Solaris
UCP Version: Oracle 12c
Oracle Driver: ojdbc7
JDK version: 1.8.0_31
Issue Description:
When tomcat shuts down our web service during a hot deployment threads are left running that tomcat cannot shutdown. I believe the threads are from the universal connection pool. We use oracle's universal connection pool (ucp.jar). After multiple deployments the threads build up and cause high CPU usage from the 2 tomcat threads.
Log Error Examples:
Feb 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/RatingService] appears to have started a thread named [UCP-worker-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Feb 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/RatingService] appears to have started a thread named [Thread-438] but has failed to stop it. This is very likely to create a memory leak.
Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/RatingService] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak.
Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/RatingService] appears to have started a thread named [ForkJoinPool.commonPool-worker-32] but has failed to stop it. This is very likely to create a memory leak.
Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/RatingService] appears to have started a thread named [Timer-121] but has failed to stop it. This is very likely to create a memory leak.
Correction made to code to try and fix issue:
I added a ServletContextListener to our web service. On the contextDestroyed method we destroy the connection pools. Our web service uses 2 connection pools.
I also upgraded to Tomcat 8 and the issue still occurred, but the messages are now warnings instead of SEVERE errors.
I've searched the web and have tried many suggestions that have not helped except destroying the connection pools, but the issue still occurs. Please help!!!
Thank you,
Chris Brown
Systems Consultant
National Council on
Compensation Insurance
901 Peninsula Corporate Circle
Boca Raton, FL 33487-1362
(P) 561-893-2324
chris_brown@ncci.com
________________________________
The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
Posted by "patrycio ." <pa...@gmail.com>.
Chris sing a song for us.
Em 04/03/2016 16:13, "Chris Brown" <Ch...@ncci.com> escreveu:
> Tomcat version: 7.0.42
> Server OS: Sun Solaris
> UCP Version: Oracle 12c
> Oracle Driver: ojdbc7
> JDK version: 1.8.0_31
>
> Issue Description:
> When tomcat shuts down our web service during a hot deployment threads are
> left running that tomcat cannot shutdown. I believe the threads are from
> the universal connection pool. We use oracle's universal connection pool
> (ucp.jar). After multiple deployments the threads build up and cause high
> CPU usage from the 2 tomcat threads.
>
> Log Error Examples:
> Feb 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/RatingService] appears to have started a
> thread named [UCP-worker-thread-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Feb 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/RatingService] appears to have started a
> thread named [Thread-438] but has failed to stop it. This is very likely to
> create a memory leak.
> Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/RatingService] appears to have started a
> thread named
> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser]
> but has failed to stop it. This is very likely to create a memory leak.
> Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/RatingService] appears to have started a
> thread named [ForkJoinPool.commonPool-worker-32] but has failed to stop it.
> This is very likely to create a memory leak.
> Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/RatingService] appears to have started a
> thread named [Timer-121] but has failed to stop it. This is very likely to
> create a memory leak.
>
> Correction made to code to try and fix issue:
> I added a ServletContextListener to our web service. On the
> contextDestroyed method we destroy the connection pools. Our web service
> uses 2 connection pools.
>
> I also upgraded to Tomcat 8 and the issue still occurred, but the messages
> are now warnings instead of SEVERE errors.
>
> I've searched the web and have tried many suggestions that have not helped
> except destroying the connection pools, but the issue still occurs. Please
> help!!!
>
> Thank you,
> Chris Brown
> Systems Consultant
> National Council on
> Compensation Insurance
> 901 Peninsula Corporate Circle
> Boca Raton, FL 33487-1362
> (P) 561-893-2324
> chris_brown@ncci.com
>
>
>
> ________________________________
>
> The information contained in this e-mail message is intended only for the
> personal and confidential use of the recipient(s) named above. This message
> may be an attorney-client communication and/or work product and as such is
> privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that you have received this
> document in error and that any review, dissemination, distribution, or
> copying of this message is strictly prohibited. If you have received this
> communication in error, please notify us immediately by e-mail, and delete
> the original message.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: Tomcat 7 Web Service Shutdown Can't Stop Threads
Posted by Chris Brown <Ch...@ncci.com>.
Here is the code to shut down the pools. When I was running a single pool the issue did not occur. It's when I added the second pool that the issue started.
Arrays.asList(universalConnectionPoolManager.getConnectionPoolNames())
.forEach(poolName -> {
try {
universalConnectionPoolManager
.destroyConnectionPool(poolName);
} catch (Exception e) {
exceptionHandler.handleError(
e,
String.join(
":",
"Destroy Connection Pool Error",
poolName));
}
});
The jdbc driver and connection pool .jar files are in the tomcat lib folder and not in the applications lib folder.
I never tried Tomcat's version of connection pooling.
Thank you,
Chris Brown
Systems Consultant
National Council on
Compensation Insurance
901 Peninsula Corporate Circle
Boca Raton, FL 33487-1362
(P) 561-893-2324
chris_brown@ncci.com
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Monday, March 07, 2016 10:33 AM
To: Tomcat Users List
Subject: Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris,
On 3/7/16 9:51 AM, Chris Brown wrote:
> Tomcat is shutting down the web service when I deploy a new .war file.
Okay. You said you were closing the connection pool, though:
On 3/4/16 2:13 PM, Chris Brown wrote:
> I added a ServletContextListener to our web service. On the
> contextDestroyed method we destroy the connection pools. Our web
> service uses 2 connection pools.
So, how, precisely, are you shutting those pools down?
> There is no documentation on the threads and what they are for in
> oracle or tomcats documentation.
Tomcat would not contain information on a Vendor's product. A quick Google search for "oracle ojdbc7 ucp-worker-thread" yields the following very relevant result:
http://stackoverflow.com/questions/9671270/oracle-ucp-driver-and-tomcat-
threads-failing-to-stop
Also, searching for the OJDBC manual has a FAQ which mentions threads and their lifetimes:
http://www.oracleimg.com/technetwork/articles/sql/jdbc-faq-090281.html
> The threads are not shutting down and they stay alive when the web
> service is shut down. I know this because I count the number of
> threads each time the web service context is destroyed and number
> keeps growing for each deployment of the .war file.
Oh, I believe you that they are piling up.
Is there a particular reason you are using Oracle's connection pool rather than the one provided by the container (Tomcat)? Tomcat's pool won't leak like this one does. Then again, the Oracle driver itself might still leak, but it will be less leaky.
It's possible that the driver needs to be loaded before the application in order to avoid pinning the ClassLoader into memory.
That can be done with Tomcat's configuration, but I'd only recommend that you do this if nothing else is working.
I'd still like to know that those other threads (e.g. Thread-0) are doin g.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbdnygACgkQ9CaO5/Lv0PCClgCfUmDNGz68sT6beiMOwx2Pkewb
UGcAoJNpSVJ8wagZRiGZMS9LxO3g0KY9
=qWGg
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
________________________________
The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.
Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris,
On 3/7/16 9:51 AM, Chris Brown wrote:
> Tomcat is shutting down the web service when I deploy a new .war
> file.
Okay. You said you were closing the connection pool, though:
On 3/4/16 2:13 PM, Chris Brown wrote:
> I added a ServletContextListener to our web service. On the
> contextDestroyed method we destroy the connection pools. Our web
> service uses 2 connection pools.
So, how, precisely, are you shutting those pools down?
> There is no documentation on the threads and what they are for in
> oracle or tomcats documentation.
Tomcat would not contain information on a Vendor's product. A quick
Google search for "oracle ojdbc7 ucp-worker-thread" yields the
following very relevant result:
http://stackoverflow.com/questions/9671270/oracle-ucp-driver-and-tomcat-
threads-failing-to-stop
Also, searching for the OJDBC manual has a FAQ which mentions threads
and their lifetimes:
http://www.oracleimg.com/technetwork/articles/sql/jdbc-faq-090281.html
> The threads are not shutting down and they stay alive when the web
> service is shut down. I know this because I count the number of
> threads each time the web service context is destroyed and number
> keeps growing for each deployment of the .war file.
Oh, I believe you that they are piling up.
Is there a particular reason you are using Oracle's connection pool
rather than the one provided by the container (Tomcat)? Tomcat's pool
won't leak like this one does. Then again, the Oracle driver itself
might still leak, but it will be less leaky.
It's possible that the driver needs to be loaded before the
application in order to avoid pinning the ClassLoader into memory.
That can be done with Tomcat's configuration, but I'd only recommend
that you do this if nothing else is working.
I'd still like to know that those other threads (e.g. Thread-0) are doin
g.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbdnygACgkQ9CaO5/Lv0PCClgCfUmDNGz68sT6beiMOwx2Pkewb
UGcAoJNpSVJ8wagZRiGZMS9LxO3g0KY9
=qWGg
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 7 Web Service Shutdown Can't Stop Threads
Posted by Chris Brown <Ch...@ncci.com>.
Tomcat is shutting down the web service when I deploy a new .war file. There is no documentation on the threads and what they are for in oracle or tomcats documentation.
The threads are not shutting down and they stay alive when the web service is shut down. I know this because I count the number of threads each time the web service context is destroyed and number keeps growing for each deployment of the .war file.
Thank you,
Chris Brown
Systems Consultant
National Council on
Compensation Insurance
901 Peninsula Corporate Circle
Boca Raton, FL 33487-1362
(P) 561-893-2324
chris_brown@ncci.com
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Monday, March 07, 2016 9:47 AM
To: Tomcat Users List
Subject: Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris,
On 3/7/16 8:26 AM, Chris Brown wrote:
> The .jar comes with the Oracle 12c client. Oracle JDBC Drivers
> release 12.1.0.2.0
Thanks.
> I upgraded to tomcat 8. The latest stable version. The issue still
> occurred, but the messages in the logs were "Warnings"
> instead of being "SEVERE" errors.
You said that.
How about taking a thread dump after your web application stops, to see a few things:
1. Whether this is a spurious notification, since Tomcat was just going down faster than the threads were able to (gracefully) shut down
2. What those threads are doing, especially the ones with the somewhat anonymous names
You haven't said how you were shutting-down the thread pool.
You also haven't said whether you read any or Oracle's documentation about what those threads are doing or how they should be terminated.
- -chris
> -----Original Message----- From: Christopher Schultz
> [mailto:chris@christopherschultz.net] Sent: Friday, March 04, 2016
> 4:33 PM To: Tomcat Users List Subject: Re: Tomcat 7 Web Service
> Shutdown Can't Stop Threads
>
> Chris,
>
> On 3/4/16 2:13 PM, Chris Brown wrote:
>> Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle 12c
>> Oracle Driver: ojdbc7
>
> That doesn't look like an Oracle JDBC driver version I've seen before.
> Can you clarify?
>
>> JDK version: 1.8.0_31
>
> You might want to upgrade Tomcat and Java. Those are somewhat old.
>
>> Issue Description: When tomcat shuts down our web service during a
>> hot deployment threads are left running that tomcat cannot shutdown.
>> I believe the threads are from the universal connection pool. We use
>> oracle's universal connection pool (ucp.jar). After multiple
>> deployments the threads build up and cause high CPU usage from the 2
>> tomcat threads.
>
>> Log Error Examples: Feb 08, 2016 8:00:25 PM
>> org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application [/RatingService]
>> appears to have started a thread named [UCP-worker-thread-3] but has
>> failed to stop it. This is very likely to create a memory leak. Feb
>> 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application [/RatingService]
>> appears to have started a thread named [Thread-438] but has failed to
>> stop it. This is very likely to create a memory leak. Feb 27, 2016
>> 6:15:14 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application [/RatingService]
>> appears to have started a thread named
>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockRelea
s
>
>>
er]
>> but has failed to stop it. This is very likely to create a
>> memory leak. Feb 27, 2016 6:15:14 PM
>> org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application
>> [/RatingService] appears to have started a thread named
>> [ForkJoinPool.commonPool-worker-32] but has failed to stop it.
>> This is very likely to create a memory leak. Feb 27, 2016 6:15:14
>> PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application
>> [/RatingService] appears to have started a thread named
>> [Timer-121] but has failed to stop it. This is very likely to
>> create a memory leak.
>
>> Correction made to code to try and fix issue: I added a
>> ServletContextListener to our web service. On the
>> contextDestroyed method we destroy the connection pools. Our web
>> service uses 2 connection pools.
>
> How exactly are you destroying the connection pools?
>
>> I also upgraded to Tomcat 8 and the issue still occurred, but
>> the messages are now warnings instead of SEVERE errors.
>
> That's just an effect of Tomcat downgrading the log level under
> which it logs those issues.
>
>> I've searched the web and have tried many suggestions that have
>> not helped except destroying the connection pools, but the issue
>> still occurs. Please help!!!
>
> After you get those messages in the log, if you do a thread dump on
> the JVM, do you still see those threads running?
>
> Generally speaking, Tomcat should be able to shut down even with
> the above errors. Hopefully, none of those threads are non-daemon
> threads, meaning that they shouldn't stop the JVM from stopping
> once the JVM's main thread terminates. If your web application
> leaves those threads running after deployment, then multiple
> re-deployments without a Tomcat/JVM restart will likely result in a
> large memory leak. (Those threads will pin the web application's
> ClassLoader in memory along with all the classes it ever loaded,
> and these will pile-up until you have exhausted your permgen or
> other heap spaces).
>
>> [UCP-worker-thread-3]
>
> This is almost certainly something form Oracle's UCP
>
>> [Thread-438]
>
> This could be anything. A thread dump could tell you what it might
> be.
>
>
> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleas
er
>
>
]
>
> Obviously, this is a thread from Oracle's JDBC driver.
>
>> [ForkJoinPool.commonPool-worker-32]
>
> Maybe something from UCP? Not sure.
>
>> [Timer-121]
>
> This is usually a thread started by TimerTask, a stock Java class.
> But any component may have created this thread. It might be worth
> searching your own code to see if a TimerTask is being launched and
> never shut-down when your web application is shutting down. Make
> arrangements to make sure that you shut these down when the
> application is going out of service.
>
> For the Oracle-specific threads, read the documentation or search
> online for how to stop those threads as the application is coming
> down. There could be a bug in the library(ies) that might be fixed
> in a more recent version.
>
> -chris
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ________________________________
>
> The information contained in this e-mail message is intended only
> for the personal and confidential use of the recipient(s) named
> above. This message may be an attorney-client communication and/or
> work product and as such is privileged and confidential. If the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and
> that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify us immediately by e-mail, and
> delete the original message.
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbdlFQACgkQ9CaO5/Lv0PDJTwCgnvouJT6lwijxCJnymRAP+SAe
wFsAoKqbcrXDoiAV0RccWdFNIMHiE99N
=YMav
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
________________________________
The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.
Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris,
On 3/7/16 8:26 AM, Chris Brown wrote:
> The .jar comes with the Oracle 12c client. Oracle JDBC Drivers
> release 12.1.0.2.0
Thanks.
> I upgraded to tomcat 8. The latest stable version. The issue
> still occurred, but the messages in the logs were "Warnings"
> instead of being "SEVERE" errors.
You said that.
How about taking a thread dump after your web application stops, to
see a few things:
1. Whether this is a spurious notification, since Tomcat was just
going down faster than the threads were able to (gracefully) shut down
2. What those threads are doing, especially the ones with the somewhat
anonymous names
You haven't said how you were shutting-down the thread pool.
You also haven't said whether you read any or Oracle's documentation
about what those threads are doing or how they should be terminated.
- -chris
> -----Original Message----- From: Christopher Schultz
> [mailto:chris@christopherschultz.net] Sent: Friday, March 04, 2016
> 4:33 PM To: Tomcat Users List Subject: Re: Tomcat 7 Web Service
> Shutdown Can't Stop Threads
>
> Chris,
>
> On 3/4/16 2:13 PM, Chris Brown wrote:
>> Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle
>> 12c Oracle Driver: ojdbc7
>
> That doesn't look like an Oracle JDBC driver version I've seen
> before. Can you clarify?
>
>> JDK version: 1.8.0_31
>
> You might want to upgrade Tomcat and Java. Those are somewhat old.
>
>> Issue Description: When tomcat shuts down our web service during
>> a hot deployment threads are left running that tomcat cannot
>> shutdown. I believe the threads are from the universal connection
>> pool. We use oracle's universal connection pool (ucp.jar). After
>> multiple deployments the threads build up and cause high CPU
>> usage from the 2 tomcat threads.
>
>> Log Error Examples: Feb 08, 2016 8:00:25 PM
>> org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application
>> [/RatingService] appears to have started a thread named
>> [UCP-worker-thread-3] but has failed to stop it. This is very
>> likely to create a memory leak. Feb 08, 2016 8:00:25 PM
>> org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application
>> [/RatingService] appears to have started a thread named
>> [Thread-438] but has failed to stop it. This is very likely to
>> create a memory leak. Feb 27, 2016 6:15:14 PM
>> org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application
>> [/RatingService] appears to have started a thread named
>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockRelea
s
>
>>
er]
>> but has failed to stop it. This is very likely to create a
>> memory leak. Feb 27, 2016 6:15:14 PM
>> org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application
>> [/RatingService] appears to have started a thread named
>> [ForkJoinPool.commonPool-worker-32] but has failed to stop it.
>> This is very likely to create a memory leak. Feb 27, 2016 6:15:14
>> PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads SEVERE: The web application
>> [/RatingService] appears to have started a thread named
>> [Timer-121] but has failed to stop it. This is very likely to
>> create a memory leak.
>
>> Correction made to code to try and fix issue: I added a
>> ServletContextListener to our web service. On the
>> contextDestroyed method we destroy the connection pools. Our web
>> service uses 2 connection pools.
>
> How exactly are you destroying the connection pools?
>
>> I also upgraded to Tomcat 8 and the issue still occurred, but
>> the messages are now warnings instead of SEVERE errors.
>
> That's just an effect of Tomcat downgrading the log level under
> which it logs those issues.
>
>> I've searched the web and have tried many suggestions that have
>> not helped except destroying the connection pools, but the issue
>> still occurs. Please help!!!
>
> After you get those messages in the log, if you do a thread dump on
> the JVM, do you still see those threads running?
>
> Generally speaking, Tomcat should be able to shut down even with
> the above errors. Hopefully, none of those threads are non-daemon
> threads, meaning that they shouldn't stop the JVM from stopping
> once the JVM's main thread terminates. If your web application
> leaves those threads running after deployment, then multiple
> re-deployments without a Tomcat/JVM restart will likely result in a
> large memory leak. (Those threads will pin the web application's
> ClassLoader in memory along with all the classes it ever loaded,
> and these will pile-up until you have exhausted your permgen or
> other heap spaces).
>
>> [UCP-worker-thread-3]
>
> This is almost certainly something form Oracle's UCP
>
>> [Thread-438]
>
> This could be anything. A thread dump could tell you what it might
> be.
>
>
> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleas
er
>
>
]
>
> Obviously, this is a thread from Oracle's JDBC driver.
>
>> [ForkJoinPool.commonPool-worker-32]
>
> Maybe something from UCP? Not sure.
>
>> [Timer-121]
>
> This is usually a thread started by TimerTask, a stock Java class.
> But any component may have created this thread. It might be worth
> searching your own code to see if a TimerTask is being launched and
> never shut-down when your web application is shutting down. Make
> arrangements to make sure that you shut these down when the
> application is going out of service.
>
> For the Oracle-specific threads, read the documentation or search
> online for how to stop those threads as the application is coming
> down. There could be a bug in the library(ies) that might be fixed
> in a more recent version.
>
> -chris
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ________________________________
>
> The information contained in this e-mail message is intended only
> for the personal and confidential use of the recipient(s) named
> above. This message may be an attorney-client communication and/or
> work product and as such is privileged and confidential. If the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and
> that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify us immediately by e-mail, and
> delete the original message.
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbdlFQACgkQ9CaO5/Lv0PDJTwCgnvouJT6lwijxCJnymRAP+SAe
wFsAoKqbcrXDoiAV0RccWdFNIMHiE99N
=YMav
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 7 Web Service Shutdown Can't Stop Threads
Posted by Chris Brown <Ch...@ncci.com>.
The .jar comes with the Oracle 12c client. Oracle JDBC Drivers release 12.1.0.2.0
I upgraded to tomcat 8. The latest stable version. The issue still occurred, but the messages in the logs were "Warnings" instead of being "SEVERE" errors.
Thank you,
Chris Brown
Systems Consultant
National Council on
Compensation Insurance
901 Peninsula Corporate Circle
Boca Raton, FL 33487-1362
(P) 561-893-2324
chris_brown@ncci.com
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Friday, March 04, 2016 4:33 PM
To: Tomcat Users List
Subject: Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris,
On 3/4/16 2:13 PM, Chris Brown wrote:
> Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle 12c
> Oracle Driver: ojdbc7
That doesn't look like an Oracle JDBC driver version I've seen before.
Can you clarify?
> JDK version: 1.8.0_31
You might want to upgrade Tomcat and Java. Those are somewhat old.
> Issue Description: When tomcat shuts down our web service during a hot
> deployment threads are left running that tomcat cannot shutdown. I
> believe the threads are from the universal connection pool. We use
> oracle's universal connection pool (ucp.jar). After multiple
> deployments the threads build up and cause high CPU usage from the 2
> tomcat threads.
>
> Log Error Examples: Feb 08, 2016 8:00:25 PM
> org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named [UCP-worker-thread-3] but has
> failed to stop it. This is very likely to create a memory leak. Feb
> 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named [Thread-438] but has failed to
> stop it. This is very likely to create a memory leak. Feb 27,
> 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named
> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleas
er]
> but has failed to stop it. This is very likely to create a memory
> leak. Feb 27, 2016 6:15:14 PM
> org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named
> [ForkJoinPool.commonPool-worker-32] but has failed to stop it. This is
> very likely to create a memory leak. Feb 27, 2016 6:15:14 PM
> org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named [Timer-121] but has failed to
> stop it. This is very likely to create a memory leak.
>
> Correction made to code to try and fix issue: I added a
> ServletContextListener to our web service. On the contextDestroyed
> method we destroy the connection pools. Our web service uses 2
> connection pools.
How exactly are you destroying the connection pools?
> I also upgraded to Tomcat 8 and the issue still occurred, but the
> messages are now warnings instead of SEVERE errors.
That's just an effect of Tomcat downgrading the log level under which it logs those issues.
> I've searched the web and have tried many suggestions that have not
> helped except destroying the connection pools, but the issue still
> occurs. Please help!!!
After you get those messages in the log, if you do a thread dump on the JVM, do you still see those threads running?
Generally speaking, Tomcat should be able to shut down even with the above errors. Hopefully, none of those threads are non-daemon threads, meaning that they shouldn't stop the JVM from stopping once the JVM's main thread terminates. If your web application leaves those threads running after deployment, then multiple re-deployments without a Tomcat/JVM restart will likely result in a large memory leak. (Those threads will pin the web application's ClassLoader in memory along with all the classes it ever loaded, and these will pile-up until you have exhausted your permgen or other heap spaces).
> [UCP-worker-thread-3]
This is almost certainly something form Oracle's UCP
> [Thread-438]
This could be anything. A thread dump could tell you what it might be.
>
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser
]
Obviously, this is a thread from Oracle's JDBC driver.
> [ForkJoinPool.commonPool-worker-32]
Maybe something from UCP? Not sure.
> [Timer-121]
This is usually a thread started by TimerTask, a stock Java class. But any component may have created this thread. It might be worth searching your own code to see if a TimerTask is being launched and never shut-down when your web application is shutting down. Make arrangements to make sure that you shut these down when the application is going out of service.
For the Oracle-specific threads, read the documentation or search online for how to stop those threads as the application is coming down. There could be a bug in the library(ies) that might be fixed in a more recent version.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbZ/woACgkQ9CaO5/Lv0PBwvwCeLblpAKfIen1G0nmZvoqkLgWe
qCoAnR84mW3an4+BOVKvzuQTI2fTTodx
=5+gA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
________________________________
The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris,
On 3/4/16 2:13 PM, Chris Brown wrote:
> Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle
> 12c Oracle Driver: ojdbc7
That doesn't look like an Oracle JDBC driver version I've seen before.
Can you clarify?
> JDK version: 1.8.0_31
You might want to upgrade Tomcat and Java. Those are somewhat old.
> Issue Description: When tomcat shuts down our web service during a
> hot deployment threads are left running that tomcat cannot
> shutdown. I believe the threads are from the universal connection
> pool. We use oracle's universal connection pool (ucp.jar). After
> multiple deployments the threads build up and cause high CPU usage
> from the 2 tomcat threads.
>
> Log Error Examples: Feb 08, 2016 8:00:25 PM
> org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named [UCP-worker-thread-3] but
> has failed to stop it. This is very likely to create a memory
> leak. Feb 08, 2016 8:00:25 PM
> org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named [Thread-438] but has failed
> to stop it. This is very likely to create a memory leak. Feb 27,
> 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named
> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleas
er]
> but has failed to stop it. This is very likely to create a memory
> leak. Feb 27, 2016 6:15:14 PM
> org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named
> [ForkJoinPool.commonPool-worker-32] but has failed to stop it. This
> is very likely to create a memory leak. Feb 27, 2016 6:15:14 PM
> org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads SEVERE: The web application [/RatingService]
> appears to have started a thread named [Timer-121] but has failed
> to stop it. This is very likely to create a memory leak.
>
> Correction made to code to try and fix issue: I added a
> ServletContextListener to our web service. On the contextDestroyed
> method we destroy the connection pools. Our web service uses 2
> connection pools.
How exactly are you destroying the connection pools?
> I also upgraded to Tomcat 8 and the issue still occurred, but the
> messages are now warnings instead of SEVERE errors.
That's just an effect of Tomcat downgrading the log level under which
it logs those issues.
> I've searched the web and have tried many suggestions that have
> not helped except destroying the connection pools, but the issue
> still occurs. Please help!!!
After you get those messages in the log, if you do a thread dump on
the JVM, do you still see those threads running?
Generally speaking, Tomcat should be able to shut down even with the
above errors. Hopefully, none of those threads are non-daemon threads,
meaning that they shouldn't stop the JVM from stopping once the JVM's
main thread terminates. If your web application leaves those threads
running after deployment, then multiple re-deployments without a
Tomcat/JVM restart will likely result in a large memory leak. (Those
threads will pin the web application's ClassLoader in memory along
with all the classes it ever loaded, and these will pile-up until you
have exhausted your permgen or other heap spaces).
> [UCP-worker-thread-3]
This is almost certainly something form Oracle's UCP
> [Thread-438]
This could be anything. A thread dump could tell you what it might be.
>
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser
]
Obviously, this is a thread from Oracle's JDBC driver.
> [ForkJoinPool.commonPool-worker-32]
Maybe something from UCP? Not sure.
> [Timer-121]
This is usually a thread started by TimerTask, a stock Java class. But
any component may have created this thread. It might be worth
searching your own code to see if a TimerTask is being launched and
never shut-down when your web application is shutting down. Make
arrangements to make sure that you shut these down when the
application is going out of service.
For the Oracle-specific threads, read the documentation or search
online for how to stop those threads as the application is coming
down. There could be a bug in the library(ies) that might be fixed in
a more recent version.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbZ/woACgkQ9CaO5/Lv0PBwvwCeLblpAKfIen1G0nmZvoqkLgWe
qCoAnR84mW3an4+BOVKvzuQTI2fTTodx
=5+gA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org