You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Yuval Schwartz <yu...@gmail.com> on 2016/02/01 15:54:04 UTC
Re: WebAppClassLoaderBase.clearReferencesThreads warning
Hello Mark,
I think that the issue below was related to the way I was shutting down an
instance of ScheduledExecutorService.
I changed the way it is shutdown when the context is destroyed...I will
update here if I don't receive any more warnings.
However, I now get a warning:
WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor
The executor associated with thread pool [http-apr-8080] has not fully
shutdown. Some application threads may still be running
I am not sure if this is related or not?
When I do a heap dump with jstack I see http-apr-8080-exec-[1-10] but I
don't see any thread named http-apr-8080.
Any ideas on how to trouble shoot this?
Thanks.
On Sun, Jan 31, 2016 at 8:18 PM, Yuval Schwartz <yu...@gmail.com>
wrote:
>
>
> On Sunday, 31 January 2016, Mark Thomas <ma...@apache.org> wrote:
>
>> On 31/01/2016 08:06, Yuval Schwartz wrote:
>> > Hellow Mark,
>> >
>> > Thanks for the reply.
>> > Follow up questions below.
>> >
>> > On Fri, Jan 29, 2016 at 6:22 PM, Mark Thomas <ma...@apache.org> wrote:
>> >
>> >> On 29/01/2016 14:34, Yuval Schwartz wrote:
>> >>> Hello,
>> >>>
>> >>> tomcat version: 8.0.22
>> >>> java: jdk1.8.0_05
>> >>> server: Amazon Linux AMI
>> >>>
>> >>> I get the following warning message in my catalina log when I
>> undeploy a
>> >>> web application:
>> >>>
>> >>> *WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >>>
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
>> >> The
>> >>> web application [ROOT##002] appears to have started a thread named
>> >>> [pool-2-thread-1] but has failed to stop it. This is very likely to
>> >> create
>> >>> a memory leak. Stack trace of thread*
>> >>>
>> >>> As you can see, for some reason, I don't get a stack trace of the
>> thread.
>> >>> Therefore, I have no idea how to debug this warning.
>> >>>
>> >>> This particular warning was generated when tomcat detected an unused
>> >>> version and undeployed it (I set undeployOldVersions="true").
>> >>>
>> >>> Does anyone know how I can debug this warning. How can I know more
>> about
>> >>> this thread?
>> >> That looks like a thread from Commons Pool (possibly via DBCP). The
>> only
>> >> way to be sure you have a leak or not is to use a profiler. See
>> >>
>> >>
>> http://home.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>> >
>> >
>> > Is VisualVM a good profiler to start with considering my system?
>>
>> Sorry, don't know. Never used it. I use YourKit, mainly because they
>> kindly donate free licenses to OpenSource developers to use with their
>> OpenSource projects.
>
>
> Thank you Mark,
>
> I installed VisualVM as a profiler.
> I can now see all of the threads that tomcat is "using" (?not sure if
> that's the correct term) (there's 35 live threads and 33 daemon threads).
> Do you have any recommendations for how I might trouble shoot the original
> warning that I got? Would I do a "heap dump"?
> I have restarted tomcat since the time of the original warning message, so
> I can't expect to find the same thread, correct?
> Any tips?
>
> FYI: I never saw my cpu climbing or anything, I am troubleshooting this
> warning solely on seeing the "Warning" message. Is this overkill? Should I
> wait until something more severe occurs? (I have not yet gone live with my
> webapp but planned on doing so soon).
>
> Thanks.
>
>
>>
>> Mark
>>
>> > I would just like to confirm that this warning is not something serious.
>> > I am currently having a little trouble running VisualVM (jstatd) from a
>> > remote machine but can struggle with it a little more to try to get it
>> > working.
>> >
>> >
>> > Thanks.
>> >
>> >
>> >
>> >>
>> >>> I used the manager app and clicked "find leaks" but it said that there
>> >> were
>> >>> none.
>> >>> (Can someone also explain why I have to be cautious when using this
>> >> feature
>> >>> on production environments).
>> >>
>> >> It triggers a full GC which may (hopefully briefly) pause the entire
>> JVM.
>> >>
>> >> Mark
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >>
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
Re: WebAppClassLoaderBase.clearReferencesThreads warning
Posted by "Terence M. Bandoian" <te...@tmbsw.com>.
On 2/1/2016 10:12 AM, Yuval Schwartz wrote:
> Hello Terence,
>
> Thanks for the input.
> I shutdown the ScheduledExecutorService in the contextDestroyed method of
> the app's ServletContextListener class as well.
> I also call shutDownNow() followed by an if statement with
> !awaitTermination() as the condition.
>
> Are you sure that the new warning that I'm getting is also related to the
> ScheduledExecutorService?
> The warning again is:
>
> WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor
> The executor associated with thread pool [http-apr-8080] has not fully
> shutdown. Some application threads may still be running
>
> I didn't find much information about http-apr-8080 on google.
> When I do a thread dump while tomcat is running I see http-apr-8080-exec-1,
> http-apr-8080-exec-2,...http-apr-8080-exec-10.
> I'm not sure what these do? Are these threads the same as http-apr-8080
> that is referenced in the warning?
>
> How else can I go about debugging this message?
>
> Thank you
Hi, Yuval-
I'm not sure the new warning is related to the way the
ScheduledExecutorService is stopped. You mentioned though that the
previous warning was related so I thought I'd share my experience.
Here's the code I used:
public void contextDestroyed( ServletContextEvent sce )
{
if ( executor != null )
{
boolean isTerminated = false;
executor.shutdown();
do
{
try
{
isTerminated = executor.awaitTermination(
1, TimeUnit.SECONDS );
}
catch ( InterruptedException ignore )
{
}
}
while ( !isTerminated );
executor = null;
Thread.yield();
}
}
Notice the loop.
For the new warning, my suggestion would be to find out who owns the
thread in question. Can you do that with the profiler?
Hope that helps.
-Terence Bandoian
>
> On Mon, Feb 1, 2016 at 5:59 PM, Terence M. Bandoian <te...@tmbsw.com>
> wrote:
>
>> On 2/1/2016 8:54 AM, Yuval Schwartz wrote:
>>
>>> Hello Mark,
>>>
>>> I think that the issue below was related to the way I was shutting down an
>>> instance of ScheduledExecutorService.
>>> I changed the way it is shutdown when the context is destroyed...I will
>>> update here if I don't receive any more warnings.
>>>
>>> However, I now get a warning:
>>>
>>> WARNING [main] org.apache.tomcat.util.net
>>> .AbstractEndpoint.shutdownExecutor
>>> The executor associated with thread pool [http-apr-8080] has not fully
>>> shutdown. Some application threads may still be running
>>>
>>> I am not sure if this is related or not?
>>> When I do a heap dump with jstack I see http-apr-8080-exec-[1-10] but I
>>> don't see any thread named http-apr-8080.
>>> Any ideas on how to trouble shoot this?
>>>
>>> Thanks.
>>>
>>
>> Hi, Yuval-
>>
>> Where are you shutting down the ScheduledExecutorService? I ran into a
>> similar problem trying to stop a ScheduledExecutorService in the
>> contextDestroyed method of a ServletContextListener. Adding a call to
>> Thread.yield() after the executor was terminated removed the warning. To
>> terminate the executor, I used the shutdown() method followed by a loop
>> with a call to awaitTermination().
>>
>> -Terence Bandoian
>>
>>
>>
>>
>>> On Sun, Jan 31, 2016 at 8:18 PM, Yuval Schwartz <yuval.schwartz@gmail.com
>>> wrote:
>>>
>>>
>>>> On Sunday, 31 January 2016, Mark Thomas <ma...@apache.org> wrote:
>>>>
>>>> On 31/01/2016 08:06, Yuval Schwartz wrote:
>>>>>> Hellow Mark,
>>>>>>
>>>>>> Thanks for the reply.
>>>>>> Follow up questions below.
>>>>>>
>>>>>> On Fri, Jan 29, 2016 at 6:22 PM, Mark Thomas <ma...@apache.org> wrote:
>>>>>>
>>>>>> On 29/01/2016 14:34, Yuval Schwartz wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> tomcat version: 8.0.22
>>>>>>>> java: jdk1.8.0_05
>>>>>>>> server: Amazon Linux AMI
>>>>>>>>
>>>>>>>> I get the following warning message in my catalina log when I
>>>>>>>>
>>>>>>> undeploy a
>>>>>> web application:
>>>>>>>> *WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>>>>>>>>
>>>>>>>>
>>>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
>>>>>
>>>>>> The
>>>>>>>> web application [ROOT##002] appears to have started a thread named
>>>>>>>> [pool-2-thread-1] but has failed to stop it. This is very likely to
>>>>>>>>
>>>>>>> create
>>>>>>>
>>>>>>>> a memory leak. Stack trace of thread*
>>>>>>>>
>>>>>>>> As you can see, for some reason, I don't get a stack trace of the
>>>>>>>>
>>>>>>> thread.
>>>>>> Therefore, I have no idea how to debug this warning.
>>>>>>>> This particular warning was generated when tomcat detected an unused
>>>>>>>> version and undeployed it (I set undeployOldVersions="true").
>>>>>>>>
>>>>>>>> Does anyone know how I can debug this warning. How can I know more
>>>>>>>>
>>>>>>> about
>>>>>> this thread?
>>>>>>> That looks like a thread from Commons Pool (possibly via DBCP). The
>>>>>>>
>>>>>> only
>>>>>> way to be sure you have a leak or not is to use a profiler. See
>>>>>>>
>>>>>>>
>>>>> http://home.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>>>>
>>>>>> Is VisualVM a good profiler to start with considering my system?
>>>>>>
>>>>> Sorry, don't know. Never used it. I use YourKit, mainly because they
>>>>> kindly donate free licenses to OpenSource developers to use with their
>>>>> OpenSource projects.
>>>>>
>>>> Thank you Mark,
>>>>
>>>> I installed VisualVM as a profiler.
>>>> I can now see all of the threads that tomcat is "using" (?not sure if
>>>> that's the correct term) (there's 35 live threads and 33 daemon threads).
>>>> Do you have any recommendations for how I might trouble shoot the
>>>> original
>>>> warning that I got? Would I do a "heap dump"?
>>>> I have restarted tomcat since the time of the original warning message,
>>>> so
>>>> I can't expect to find the same thread, correct?
>>>> Any tips?
>>>>
>>>> FYI: I never saw my cpu climbing or anything, I am troubleshooting this
>>>> warning solely on seeing the "Warning" message. Is this overkill? Should
>>>> I
>>>> wait until something more severe occurs? (I have not yet gone live with
>>>> my
>>>> webapp but planned on doing so soon).
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> Mark
>>>>> I would just like to confirm that this warning is not something serious.
>>>>>> I am currently having a little trouble running VisualVM (jstatd) from a
>>>>>> remote machine but can struggle with it a little more to try to get it
>>>>>> working.
>>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I used the manager app and clicked "find leaks" but it said that there
>>>>>>> were
>>>>>>>
>>>>>>>> none.
>>>>>>>> (Can someone also explain why I have to be cautious when using this
>>>>>>>>
>>>>>>> feature
>>>>>>>
>>>>>>>> on production environments).
>>>>>>>>
>>>>>>> It triggers a full GC which may (hopefully briefly) pause the entire
>>>>>>>
>>>>>> JVM.
>>>>>> Mark
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebAppClassLoaderBase.clearReferencesThreads warning
Posted by Yuval Schwartz <yu...@gmail.com>.
Hello Terence,
Can you explain what you mean when you say "who owns the thread"?
On Tue, Feb 2, 2016 at 5:18 PM, Terence M. Bandoian <te...@tmbsw.com>
wrote:
> On 2/2/2016 2:49 AM, Yuval Schwartz wrote:
>
>>
>>
>> On Mon, Feb 1, 2016 at 7:36 PM, Terence M. Bandoian <terence@tmbsw.com
>> <ma...@tmbsw.com>> wrote:
>>
>> On 2/1/2016 10:12 AM, Yuval Schwartz wrote:
>>
>> Hello Terence,
>>
>> Thanks for the input.
>> I shutdown the ScheduledExecutorService in the
>> contextDestroyed method of
>> the app's ServletContextListener class as well.
>> I also call shutDownNow() followed by an if statement with
>> !awaitTermination() as the condition.
>>
>> Are you sure that the new warning that I'm getting is also
>> related to the
>> ScheduledExecutorService?
>> The warning again is:
>>
>> WARNING [main] org.apache.tomcat.util.net
>> <http://org.apache.tomcat.util.net
>> >.AbstractEndpoint.shutdownExecutor
>> The executor associated with thread pool [http-apr-8080] has
>> not fully
>> shutdown. Some application threads may still be running
>>
>> I didn't find much information about http-apr-8080 on google.
>> When I do a thread dump while tomcat is running I see
>> http-apr-8080-exec-1,
>> http-apr-8080-exec-2,...http-apr-8080-exec-10.
>> I'm not sure what these do? Are these threads the same as
>> http-apr-8080
>> that is referenced in the warning?
>>
>> How else can I go about debugging this message?
>>
>> Thank you
>>
>>
>>
>> Hi, Yuval-
>>
>> I'm not sure the new warning is related to the way the
>> ScheduledExecutorService is stopped. You mentioned though that
>> the previous warning was related so I thought I'd share my
>> experience. Here's the code I used:
>>
>> public void contextDestroyed( ServletContextEvent sce )
>> {
>> if ( executor != null )
>> {
>> boolean isTerminated = false;
>>
>> executor.shutdown();
>>
>> do
>> {
>> try
>> {
>> isTerminated = executor.awaitTermination(
>> 1, TimeUnit.SECONDS );
>> }
>> catch ( InterruptedException ignore )
>> {
>> }
>> }
>> while ( !isTerminated );
>>
>> executor = null;
>>
>> Thread.yield();
>>
>>
>> Java docs say use of this method is rarely necessary...so I'm a little
>> hesitant to use it.
>>
>> }
>> }
>>
>> Notice the loop.
>>
>> For the new warning, my suggestion would be to find out who owns
>> the thread in question. Can you do that with the profiler?
>>
>>
>> Thanks. I don't use a profiler. I'm using Jstack, and at the time the
>> application is running, when I do a thread dump, I don't see this thread
>> ("http-apr-8080").
>>
>> I also did some testing on my development environment and printing the
>> threads right before tomcat shuts down prints some threads, but again, none
>> are named "http-apr-8080" (although some are named http-apr-8080-exec-...).
>>
>>
>> Hope that helps.
>>
>> -Terence Bandoian
>>
>>
>>
>>
>> On Mon, Feb 1, 2016 at 5:59 PM, Terence M. Bandoian
>> <terence@tmbsw.com <ma...@tmbsw.com>>
>> wrote:
>>
>> On 2/1/2016 8:54 AM, Yuval Schwartz wrote:
>>
>> Hello Mark,
>>
>> I think that the issue below was related to the way I
>> was shutting down an
>> instance of ScheduledExecutorService.
>> I changed the way it is shutdown when the context is
>> destroyed...I will
>> update here if I don't receive any more warnings.
>>
>> However, I now get a warning:
>>
>> WARNING [main] org.apache.tomcat.util.net
>> <http://org.apache.tomcat.util.net>
>> .AbstractEndpoint.shutdownExecutor
>> The executor associated with thread pool
>> [http-apr-8080] has not fully
>> shutdown. Some application threads may still be running
>>
>> I am not sure if this is related or not?
>> When I do a heap dump with jstack I see
>> http-apr-8080-exec-[1-10] but I
>> don't see any thread named http-apr-8080.
>> Any ideas on how to trouble shoot this?
>>
>> Thanks.
>>
>>
>> Hi, Yuval-
>>
>> Where are you shutting down the ScheduledExecutorService?
>> I ran into a
>> similar problem trying to stop a ScheduledExecutorService
>> in the
>> contextDestroyed method of a ServletContextListener.
>> Adding a call to
>> Thread.yield() after the executor was terminated removed
>> the warning. To
>> terminate the executor, I used the shutdown() method
>> followed by a loop
>> with a call to awaitTermination().
>>
>> -Terence Bandoian
>>
>>
>>
>>
>> On Sun, Jan 31, 2016 at 8:18 PM, Yuval Schwartz
>> <yuval.schwartz@gmail.com
>> <ma...@gmail.com>
>> wrote:
>>
>>
>> On Sunday, 31 January 2016, Mark Thomas
>> <markt@apache.org <ma...@apache.org>> wrote:
>>
>> On 31/01/2016 08:06, Yuval Schwartz wrote:
>>
>> Hellow Mark,
>>
>> Thanks for the reply.
>> Follow up questions below.
>>
>> On Fri, Jan 29, 2016 at 6:22 PM, Mark
>> Thomas <markt@apache.org
>> <ma...@apache.org>> wrote:
>>
>> On 29/01/2016 14:34, Yuval Schwartz wrote:
>>
>> Hello,
>>
>> tomcat version: 8.0.22
>> java: jdk1.8.0_05
>> server: Amazon Linux AMI
>>
>> I get the following warning
>> message in my catalina log when I
>>
>> undeploy a
>>
>> web application:
>>
>> *WARNING
>>
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>>
>>
>>
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
>>
>> The
>>
>> web application [ROOT##002]
>> appears to have started a thread named
>> [pool-2-thread-1] but has failed
>> to stop it. This is very likely to
>>
>> create
>>
>> a memory leak. Stack trace of thread*
>>
>> As you can see, for some reason, I
>> don't get a stack trace of the
>>
>> thread.
>>
>> Therefore, I have no idea how to debug
>> this warning.
>>
>> This particular warning was
>> generated when tomcat detected an
>> unused
>> version and undeployed it (I set
>> undeployOldVersions="true").
>>
>> Does anyone know how I can debug
>> this warning. How can I know more
>>
>> about
>>
>> this thread?
>>
>> That looks like a thread from Commons
>> Pool (possibly via DBCP). The
>>
>> only
>> way to be sure you have a leak or not is
>> to use a profiler. See
>>
>>
>>
>>
>> http://home.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>> <
>> http://home.apache.org/%7Emarkt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>> >
>>
>>
>> Is VisualVM a good profiler to start with
>> considering my system?
>>
>> Sorry, don't know. Never used it. I use
>> YourKit, mainly because they
>> kindly donate free licenses to OpenSource
>> developers to use with their
>> OpenSource projects.
>>
>> Thank you Mark,
>>
>> I installed VisualVM as a profiler.
>> I can now see all of the threads that tomcat is
>> "using" (?not sure if
>> that's the correct term) (there's 35 live threads
>> and 33 daemon threads).
>> Do you have any recommendations for how I might
>> trouble shoot the
>> original
>> warning that I got? Would I do a "heap dump"?
>> I have restarted tomcat since the time of the
>> original warning message,
>> so
>> I can't expect to find the same thread, correct?
>> Any tips?
>>
>> FYI: I never saw my cpu climbing or anything, I am
>> troubleshooting this
>> warning solely on seeing the "Warning" message. Is
>> this overkill? Should
>> I
>> wait until something more severe occurs? (I have
>> not yet gone live with
>> my
>> webapp but planned on doing so soon).
>>
>> Thanks.
>>
>>
>> Mark
>>
>> I would just like to confirm that this warning
>> is not something serious.
>>
>> I am currently having a little trouble
>> running VisualVM (jstatd) from a
>> remote machine but can struggle with it a
>> little more to try to get it
>> working.
>>
>>
>> Thanks.
>>
>>
>>
>> I used the manager app and clicked "find
>> leaks" but it said that there
>>
>> were
>>
>> none.
>> (Can someone also explain why I
>> have to be cautious when using this
>>
>> feature
>>
>> on production environments).
>>
>> It triggers a full GC which may
>> (hopefully briefly) pause the entire
>>
>> JVM.
>> Mark
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> users-unsubscribe@tomcat.apache.org
>> <mailto:
>> users-unsubscribe@tomcat.apache.org>
>> For additional commands, e-mail:
>> users-help@tomcat.apache.org
>> <ma...@tomcat.apache.org>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> users-unsubscribe@tomcat.apache.org
>> <ma...@tomcat.apache.org>
>> For additional commands, e-mail:
>> users-help@tomcat.apache.org
>> <ma...@tomcat.apache.org>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> users-unsubscribe@tomcat.apache.org
>> <ma...@tomcat.apache.org>
>> For additional commands, e-mail:
>> users-help@tomcat.apache.org
>> <ma...@tomcat.apache.org>
>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: WebAppClassLoaderBase.clearReferencesThreads warning
Posted by "Terence M. Bandoian" <te...@tmbsw.com>.
On 2/2/2016 2:49 AM, Yuval Schwartz wrote:
>
>
> On Mon, Feb 1, 2016 at 7:36 PM, Terence M. Bandoian <terence@tmbsw.com
> <ma...@tmbsw.com>> wrote:
>
> On 2/1/2016 10:12 AM, Yuval Schwartz wrote:
>
> Hello Terence,
>
> Thanks for the input.
> I shutdown the ScheduledExecutorService in the
> contextDestroyed method of
> the app's ServletContextListener class as well.
> I also call shutDownNow() followed by an if statement with
> !awaitTermination() as the condition.
>
> Are you sure that the new warning that I'm getting is also
> related to the
> ScheduledExecutorService?
> The warning again is:
>
> WARNING [main] org.apache.tomcat.util.net
> <http://org.apache.tomcat.util.net>.AbstractEndpoint.shutdownExecutor
> The executor associated with thread pool [http-apr-8080] has
> not fully
> shutdown. Some application threads may still be running
>
> I didn't find much information about http-apr-8080 on google.
> When I do a thread dump while tomcat is running I see
> http-apr-8080-exec-1,
> http-apr-8080-exec-2,...http-apr-8080-exec-10.
> I'm not sure what these do? Are these threads the same as
> http-apr-8080
> that is referenced in the warning?
>
> How else can I go about debugging this message?
>
> Thank you
>
>
>
> Hi, Yuval-
>
> I'm not sure the new warning is related to the way the
> ScheduledExecutorService is stopped. You mentioned though that
> the previous warning was related so I thought I'd share my
> experience. Here's the code I used:
>
> public void contextDestroyed( ServletContextEvent sce )
> {
> if ( executor != null )
> {
> boolean isTerminated = false;
>
> executor.shutdown();
>
> do
> {
> try
> {
> isTerminated = executor.awaitTermination(
> 1, TimeUnit.SECONDS );
> }
> catch ( InterruptedException ignore )
> {
> }
> }
> while ( !isTerminated );
>
> executor = null;
>
> Thread.yield();
>
>
> Java docs say use of this method is rarely necessary...so I'm a little
> hesitant to use it.
>
> }
> }
>
> Notice the loop.
>
> For the new warning, my suggestion would be to find out who owns
> the thread in question. Can you do that with the profiler?
>
>
> Thanks. I don't use a profiler. I'm using Jstack, and at the time the
> application is running, when I do a thread dump, I don't see this
> thread ("http-apr-8080").
>
> I also did some testing on my development environment and printing the
> threads right before tomcat shuts down prints some threads, but again,
> none are named "http-apr-8080" (although some are named
> http-apr-8080-exec-...).
>
>
> Hope that helps.
>
> -Terence Bandoian
>
>
>
>
> On Mon, Feb 1, 2016 at 5:59 PM, Terence M. Bandoian
> <terence@tmbsw.com <ma...@tmbsw.com>>
> wrote:
>
> On 2/1/2016 8:54 AM, Yuval Schwartz wrote:
>
> Hello Mark,
>
> I think that the issue below was related to the way I
> was shutting down an
> instance of ScheduledExecutorService.
> I changed the way it is shutdown when the context is
> destroyed...I will
> update here if I don't receive any more warnings.
>
> However, I now get a warning:
>
> WARNING [main] org.apache.tomcat.util.net
> <http://org.apache.tomcat.util.net>
> .AbstractEndpoint.shutdownExecutor
> The executor associated with thread pool
> [http-apr-8080] has not fully
> shutdown. Some application threads may still be running
>
> I am not sure if this is related or not?
> When I do a heap dump with jstack I see
> http-apr-8080-exec-[1-10] but I
> don't see any thread named http-apr-8080.
> Any ideas on how to trouble shoot this?
>
> Thanks.
>
>
> Hi, Yuval-
>
> Where are you shutting down the ScheduledExecutorService?
> I ran into a
> similar problem trying to stop a ScheduledExecutorService
> in the
> contextDestroyed method of a ServletContextListener.
> Adding a call to
> Thread.yield() after the executor was terminated removed
> the warning. To
> terminate the executor, I used the shutdown() method
> followed by a loop
> with a call to awaitTermination().
>
> -Terence Bandoian
>
>
>
>
> On Sun, Jan 31, 2016 at 8:18 PM, Yuval Schwartz
> <yuval.schwartz@gmail.com
> <ma...@gmail.com>
> wrote:
>
>
> On Sunday, 31 January 2016, Mark Thomas
> <markt@apache.org <ma...@apache.org>> wrote:
>
> On 31/01/2016 08:06, Yuval Schwartz wrote:
>
> Hellow Mark,
>
> Thanks for the reply.
> Follow up questions below.
>
> On Fri, Jan 29, 2016 at 6:22 PM, Mark
> Thomas <markt@apache.org
> <ma...@apache.org>> wrote:
>
> On 29/01/2016 14:34, Yuval Schwartz wrote:
>
> Hello,
>
> tomcat version: 8.0.22
> java: jdk1.8.0_05
> server: Amazon Linux AMI
>
> I get the following warning
> message in my catalina log when I
>
> undeploy a
>
> web application:
>
> *WARNING
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>
>
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
>
> The
>
> web application [ROOT##002]
> appears to have started a thread named
> [pool-2-thread-1] but has failed
> to stop it. This is very likely to
>
> create
>
> a memory leak. Stack trace of thread*
>
> As you can see, for some reason, I
> don't get a stack trace of the
>
> thread.
>
> Therefore, I have no idea how to debug
> this warning.
>
> This particular warning was
> generated when tomcat detected an
> unused
> version and undeployed it (I set
> undeployOldVersions="true").
>
> Does anyone know how I can debug
> this warning. How can I know more
>
> about
>
> this thread?
>
> That looks like a thread from Commons
> Pool (possibly via DBCP). The
>
> only
> way to be sure you have a leak or not is
> to use a profiler. See
>
>
>
> http://home.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
> <http://home.apache.org/%7Emarkt/presentations/2010-11-04-Memory-Leaks-60mins.pdf>
>
> Is VisualVM a good profiler to start with
> considering my system?
>
> Sorry, don't know. Never used it. I use
> YourKit, mainly because they
> kindly donate free licenses to OpenSource
> developers to use with their
> OpenSource projects.
>
> Thank you Mark,
>
> I installed VisualVM as a profiler.
> I can now see all of the threads that tomcat is
> "using" (?not sure if
> that's the correct term) (there's 35 live threads
> and 33 daemon threads).
> Do you have any recommendations for how I might
> trouble shoot the
> original
> warning that I got? Would I do a "heap dump"?
> I have restarted tomcat since the time of the
> original warning message,
> so
> I can't expect to find the same thread, correct?
> Any tips?
>
> FYI: I never saw my cpu climbing or anything, I am
> troubleshooting this
> warning solely on seeing the "Warning" message. Is
> this overkill? Should
> I
> wait until something more severe occurs? (I have
> not yet gone live with
> my
> webapp but planned on doing so soon).
>
> Thanks.
>
>
> Mark
>
> I would just like to confirm that this warning
> is not something serious.
>
> I am currently having a little trouble
> running VisualVM (jstatd) from a
> remote machine but can struggle with it a
> little more to try to get it
> working.
>
>
> Thanks.
>
>
>
> I used the manager app and clicked "find
> leaks" but it said that there
>
> were
>
> none.
> (Can someone also explain why I
> have to be cautious when using this
>
> feature
>
> on production environments).
>
> It triggers a full GC which may
> (hopefully briefly) pause the entire
>
> JVM.
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> <ma...@tomcat.apache.org>
> For additional commands, e-mail:
> users-help@tomcat.apache.org
> <ma...@tomcat.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> <ma...@tomcat.apache.org>
> For additional commands, e-mail:
> users-help@tomcat.apache.org
> <ma...@tomcat.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> <ma...@tomcat.apache.org>
> For additional commands, e-mail:
> users-help@tomcat.apache.org
> <ma...@tomcat.apache.org>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebAppClassLoaderBase.clearReferencesThreads warning
Posted by Yuval Schwartz <yu...@gmail.com>.
Hello Terence,
Thanks for the input.
I shutdown the ScheduledExecutorService in the contextDestroyed method of
the app's ServletContextListener class as well.
I also call shutDownNow() followed by an if statement with
!awaitTermination() as the condition.
Are you sure that the new warning that I'm getting is also related to the
ScheduledExecutorService?
The warning again is:
WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor
The executor associated with thread pool [http-apr-8080] has not fully
shutdown. Some application threads may still be running
I didn't find much information about http-apr-8080 on google.
When I do a thread dump while tomcat is running I see http-apr-8080-exec-1,
http-apr-8080-exec-2,...http-apr-8080-exec-10.
I'm not sure what these do? Are these threads the same as http-apr-8080
that is referenced in the warning?
How else can I go about debugging this message?
Thank you
On Mon, Feb 1, 2016 at 5:59 PM, Terence M. Bandoian <te...@tmbsw.com>
wrote:
> On 2/1/2016 8:54 AM, Yuval Schwartz wrote:
>
>> Hello Mark,
>>
>> I think that the issue below was related to the way I was shutting down an
>> instance of ScheduledExecutorService.
>> I changed the way it is shutdown when the context is destroyed...I will
>> update here if I don't receive any more warnings.
>>
>> However, I now get a warning:
>>
>> WARNING [main] org.apache.tomcat.util.net
>> .AbstractEndpoint.shutdownExecutor
>> The executor associated with thread pool [http-apr-8080] has not fully
>> shutdown. Some application threads may still be running
>>
>> I am not sure if this is related or not?
>> When I do a heap dump with jstack I see http-apr-8080-exec-[1-10] but I
>> don't see any thread named http-apr-8080.
>> Any ideas on how to trouble shoot this?
>>
>> Thanks.
>>
>
>
> Hi, Yuval-
>
> Where are you shutting down the ScheduledExecutorService? I ran into a
> similar problem trying to stop a ScheduledExecutorService in the
> contextDestroyed method of a ServletContextListener. Adding a call to
> Thread.yield() after the executor was terminated removed the warning. To
> terminate the executor, I used the shutdown() method followed by a loop
> with a call to awaitTermination().
>
> -Terence Bandoian
>
>
>
>
>> On Sun, Jan 31, 2016 at 8:18 PM, Yuval Schwartz <yuval.schwartz@gmail.com
>> >
>> wrote:
>>
>>
>>> On Sunday, 31 January 2016, Mark Thomas <ma...@apache.org> wrote:
>>>
>>> On 31/01/2016 08:06, Yuval Schwartz wrote:
>>>>
>>>>> Hellow Mark,
>>>>>
>>>>> Thanks for the reply.
>>>>> Follow up questions below.
>>>>>
>>>>> On Fri, Jan 29, 2016 at 6:22 PM, Mark Thomas <ma...@apache.org> wrote:
>>>>>
>>>>> On 29/01/2016 14:34, Yuval Schwartz wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> tomcat version: 8.0.22
>>>>>>> java: jdk1.8.0_05
>>>>>>> server: Amazon Linux AMI
>>>>>>>
>>>>>>> I get the following warning message in my catalina log when I
>>>>>>>
>>>>>> undeploy a
>>>>
>>>>> web application:
>>>>>>>
>>>>>>> *WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>>>>>>>
>>>>>>>
>>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
>>>>
>>>>> The
>>>>>>
>>>>>>> web application [ROOT##002] appears to have started a thread named
>>>>>>> [pool-2-thread-1] but has failed to stop it. This is very likely to
>>>>>>>
>>>>>> create
>>>>>>
>>>>>>> a memory leak. Stack trace of thread*
>>>>>>>
>>>>>>> As you can see, for some reason, I don't get a stack trace of the
>>>>>>>
>>>>>> thread.
>>>>
>>>>> Therefore, I have no idea how to debug this warning.
>>>>>>>
>>>>>>> This particular warning was generated when tomcat detected an unused
>>>>>>> version and undeployed it (I set undeployOldVersions="true").
>>>>>>>
>>>>>>> Does anyone know how I can debug this warning. How can I know more
>>>>>>>
>>>>>> about
>>>>
>>>>> this thread?
>>>>>>>
>>>>>> That looks like a thread from Commons Pool (possibly via DBCP). The
>>>>>>
>>>>> only
>>>>
>>>>> way to be sure you have a leak or not is to use a profiler. See
>>>>>>
>>>>>>
>>>>>>
>>>> http://home.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>>>
>>>>>
>>>>> Is VisualVM a good profiler to start with considering my system?
>>>>>
>>>> Sorry, don't know. Never used it. I use YourKit, mainly because they
>>>> kindly donate free licenses to OpenSource developers to use with their
>>>> OpenSource projects.
>>>>
>>>
>>> Thank you Mark,
>>>
>>> I installed VisualVM as a profiler.
>>> I can now see all of the threads that tomcat is "using" (?not sure if
>>> that's the correct term) (there's 35 live threads and 33 daemon threads).
>>> Do you have any recommendations for how I might trouble shoot the
>>> original
>>> warning that I got? Would I do a "heap dump"?
>>> I have restarted tomcat since the time of the original warning message,
>>> so
>>> I can't expect to find the same thread, correct?
>>> Any tips?
>>>
>>> FYI: I never saw my cpu climbing or anything, I am troubleshooting this
>>> warning solely on seeing the "Warning" message. Is this overkill? Should
>>> I
>>> wait until something more severe occurs? (I have not yet gone live with
>>> my
>>> webapp but planned on doing so soon).
>>>
>>> Thanks.
>>>
>>>
>>> Mark
>>>>
>>>> I would just like to confirm that this warning is not something serious.
>>>>> I am currently having a little trouble running VisualVM (jstatd) from a
>>>>> remote machine but can struggle with it a little more to try to get it
>>>>> working.
>>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>> I used the manager app and clicked "find leaks" but it said that there
>>>>>>>
>>>>>> were
>>>>>>
>>>>>>> none.
>>>>>>> (Can someone also explain why I have to be cautious when using this
>>>>>>>
>>>>>> feature
>>>>>>
>>>>>>> on production environments).
>>>>>>>
>>>>>> It triggers a full GC which may (hopefully briefly) pause the entire
>>>>>>
>>>>> JVM.
>>>>
>>>>> Mark
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: WebAppClassLoaderBase.clearReferencesThreads warning
Posted by "Terence M. Bandoian" <te...@tmbsw.com>.
On 2/1/2016 8:54 AM, Yuval Schwartz wrote:
> Hello Mark,
>
> I think that the issue below was related to the way I was shutting down an
> instance of ScheduledExecutorService.
> I changed the way it is shutdown when the context is destroyed...I will
> update here if I don't receive any more warnings.
>
> However, I now get a warning:
>
> WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor
> The executor associated with thread pool [http-apr-8080] has not fully
> shutdown. Some application threads may still be running
>
> I am not sure if this is related or not?
> When I do a heap dump with jstack I see http-apr-8080-exec-[1-10] but I
> don't see any thread named http-apr-8080.
> Any ideas on how to trouble shoot this?
>
> Thanks.
Hi, Yuval-
Where are you shutting down the ScheduledExecutorService? I ran into a
similar problem trying to stop a ScheduledExecutorService in the
contextDestroyed method of a ServletContextListener. Adding a call to
Thread.yield() after the executor was terminated removed the warning.
To terminate the executor, I used the shutdown() method followed by a
loop with a call to awaitTermination().
-Terence Bandoian
>
> On Sun, Jan 31, 2016 at 8:18 PM, Yuval Schwartz <yu...@gmail.com>
> wrote:
>
>>
>> On Sunday, 31 January 2016, Mark Thomas <ma...@apache.org> wrote:
>>
>>> On 31/01/2016 08:06, Yuval Schwartz wrote:
>>>> Hellow Mark,
>>>>
>>>> Thanks for the reply.
>>>> Follow up questions below.
>>>>
>>>> On Fri, Jan 29, 2016 at 6:22 PM, Mark Thomas <ma...@apache.org> wrote:
>>>>
>>>>> On 29/01/2016 14:34, Yuval Schwartz wrote:
>>>>>> Hello,
>>>>>>
>>>>>> tomcat version: 8.0.22
>>>>>> java: jdk1.8.0_05
>>>>>> server: Amazon Linux AMI
>>>>>>
>>>>>> I get the following warning message in my catalina log when I
>>> undeploy a
>>>>>> web application:
>>>>>>
>>>>>> *WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>>>>>>
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
>>>>> The
>>>>>> web application [ROOT##002] appears to have started a thread named
>>>>>> [pool-2-thread-1] but has failed to stop it. This is very likely to
>>>>> create
>>>>>> a memory leak. Stack trace of thread*
>>>>>>
>>>>>> As you can see, for some reason, I don't get a stack trace of the
>>> thread.
>>>>>> Therefore, I have no idea how to debug this warning.
>>>>>>
>>>>>> This particular warning was generated when tomcat detected an unused
>>>>>> version and undeployed it (I set undeployOldVersions="true").
>>>>>>
>>>>>> Does anyone know how I can debug this warning. How can I know more
>>> about
>>>>>> this thread?
>>>>> That looks like a thread from Commons Pool (possibly via DBCP). The
>>> only
>>>>> way to be sure you have a leak or not is to use a profiler. See
>>>>>
>>>>>
>>> http://home.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>>>
>>>> Is VisualVM a good profiler to start with considering my system?
>>> Sorry, don't know. Never used it. I use YourKit, mainly because they
>>> kindly donate free licenses to OpenSource developers to use with their
>>> OpenSource projects.
>>
>> Thank you Mark,
>>
>> I installed VisualVM as a profiler.
>> I can now see all of the threads that tomcat is "using" (?not sure if
>> that's the correct term) (there's 35 live threads and 33 daemon threads).
>> Do you have any recommendations for how I might trouble shoot the original
>> warning that I got? Would I do a "heap dump"?
>> I have restarted tomcat since the time of the original warning message, so
>> I can't expect to find the same thread, correct?
>> Any tips?
>>
>> FYI: I never saw my cpu climbing or anything, I am troubleshooting this
>> warning solely on seeing the "Warning" message. Is this overkill? Should I
>> wait until something more severe occurs? (I have not yet gone live with my
>> webapp but planned on doing so soon).
>>
>> Thanks.
>>
>>
>>> Mark
>>>
>>>> I would just like to confirm that this warning is not something serious.
>>>> I am currently having a little trouble running VisualVM (jstatd) from a
>>>> remote machine but can struggle with it a little more to try to get it
>>>> working.
>>>>
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>>>> I used the manager app and clicked "find leaks" but it said that there
>>>>> were
>>>>>> none.
>>>>>> (Can someone also explain why I have to be cautious when using this
>>>>> feature
>>>>>> on production environments).
>>>>> It triggers a full GC which may (hopefully briefly) pause the entire
>>> JVM.
>>>>> Mark
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org