You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Chris Gamache <cg...@gmail.com> on 2017/05/27 12:27:49 UTC
Proper use of Executors
Hi all,
I'm using org.apache.commons:commons-exec:1.3 on Java 8.
I'm having an issue where my Tomcat server is bleeding out hundreds of
threads and all of the memory in the form of Executors that I'm running but
don't seem to be closing down ... When the server finally grinds to a halt
I have to restart. When I do it looks like this at shutdown time:
<snip>
27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [ROOT##000252] appears to have started a thread named [Exec
Default Executor] but has failed to stop it. This is very likely to create
a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
java.lang.Thread.run(Thread.java:745)
27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [ROOT##000252] appears to have started a thread named [Exec
Stream Pumper] but has failed to stop it. This is very likely to create a
memory leak. Stack trace of thread:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:255)
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
java.io.BufferedInputStream.read(BufferedInputStream.java:345)
java.io.FilterInputStream.read(FilterInputStream.java:107)
org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
java.lang.Thread.run(Thread.java:745)
</snip>
And my thread dump is a mile long.
I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
relevant code. It is called from a regular method in a regular class,
nothing fancy:
CommandLine cmdLine = CommandLine.parse(command.toString());
DefaultExecutor executor = new DefaultExecutor();
PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
executor.setStreamHandler(esh);
executor.execute(cmdLine);
`is` and `os` are passed in on the constructor. Their opens and closes are
managed well and cleaned up on the outside of this class...
Are there further steps I'm missing to ensure the threads I'm creating are
getting shut down properly and the resources they are using are being
returned?
Any help is much appreciated.
Re: Proper use of Executors
Posted by Martin Gainty <mg...@hotmail.com>.
MG>below
________________________________
From: Siegfried Goeschl <si...@it20one.com>
Sent: Sunday, May 28, 2017 3:41 PM
To: Commons Users List
Cc: cgamache@gmail.com
Subject: Re: Proper use of Executors
Hi Chris,
there are couple of things to consider
* You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
* That could happen if the process being executed actually writes some error messages :-)
* Are you 100% sure that the processes will terminate? See ExecuteWatchdog
* You might habe a look at ProcessDestroyer to cleanup during shutdown
MG>correct ThreadMonitor needs a watchdog for timeout e.g.
ExecuteWatchdog watchDog = new ExecuteWatchdog(60L * 60L * 1000L);
MG>once timeout is achieved you will need ProcessDestroyer to kill the process
ShutdownHookProcessDestroyer processDestroyer = new ShutdownHookProcessDestroyer();
MG>Executor needs to identify its watchdog and ProcessDestroyer
executor.setStreamHandler(streamHandler); //PumpStreamHandler
executor.setWorkingDirectory(new File(getExecutionDirectory())); //which directory to write to
executor.setProcessDestroyer(processDestroyer); //what is processDestroyer
executor.setWatchdog(watchDog); //identify watchdog
// Async Executor doesn't block.
DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();
//Executor HashMap
// Inherit the current process's environment variables and add the user-defined ones
@SuppressWarnings("unchecked")
Map<String, String> newEnvironment = EnvironmentUtils.getProcEnvironment();
newEnvironment.putAll(this.environment);
//place to stuff the results
DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();
//use executor to execute commandLine, environment HashHap
executor.execute(command, newEnvironment, resultHandler);
//return results
return resultHandler;
http://www.programcreek.com/java-api-examples/index.php?api=org.apache.commons.exec.DefaultExecuteResultHandler
Java Code Example org.apache.commons.exec ...<http://www.programcreek.com/java-api-examples/index.php?api=org.apache.commons.exec.DefaultExecuteResultHandler>
www.programcreek.com
This page provides Java code examples for org.apache.commons.exec.DefaultExecuteResultHandler. The examples are extracted from open source Java projects from GitHub.
Thanks in advance,
Siegfried Goeschl
> On 27 May 2017, at 14:27, Chris Gamache <cg...@gmail.com> wrote:
>
> Hi all,
>
> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>
> I'm having an issue where my Tomcat server is bleeding out hundreds of
> threads and all of the memory in the form of Executors that I'm running but
> don't seem to be closing down ... When the server finally grinds to a halt
> I have to restart. When I do it looks like this at shutdown time:
>
> <snip>
>
> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Default Executor] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
> java.lang.Thread.run(Thread.java:745)
>
> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Stream Pumper] but has failed to stop it. This is very likely to create a
> memory leak. Stack trace of thread:
> java.io.FileInputStream.readBytes(Native Method)
> java.io.FileInputStream.read(FileInputStream.java:255)
> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> java.io.FilterInputStream.read(FilterInputStream.java:107)
> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> java.lang.Thread.run(Thread.java:745)
>
> </snip>
>
> And my thread dump is a mile long.
>
> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
> relevant code. It is called from a regular method in a regular class,
> nothing fancy:
>
> CommandLine cmdLine = CommandLine.parse(command.toString());
> DefaultExecutor executor = new DefaultExecutor();
> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> executor.setStreamHandler(esh);
> executor.execute(cmdLine);
>
> `is` and `os` are passed in on the constructor. Their opens and closes are
> managed well and cleaned up on the outside of this class...
> Are there further steps I'm missing to ensure the threads I'm creating are
> getting shut down properly and the resources they are using are being
> returned?
>
> Any help is much appreciated.
Re: [Re] Proper use of Executors
Posted by Siegfried Goeschl <si...@it20one.com>.
Hi Chris,
the “recurring theme” does ring a bell - I guess you already know https://www.mail-archive.com/user@commons.apache.org/msg10368.html <https://www.mail-archive.com/user@commons.apache.org/msg10368.html> :-)
Do you have some Git repo to re-produce the problem?
Thanks in advance,
Siegfried Goeschl
> On 21 Jun 2017, at 17:48, Chris Gamache <cg...@gmail.com> wrote:
>
> Ok so, it's hard to tell if this made much of a difference. There might be a few less messages? Hard to be sure.
>
> I'm still getting those messages, but I'm now suspecting Selenium is generating the majority of these messages. It is behaving badly when it comes to killing its spawned phantomjs processes, thereby leaving the executors open. Seems like a recurring theme, eh Siegfried? ;)
>
> Selenium is co-opting the Apache Commons Exec. It looks like it is using ExecuteWatchdog.INFINITE_TIMEOUT, so no help there. I see they are making a pretty good effort to kill the processes. I won't rule-out that I'm doing it wrong, but I'm pretty sure
>
> } finally {
> driver.quit();
> }
>
> is all I'm responsible for if I want Selenium to close phantomjs processes. I do have some room to upgrade Selenium and PhantomJS which might help. I didn't see any meaningful changes in Selenium that would lead me to believe that an upgrade would fix the issue. But, it's a good practice and any patches will have to be made at the head of the project anyway.
>
> Thanks for all the help!
>
> CG
>
>
> On Mon, Jun 19, 2017 at 4:11 PM, Siegfried Goeschl <siegfried.goeschl@it20one.com <ma...@it20one.com>> wrote:
> Staying tuned :-)
>
> > On 5 Jun 2017, at 13:59, Chris Gamache <cgamache@gmail.com <ma...@gmail.com>> wrote:
> >
> > Hi Siegfried,
> >
> > I have implemented the executor watchdogs and shutdown hooks. I will know if our efforts have been fruitful when we make our first restart which I'm sure will be soon. I promise I'll write back and let you know how it goes.
> >
> > Thank you vary much for checking in with me!
> >
> >> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <siegfried.goeschl@it20one.com <ma...@it20one.com>> wrote:
> >>
> >> Hi Chris,
> >>
> >> any new findings from your side?
> >>
> >> Thanks in advance,
> >>
> >> Siegfried Goeschl
> >>
> >>> On 28 May 2017, at 21:41, Siegfried Goeschl <siegfried.goeschl@it20one.com <ma...@it20one.com>> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> there are couple of things to consider
> >>>
> >>> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
> >>> * That could happen if the process being executed actually writes some error messages :-)
> >>> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
> >>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
> >>>
> >>> Thanks in advance,
> >>>
> >>> Siegfried Goeschl
> >>>
> >>>
> >>>> On 27 May 2017, at 14:27, Chris Gamache <cgamache@gmail.com <ma...@gmail.com>> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
> >>>>
> >>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
> >>>> threads and all of the memory in the form of Executors that I'm running but
> >>>> don't seem to be closing down ... When the server finally grinds to a halt
> >>>> I have to restart. When I do it looks like this at shutdown time:
> >>>>
> >>>> <snip>
> >>>>
> >>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> >>>> web application [ROOT##000252] appears to have started a thread named [Exec
> >>>> Default Executor] but has failed to stop it. This is very likely to create
> >>>> a memory leak. Stack trace of thread:
> >>>> java.lang.Object.wait(Native Method)
> >>>> java.lang.Object.wait(Object.java:502)
> >>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> >>>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
> >>>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
> >>>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> >>>> web application [ROOT##000252] appears to have started a thread named [Exec
> >>>> Stream Pumper] but has failed to stop it. This is very likely to create a
> >>>> memory leak. Stack trace of thread:
> >>>> java.io.FileInputStream.readBytes(Native Method)
> >>>> java.io.FileInputStream.read(FileInputStream.java:255)
> >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> >>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
> >>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> </snip>
> >>>>
> >>>> And my thread dump is a mile long.
> >>>>
> >>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
> >>>> relevant code. It is called from a regular method in a regular class,
> >>>> nothing fancy:
> >>>>
> >>>> CommandLine cmdLine = CommandLine.parse(command.toString());
> >>>> DefaultExecutor executor = new DefaultExecutor();
> >>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> >>>> executor.setStreamHandler(esh);
> >>>> executor.execute(cmdLine);
> >>>>
> >>>> `is` and `os` are passed in on the constructor. Their opens and closes are
> >>>> managed well and cleaned up on the outside of this class...
> >>>> Are there further steps I'm missing to ensure the threads I'm creating are
> >>>> getting shut down properly and the resources they are using are being
> >>>> returned?
> >>>>
> >>>> Any help is much appreciated.
> >>>
> >>
>
>
Re: [Re] Proper use of Executors
Posted by Chris Gamache <cg...@gmail.com>.
Ok so, it's hard to tell if this made much of a difference. There might be
a few less messages? Hard to be sure.
I'm still getting those messages, but I'm now suspecting Selenium is
generating the majority of these messages. It is behaving badly when it
comes to killing its spawned phantomjs processes, thereby leaving the
executors open. Seems like a recurring theme, eh Siegfried? ;)
Selenium is co-opting the Apache Commons Exec. It looks like it is using
ExecuteWatchdog.INFINITE_TIMEOUT, so no help there. I see they are making a
pretty good effort to kill the processes. I won't rule-out that I'm doing
it wrong, but I'm pretty sure
} finally {
driver.quit();
}
is all I'm responsible for if I want Selenium to close phantomjs processes.
I do have some room to upgrade Selenium and PhantomJS which might help. I
didn't see any meaningful changes in Selenium that would lead me to believe
that an upgrade would fix the issue. But, it's a good practice and any
patches will have to be made at the head of the project anyway.
Thanks for all the help!
CG
On Mon, Jun 19, 2017 at 4:11 PM, Siegfried Goeschl <
siegfried.goeschl@it20one.com> wrote:
> Staying tuned :-)
>
> > On 5 Jun 2017, at 13:59, Chris Gamache <cg...@gmail.com> wrote:
> >
> > Hi Siegfried,
> >
> > I have implemented the executor watchdogs and shutdown hooks. I will
> know if our efforts have been fruitful when we make our first restart which
> I'm sure will be soon. I promise I'll write back and let you know how it
> goes.
> >
> > Thank you vary much for checking in with me!
> >
> >> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <
> siegfried.goeschl@it20one.com> wrote:
> >>
> >> Hi Chris,
> >>
> >> any new findings from your side?
> >>
> >> Thanks in advance,
> >>
> >> Siegfried Goeschl
> >>
> >>> On 28 May 2017, at 21:41, Siegfried Goeschl <
> siegfried.goeschl@it20one.com> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> there are couple of things to consider
> >>>
> >>> * You are using a PumpStreamHander but the STDERR is not consumed.
> Each process has an internal buffer (size depends on the OS) and when the
> buffer is full any write to STDERR is blocked
> >>> * That could happen if the process being executed actually writes some
> error messages :-)
> >>> * Are you 100% sure that the processes will terminate? See
> ExecuteWatchdog
> >>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
> >>>
> >>> Thanks in advance,
> >>>
> >>> Siegfried Goeschl
> >>>
> >>>
> >>>> On 27 May 2017, at 14:27, Chris Gamache <cg...@gmail.com> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
> >>>>
> >>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
> >>>> threads and all of the memory in the form of Executors that I'm
> running but
> >>>> don't seem to be closing down ... When the server finally grinds to a
> halt
> >>>> I have to restart. When I do it looks like this at shutdown time:
> >>>>
> >>>> <snip>
> >>>>
> >>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> [Exec
> >>>> Default Executor] but has failed to stop it. This is very likely to
> create
> >>>> a memory leak. Stack trace of thread:
> >>>> java.lang.Object.wait(Native Method)
> >>>> java.lang.Object.wait(Object.java:502)
> >>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> >>>> org.apache.commons.exec.DefaultExecutor.executeInternal(
> DefaultExecutor.java:364)
> >>>> org.apache.commons.exec.DefaultExecutor.access$200(
> DefaultExecutor.java:48)
> >>>> org.apache.commons.exec.DefaultExecutor$1.run(
> DefaultExecutor.java:200)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> [Exec
> >>>> Stream Pumper] but has failed to stop it. This is very likely to
> create a
> >>>> memory leak. Stack trace of thread:
> >>>> java.io.FileInputStream.readBytes(Native Method)
> >>>> java.io.FileInputStream.read(FileInputStream.java:255)
> >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> >>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
> >>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> </snip>
> >>>>
> >>>> And my thread dump is a mile long.
> >>>>
> >>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's
> the
> >>>> relevant code. It is called from a regular method in a regular class,
> >>>> nothing fancy:
> >>>>
> >>>> CommandLine cmdLine = CommandLine.parse(command.toString());
> >>>> DefaultExecutor executor = new DefaultExecutor();
> >>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> >>>> executor.setStreamHandler(esh);
> >>>> executor.execute(cmdLine);
> >>>>
> >>>> `is` and `os` are passed in on the constructor. Their opens and
> closes are
> >>>> managed well and cleaned up on the outside of this class...
> >>>> Are there further steps I'm missing to ensure the threads I'm
> creating are
> >>>> getting shut down properly and the resources they are using are being
> >>>> returned?
> >>>>
> >>>> Any help is much appreciated.
> >>>
> >>
>
>
Re: [Re] Proper use of Executors
Posted by Siegfried Goeschl <si...@it20one.com>.
Staying tuned :-)
> On 5 Jun 2017, at 13:59, Chris Gamache <cg...@gmail.com> wrote:
>
> Hi Siegfried,
>
> I have implemented the executor watchdogs and shutdown hooks. I will know if our efforts have been fruitful when we make our first restart which I'm sure will be soon. I promise I'll write back and let you know how it goes.
>
> Thank you vary much for checking in with me!
>
>> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <si...@it20one.com> wrote:
>>
>> Hi Chris,
>>
>> any new findings from your side?
>>
>> Thanks in advance,
>>
>> Siegfried Goeschl
>>
>>> On 28 May 2017, at 21:41, Siegfried Goeschl <si...@it20one.com> wrote:
>>>
>>> Hi Chris,
>>>
>>> there are couple of things to consider
>>>
>>> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
>>> * That could happen if the process being executed actually writes some error messages :-)
>>> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
>>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
>>>
>>> Thanks in advance,
>>>
>>> Siegfried Goeschl
>>>
>>>
>>>> On 27 May 2017, at 14:27, Chris Gamache <cg...@gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>>>>
>>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
>>>> threads and all of the memory in the form of Executors that I'm running but
>>>> don't seem to be closing down ... When the server finally grinds to a halt
>>>> I have to restart. When I do it looks like this at shutdown time:
>>>>
>>>> <snip>
>>>>
>>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
>>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>>> Default Executor] but has failed to stop it. This is very likely to create
>>>> a memory leak. Stack trace of thread:
>>>> java.lang.Object.wait(Native Method)
>>>> java.lang.Object.wait(Object.java:502)
>>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
>>>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
>>>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
>>>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
>>>> java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
>>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>>> Stream Pumper] but has failed to stop it. This is very likely to create a
>>>> memory leak. Stack trace of thread:
>>>> java.io.FileInputStream.readBytes(Native Method)
>>>> java.io.FileInputStream.read(FileInputStream.java:255)
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
>>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
>>>> java.lang.Thread.run(Thread.java:745)
>>>>
>>>> </snip>
>>>>
>>>> And my thread dump is a mile long.
>>>>
>>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
>>>> relevant code. It is called from a regular method in a regular class,
>>>> nothing fancy:
>>>>
>>>> CommandLine cmdLine = CommandLine.parse(command.toString());
>>>> DefaultExecutor executor = new DefaultExecutor();
>>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
>>>> executor.setStreamHandler(esh);
>>>> executor.execute(cmdLine);
>>>>
>>>> `is` and `os` are passed in on the constructor. Their opens and closes are
>>>> managed well and cleaned up on the outside of this class...
>>>> Are there further steps I'm missing to ensure the threads I'm creating are
>>>> getting shut down properly and the resources they are using are being
>>>> returned?
>>>>
>>>> Any help is much appreciated.
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [Re] Proper use of Executors
Posted by Chris Gamache <cg...@gmail.com>.
Hi Siegfried,
I have implemented the executor watchdogs and shutdown hooks. I will know if our efforts have been fruitful when we make our first restart which I'm sure will be soon. I promise I'll write back and let you know how it goes.
Thank you vary much for checking in with me!
> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <si...@it20one.com> wrote:
>
> Hi Chris,
>
> any new findings from your side?
>
> Thanks in advance,
>
> Siegfried Goeschl
>
>> On 28 May 2017, at 21:41, Siegfried Goeschl <si...@it20one.com> wrote:
>>
>> Hi Chris,
>>
>> there are couple of things to consider
>>
>> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
>> * That could happen if the process being executed actually writes some error messages :-)
>> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
>>
>> Thanks in advance,
>>
>> Siegfried Goeschl
>>
>>
>>> On 27 May 2017, at 14:27, Chris Gamache <cg...@gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>>>
>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
>>> threads and all of the memory in the form of Executors that I'm running but
>>> don't seem to be closing down ... When the server finally grinds to a halt
>>> I have to restart. When I do it looks like this at shutdown time:
>>>
>>> <snip>
>>>
>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>> Default Executor] but has failed to stop it. This is very likely to create
>>> a memory leak. Stack trace of thread:
>>> java.lang.Object.wait(Native Method)
>>> java.lang.Object.wait(Object.java:502)
>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
>>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
>>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
>>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
>>> java.lang.Thread.run(Thread.java:745)
>>>
>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>> Stream Pumper] but has failed to stop it. This is very likely to create a
>>> memory leak. Stack trace of thread:
>>> java.io.FileInputStream.readBytes(Native Method)
>>> java.io.FileInputStream.read(FileInputStream.java:255)
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
>>> java.lang.Thread.run(Thread.java:745)
>>>
>>> </snip>
>>>
>>> And my thread dump is a mile long.
>>>
>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
>>> relevant code. It is called from a regular method in a regular class,
>>> nothing fancy:
>>>
>>> CommandLine cmdLine = CommandLine.parse(command.toString());
>>> DefaultExecutor executor = new DefaultExecutor();
>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
>>> executor.setStreamHandler(esh);
>>> executor.execute(cmdLine);
>>>
>>> `is` and `os` are passed in on the constructor. Their opens and closes are
>>> managed well and cleaned up on the outside of this class...
>>> Are there further steps I'm missing to ensure the threads I'm creating are
>>> getting shut down properly and the resources they are using are being
>>> returned?
>>>
>>> Any help is much appreciated.
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
[Re] Proper use of Executors
Posted by Siegfried Goeschl <si...@it20one.com>.
Hi Chris,
any new findings from your side?
Thanks in advance,
Siegfried Goeschl
> On 28 May 2017, at 21:41, Siegfried Goeschl <si...@it20one.com> wrote:
>
> Hi Chris,
>
> there are couple of things to consider
>
> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
> * That could happen if the process being executed actually writes some error messages :-)
> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
> * You might habe a look at ProcessDestroyer to cleanup during shutdown
>
> Thanks in advance,
>
> Siegfried Goeschl
>
>
>> On 27 May 2017, at 14:27, Chris Gamache <cg...@gmail.com> wrote:
>>
>> Hi all,
>>
>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>>
>> I'm having an issue where my Tomcat server is bleeding out hundreds of
>> threads and all of the memory in the form of Executors that I'm running but
>> don't seem to be closing down ... When the server finally grinds to a halt
>> I have to restart. When I do it looks like this at shutdown time:
>>
>> <snip>
>>
>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>> web application [ROOT##000252] appears to have started a thread named [Exec
>> Default Executor] but has failed to stop it. This is very likely to create
>> a memory leak. Stack trace of thread:
>> java.lang.Object.wait(Native Method)
>> java.lang.Object.wait(Object.java:502)
>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
>> java.lang.Thread.run(Thread.java:745)
>>
>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>> web application [ROOT##000252] appears to have started a thread named [Exec
>> Stream Pumper] but has failed to stop it. This is very likely to create a
>> memory leak. Stack trace of thread:
>> java.io.FileInputStream.readBytes(Native Method)
>> java.io.FileInputStream.read(FileInputStream.java:255)
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>> java.io.FilterInputStream.read(FilterInputStream.java:107)
>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
>> java.lang.Thread.run(Thread.java:745)
>>
>> </snip>
>>
>> And my thread dump is a mile long.
>>
>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
>> relevant code. It is called from a regular method in a regular class,
>> nothing fancy:
>>
>> CommandLine cmdLine = CommandLine.parse(command.toString());
>> DefaultExecutor executor = new DefaultExecutor();
>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
>> executor.setStreamHandler(esh);
>> executor.execute(cmdLine);
>>
>> `is` and `os` are passed in on the constructor. Their opens and closes are
>> managed well and cleaned up on the outside of this class...
>> Are there further steps I'm missing to ensure the threads I'm creating are
>> getting shut down properly and the resources they are using are being
>> returned?
>>
>> Any help is much appreciated.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: Proper use of Executors
Posted by Siegfried Goeschl <si...@it20one.com>.
Hi Chris,
there are couple of things to consider
* You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
* That could happen if the process being executed actually writes some error messages :-)
* Are you 100% sure that the processes will terminate? See ExecuteWatchdog
* You might habe a look at ProcessDestroyer to cleanup during shutdown
Thanks in advance,
Siegfried Goeschl
> On 27 May 2017, at 14:27, Chris Gamache <cg...@gmail.com> wrote:
>
> Hi all,
>
> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>
> I'm having an issue where my Tomcat server is bleeding out hundreds of
> threads and all of the memory in the form of Executors that I'm running but
> don't seem to be closing down ... When the server finally grinds to a halt
> I have to restart. When I do it looks like this at shutdown time:
>
> <snip>
>
> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Default Executor] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
> java.lang.Thread.run(Thread.java:745)
>
> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Stream Pumper] but has failed to stop it. This is very likely to create a
> memory leak. Stack trace of thread:
> java.io.FileInputStream.readBytes(Native Method)
> java.io.FileInputStream.read(FileInputStream.java:255)
> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> java.io.FilterInputStream.read(FilterInputStream.java:107)
> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> java.lang.Thread.run(Thread.java:745)
>
> </snip>
>
> And my thread dump is a mile long.
>
> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
> relevant code. It is called from a regular method in a regular class,
> nothing fancy:
>
> CommandLine cmdLine = CommandLine.parse(command.toString());
> DefaultExecutor executor = new DefaultExecutor();
> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> executor.setStreamHandler(esh);
> executor.execute(cmdLine);
>
> `is` and `os` are passed in on the constructor. Their opens and closes are
> managed well and cleaned up on the outside of this class...
> Are there further steps I'm missing to ensure the threads I'm creating are
> getting shut down properly and the resources they are using are being
> returned?
>
> Any help is much appreciated.