You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by mdo <ma...@gmail.com> on 2013/05/02 18:20:22 UTC
2.11, File2, Quartz: threads leaking
Hello,
I've constructed a thread leak somehow. I found some discussions/issues
describing past thread leaking problems, e.g.:
http://stackoverflow.com/q/10071814
http://stackoverflow.com/q/14992793
https://issues.apache.org/jira/browse/CAMEL-5072
http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-td5548338.html
So obviously there has been some work to fix those issues. In one case Claus
advised the user that he must stop a producer template. I'm not sure how to
solve my problem. Find my setup description of a Quartz-triggered file
consumer route below.
A file consumer:
from("file:///tmp/x/?sendEmptyMessageWhenIdle=true")
.noAutoStartup()
.routeId(routeId)
.choice()
.when(body().isNull())
...
The null-Body is used as route-stop-trigger, I use a processor that starts
up a thread as described here:
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
... with these actions:
exchange.getContext().getInflightRepository().remove(exchange);
exchange.getContext().stopRoute(exchange.getFromRouteId());
I also tried a stop() on the getFromEndpoint() here.
A Quartz route, periodically firing the file consumer route based on cron
definitions:
from("quartz://timer-"+routeId+"?stateful=true&cron=...")
.routeId("quartz-"+routeId)
.process(new RouteStartProcessor(routeId));
process() in RouteStartProcessor:
CamelContext context = exchange.getContext();
context.startRoute(routeId);
I now added a loop to the RouteStartProcessor waiting for the file consumer
route to shutdown itself (to prevent overlaps), evaluated by:
routeStopped = context.getRouteStatus(routeId).isStopped();
This actually does work fine, so I assume that the route gets stopped
successfully. Logs: o.a.c.impl.DefaultShutdownStrategy - Graceful shutdown
of 1 routes completed in 0 seconds
Whenever Quartz fires and the consumer route gets re-started this brings up
a new thread id which I can see in the logs.
Albeit the route obviously is shut down, the threads stay alive, all in
state: java.lang.Thread.State: WAITING (parking)
I dug around a bit in the VM's MBeans, there are:
* 2 consumers, a FileConsumer and a QuartzConsumer
* a large amount of threadpools (equal to the number of waiting threads),
having IDs like FileConsumer(0x138dc688)
Attributes of an example instance:
Thread pool profile: --
Shutdown: false
Active count: 0
Pool size: 1
Task queue size: 0
Route:
Keep alive time: 0
Source:
Completed task count: 2
Core pool size: 1
Largest pool size: 1
Task count: 2
Id: FileConsumer(0x138dc688)
Maximum pool size: 2147483647
Task count: 2
My environment:
Linux
JDK 7
GlassFish 3
CDI Camel context
Camel 2.11.0
Any hints? Thanks in advance!
Regards, mdo.
--
View this message in context: http://camel.465427.n5.nabble.com/2-11-File2-Quartz-threads-leaking-tp5731922.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: 2.11, File2, Quartz: threads leaking
Posted by mdo <ma...@gmail.com>.
One addition regarding the thread count: besides the MBeans, VisualVM and
jstack do list them as live threads, so I'm able to pile up hundreds of live
threads by triggering my route that often.
--
View this message in context: http://camel.465427.n5.nabble.com/2-11-File2-Quartz-threads-leaking-tp5731922p5731957.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: 2.11, File2, Quartz: threads leaking
Posted by Claus Ibsen <cl...@gmail.com>.
If you want to cleanup and remove everything from a route you need to
stop it graceful, and then remove the route as well.
And if you have some endpoints that are not longer in use, then remove
them as well.
On Fri, May 3, 2013 at 12:30 PM, mdo <ma...@gmail.com> wrote:
> Thanks Claus, I can confirm that suspend-resume-cycles resolve my problem of
> leaked threads.
>
> I'm now doing a selective start:
> if (!context.getRouteStatus(routeId).isStarted()) {
> context.startRoute(routeId);
> } else {
> context.resumeRoute(routeId);
> }
>
> ... and changed stopRoute() to suspendRoute():
> //camelContext.stopRoute(routeId);
> camelContext.suspendRoute(routeId);
>
> The docs regarding the lifecycle state:
> "End users is encouraged to use suspend/resume if you are temporary stopping
> a Camel application."
>
> But I'm quite unsure: am I not supposed to re-start a stopped route, do you
> consider the piling of threads a bug in Camel or a mis-usage?
>
> I don't have an URL at hand but I remember that I found advice to use a
> Quartz trigger to periodically start and stop a route on this list. Maybe
> you could point out the pitfalls in the wiki somewhat more emphasized? Seems
> a pretty usual use case to me.
>
> My intention in stopping the route was to completely clean up the
> environment (without removing the route definition itself). By now I see
> that a suspended FTP2/SFTP endpoint keeps the remote connection open. Will
> go and shut it down somehow else now ...
>
> Regards, mdo.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/2-11-File2-Quartz-threads-leaking-tp5731922p5731963.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: 2.11, File2, Quartz: threads leaking
Posted by mdo <ma...@gmail.com>.
Thanks Claus, I can confirm that suspend-resume-cycles resolve my problem of
leaked threads.
I'm now doing a selective start:
if (!context.getRouteStatus(routeId).isStarted()) {
context.startRoute(routeId);
} else {
context.resumeRoute(routeId);
}
... and changed stopRoute() to suspendRoute():
//camelContext.stopRoute(routeId);
camelContext.suspendRoute(routeId);
The docs regarding the lifecycle state:
"End users is encouraged to use suspend/resume if you are temporary stopping
a Camel application."
But I'm quite unsure: am I not supposed to re-start a stopped route, do you
consider the piling of threads a bug in Camel or a mis-usage?
I don't have an URL at hand but I remember that I found advice to use a
Quartz trigger to periodically start and stop a route on this list. Maybe
you could point out the pitfalls in the wiki somewhat more emphasized? Seems
a pretty usual use case to me.
My intention in stopping the route was to completely clean up the
environment (without removing the route definition itself). By now I see
that a suspended FTP2/SFTP endpoint keeps the remote connection open. Will
go and shut it down somehow else now ...
Regards, mdo.
--
View this message in context: http://camel.465427.n5.nabble.com/2-11-File2-Quartz-threads-leaking-tp5731922p5731963.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: 2.11, File2, Quartz: threads leaking
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You can use suspend/resume on the route, if you just want it temporary
paused for a while. Then the thread pools and whatnot is kept alive.
See lifecycle details at
http://camel.apache.org/lifecycle
On Fri, May 3, 2013 at 10:07 AM, mdo <ma...@gmail.com> wrote:
> Hello Claus,
>
>
> Claus Ibsen-2 wrote
>> Check this page again as its updated how to stop a route from a route
>> http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
>
> in fact this is what I'm doing. Also, when I look at the page's history
> (<https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=23339650>)
> it seems that the last change dates back to 2012 (if the history is
> correct).
>
>
> Claus Ibsen-2 wrote
>> Also there was a bug so the MBean for the scheduled thread pool was
>> not removed when you removed the route.
>
> I don't remove the route — do I have to? I stop the route and on the next
> Quartz run a processor does start the route again which results in a new
> thread.
>
>
> Claus Ibsen-2 wrote
>> This has been fixed in 2.11.0, and upcoming releases of 2.10.x.
>> The thread pool is stopped though. Its just the mbean not being removed.
>
> I'm using 2.11.0.
>
> This is my current process method for stopping the route:
>
> @Override
> public void process(final Exchange exchange) throws Exception {
> if (stop == null || !stop.isAlive()) {
> stop = new Thread() {
> @Override
> public void run() {
> try {
>
> exchange.getContext().stopRoute(exchange.getFromRouteId());
> } catch (Exception e) {
> logger.error(e.getMessage());
> }
> }
> };
> }
> stop.start();
> }
>
> The additional check for isAlive is there because the processor instance is
> reused with each route-start-stop-cycle.
>
> Regards, mdo.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/2-11-File2-Quartz-threads-leaking-tp5731922p5731956.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: 2.11, File2, Quartz: threads leaking
Posted by mdo <ma...@gmail.com>.
Hello Claus,
Claus Ibsen-2 wrote
> Check this page again as its updated how to stop a route from a route
> http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
in fact this is what I'm doing. Also, when I look at the page's history
(<https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=23339650>)
it seems that the last change dates back to 2012 (if the history is
correct).
Claus Ibsen-2 wrote
> Also there was a bug so the MBean for the scheduled thread pool was
> not removed when you removed the route.
I don't remove the route — do I have to? I stop the route and on the next
Quartz run a processor does start the route again which results in a new
thread.
Claus Ibsen-2 wrote
> This has been fixed in 2.11.0, and upcoming releases of 2.10.x.
> The thread pool is stopped though. Its just the mbean not being removed.
I'm using 2.11.0.
This is my current process method for stopping the route:
@Override
public void process(final Exchange exchange) throws Exception {
if (stop == null || !stop.isAlive()) {
stop = new Thread() {
@Override
public void run() {
try {
exchange.getContext().stopRoute(exchange.getFromRouteId());
} catch (Exception e) {
logger.error(e.getMessage());
}
}
};
}
stop.start();
}
The additional check for isAlive is there because the processor instance is
reused with each route-start-stop-cycle.
Regards, mdo.
--
View this message in context: http://camel.465427.n5.nabble.com/2-11-File2-Quartz-threads-leaking-tp5731922p5731956.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: 2.11, File2, Quartz: threads leaking
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Check this page again as its updated how to stop a route from a route
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
Also there was a bug so the MBean for the scheduled thread pool was
not removed when you removed the route.
This has been fixed in 2.11.0, and upcoming releases of 2.10.x.
The thread pool is stopped though. Its just the mbean not being removed.
On Thu, May 2, 2013 at 6:20 PM, mdo <ma...@gmail.com> wrote:
> Hello,
>
> I've constructed a thread leak somehow. I found some discussions/issues
> describing past thread leaking problems, e.g.:
> http://stackoverflow.com/q/10071814
> http://stackoverflow.com/q/14992793
> https://issues.apache.org/jira/browse/CAMEL-5072
> http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-td5548338.html
>
> So obviously there has been some work to fix those issues. In one case Claus
> advised the user that he must stop a producer template. I'm not sure how to
> solve my problem. Find my setup description of a Quartz-triggered file
> consumer route below.
>
> A file consumer:
> from("file:///tmp/x/?sendEmptyMessageWhenIdle=true")
> .noAutoStartup()
> .routeId(routeId)
> .choice()
> .when(body().isNull())
> ...
>
> The null-Body is used as route-stop-trigger, I use a processor that starts
> up a thread as described here:
> http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
> ... with these actions:
> exchange.getContext().getInflightRepository().remove(exchange);
> exchange.getContext().stopRoute(exchange.getFromRouteId());
> I also tried a stop() on the getFromEndpoint() here.
>
>
> A Quartz route, periodically firing the file consumer route based on cron
> definitions:
> from("quartz://timer-"+routeId+"?stateful=true&cron=...")
> .routeId("quartz-"+routeId)
> .process(new RouteStartProcessor(routeId));
>
> process() in RouteStartProcessor:
> CamelContext context = exchange.getContext();
> context.startRoute(routeId);
>
> I now added a loop to the RouteStartProcessor waiting for the file consumer
> route to shutdown itself (to prevent overlaps), evaluated by:
> routeStopped = context.getRouteStatus(routeId).isStopped();
>
> This actually does work fine, so I assume that the route gets stopped
> successfully. Logs: o.a.c.impl.DefaultShutdownStrategy - Graceful shutdown
> of 1 routes completed in 0 seconds
>
> Whenever Quartz fires and the consumer route gets re-started this brings up
> a new thread id which I can see in the logs.
>
> Albeit the route obviously is shut down, the threads stay alive, all in
> state: java.lang.Thread.State: WAITING (parking)
>
> I dug around a bit in the VM's MBeans, there are:
> * 2 consumers, a FileConsumer and a QuartzConsumer
> * a large amount of threadpools (equal to the number of waiting threads),
> having IDs like FileConsumer(0x138dc688)
>
> Attributes of an example instance:
> Thread pool profile: --
> Shutdown: false
> Active count: 0
> Pool size: 1
> Task queue size: 0
> Route:
> Keep alive time: 0
> Source:
> Completed task count: 2
> Core pool size: 1
> Largest pool size: 1
> Task count: 2
> Id: FileConsumer(0x138dc688)
> Maximum pool size: 2147483647
> Task count: 2
>
>
> My environment:
> Linux
> JDK 7
> GlassFish 3
> CDI Camel context
> Camel 2.11.0
>
>
> Any hints? Thanks in advance!
>
> Regards, mdo.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/2-11-File2-Quartz-threads-leaking-tp5731922.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen