You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by ShlomiJ <sh...@gmail.com> on 2012/03/12 14:57:57 UTC
WireTap threads number keeps on growing
hi,
I'm having problems using the Wire Tap endpoint. *Every time a message
passes in the route, a new thread of wire tap is created, stays in "wait"
state and never dies*. Eventually my application is full of threads.
I'm creating the following ExecutorService in the registry:
ThreadPoolBuilder tpBuilder = new ThreadPoolBuilder(this.context);
> ExecutorService wiretapThreadpool =
> tpBuilder.poolSize(1).maxPoolSize(1).maxQueueSize(1).build("WireTap");
> registry.put(WIRETAP_THREADPOOL, wiretapThreadpool);
My route is simple, triggered by quartz, adds properties to the message,
wire-taps and logs:
from("quartz://myScheduler?cron=0+*+*+*+*+?")
> .process(new MyMessageSetterProcessor())
> .wireTap("seda:wireTapQueue").onPrepare(new
> MyCloneProcessor()).executorServiceRef(WIRETAP_THREADPOOL)
> .log("${body}")
Am I using the WireTap wrong?
Am I not creating the thread pool correctly?
thanks
ShlomiJ
--
View this message in context: http://camel.465427.n5.nabble.com/WireTap-threads-number-keeps-on-growing-tp5557827p5557827.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: WireTap threads number keeps on growing
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Mar 13, 2012 at 3:18 PM, ShlomiJ <sh...@gmail.com> wrote:
>
> Claus Ibsen-2 wrote
>> When you provide an existing thread pool to any Camel EIPs, such as
>> the WireTap, then
>> YOU are in control of the lifecycle of the thread pool. So you have to
>> shutdown this thread pool yourself,
>> when its no longer in-use.
> I understand your point about the life cycle. But I have a problem with the
> *executorServiceRef*.
> I use executorServiceRef(WIRETAP_THREADPOOL), the thread pool (TP) is at the
> registry, and I'm in control of its life cycle.
> But actually each call to *wireTap creates its own TP*, NOT using the one
> from the registry.
>
> I would expect executorServiceRef to use the TP it is giving, not create a
> new TP. It will also allow me to use the registry-key to get hold of the TP
> and release it.
>
Yes but I said there was most likely a bug causing Camel to re-create a new TP,
try with next release of Camel or the 2.9-SNAPSHOT, or the 2.10-SNAPSHOT.
http://camel.apache.org/download
>
> ShlomiJ wrote
>> ThreadPoolBuilder tpBuilder = new ThreadPoolBuilder(this.context);
>> ExecutorService wiretapThreadpool =
>> tpBuilder.poolSize(1).maxPoolSize(1).maxQueueSize(1).build("WireTap");
>> registry.put(WIRETAP_THREADPOOL, wiretapThreadpool);
> BTW, I also checked it with /registry.put(WIRETAP_THREADPOOL,
> java.util.concurrentExecutors.newCachedThreadPool())/ (in case there was
> something wrong with the way I used the ThreadPoolBuilder).
>
> In anyway, in that particular place (many quartz-routes that are removed and
> created again) I switch for using multicast.
>
> thanks for the help
> SJ
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/WireTap-threads-number-keeps-on-growing-tp5557827p5561055.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: WireTap threads number keeps on growing
Posted by ShlomiJ <sh...@gmail.com>.
Claus Ibsen-2 wrote
> When you provide an existing thread pool to any Camel EIPs, such as
> the WireTap, then
> YOU are in control of the lifecycle of the thread pool. So you have to
> shutdown this thread pool yourself,
> when its no longer in-use.
I understand your point about the life cycle. But I have a problem with the
*executorServiceRef*.
I use executorServiceRef(WIRETAP_THREADPOOL), the thread pool (TP) is at the
registry, and I'm in control of its life cycle.
But actually each call to *wireTap creates its own TP*, NOT using the one
from the registry.
I would expect executorServiceRef to use the TP it is giving, not create a
new TP. It will also allow me to use the registry-key to get hold of the TP
and release it.
ShlomiJ wrote
> ThreadPoolBuilder tpBuilder = new ThreadPoolBuilder(this.context);
> ExecutorService wiretapThreadpool =
> tpBuilder.poolSize(1).maxPoolSize(1).maxQueueSize(1).build("WireTap");
> registry.put(WIRETAP_THREADPOOL, wiretapThreadpool);
BTW, I also checked it with /registry.put(WIRETAP_THREADPOOL,
java.util.concurrentExecutors.newCachedThreadPool())/ (in case there was
something wrong with the way I used the ThreadPoolBuilder).
In anyway, in that particular place (many quartz-routes that are removed and
created again) I switch for using multicast.
thanks for the help
SJ
--
View this message in context: http://camel.465427.n5.nabble.com/WireTap-threads-number-keeps-on-growing-tp5557827p5561055.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: WireTap threads number keeps on growing
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
See ticket
https://issues.apache.org/jira/browse/CAMEL-5079
On Tue, Mar 13, 2012 at 11:00 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Mar 13, 2012 at 9:13 AM, ShlomiJ <sh...@gmail.com> wrote:
>> OK, I'm a bit more knowledgeable now...
>>
>> The first issue is that I thought I created a pool of 1 thread.
>>
>> ShlomiJ wrote
>>> ThreadPoolBuilder tpBuilder = new ThreadPoolBuilder(this.context);
>>> ExecutorService wiretapThreadpool =
>>> tpBuilder.poolSize(*1*).maxPoolSize(*1*).maxQueueSize(*1*).build("WireTap");
>>> registry.put(WIRETAP_THREADPOOL, wiretapThreadpool);
>> But I did not. The pool created was actually of 10 threads.
>> So once 10 messages passed in the route, all 10 threads have been created
>> and reused.
>>
>
> There was an issue in Camel 2.9.1 or older. Can you try with trunk. It
> ought to use the thread pool as given.
>
>
>> Another issue is the number of calls to wireTap and the way the pool is
>> used.
>>
>> ShlomiJ wrote
>>> from("quartz://myScheduler?cron=0+*+*+*+*+?")
>>> .process(new MyMessageSetterProcessor())
>>> .wireTap("seda:wireTapQueue").onPrepare(new
>>> MyCloneProcessor()).*executorServiceRef*(WIRETAP_THREADPOOL)
>>> .log("${body}")
>> I now understand that each route has its own pool. So I have (10 *
>> #schedules) threads of wiretap.
>>
>> *But the real issue is that the wiretap threads (maybe others also?) are not
>> removed when the route is removed.*
>> I remove a route with CamelContext.getRoute(String) ->
>> CamelContext.stopRoute(String) -> CamelContext.removeRoute(String).
>> The last method returns true, but the wiretap threads are still alive.
>>
>
> When you provide an existing thread pool to any Camel EIPs, such as
> the WireTap, then
> YOU are in control of the lifecycle of the thread pool. So you have to
> shutdown this thread pool yourself,
> when its no longer in-use. Camel does not know whether it would be
> safe to shutdown a shared thread pool it was given.
>
> Only if Camel itself creates the thread pool, then it handles the
> lifecycle, and will shutdown the thread pool itself,
> but that happens normally only when Camel is shutdown itself.
>
> We could possible improve this to detect in the EIPs whether the
> thread pool can safely be shutdown when the route
> is removed or not. I will log a JIRA about this.
>
>
>> I'm using Camel 2.9.0.
>> thanks
>> SJ
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/WireTap-threads-number-keeps-on-growing-tp5557827p5560267.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: WireTap threads number keeps on growing
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Mar 13, 2012 at 9:13 AM, ShlomiJ <sh...@gmail.com> wrote:
> OK, I'm a bit more knowledgeable now...
>
> The first issue is that I thought I created a pool of 1 thread.
>
> ShlomiJ wrote
>> ThreadPoolBuilder tpBuilder = new ThreadPoolBuilder(this.context);
>> ExecutorService wiretapThreadpool =
>> tpBuilder.poolSize(*1*).maxPoolSize(*1*).maxQueueSize(*1*).build("WireTap");
>> registry.put(WIRETAP_THREADPOOL, wiretapThreadpool);
> But I did not. The pool created was actually of 10 threads.
> So once 10 messages passed in the route, all 10 threads have been created
> and reused.
>
There was an issue in Camel 2.9.1 or older. Can you try with trunk. It
ought to use the thread pool as given.
> Another issue is the number of calls to wireTap and the way the pool is
> used.
>
> ShlomiJ wrote
>> from("quartz://myScheduler?cron=0+*+*+*+*+?")
>> .process(new MyMessageSetterProcessor())
>> .wireTap("seda:wireTapQueue").onPrepare(new
>> MyCloneProcessor()).*executorServiceRef*(WIRETAP_THREADPOOL)
>> .log("${body}")
> I now understand that each route has its own pool. So I have (10 *
> #schedules) threads of wiretap.
>
> *But the real issue is that the wiretap threads (maybe others also?) are not
> removed when the route is removed.*
> I remove a route with CamelContext.getRoute(String) ->
> CamelContext.stopRoute(String) -> CamelContext.removeRoute(String).
> The last method returns true, but the wiretap threads are still alive.
>
When you provide an existing thread pool to any Camel EIPs, such as
the WireTap, then
YOU are in control of the lifecycle of the thread pool. So you have to
shutdown this thread pool yourself,
when its no longer in-use. Camel does not know whether it would be
safe to shutdown a shared thread pool it was given.
Only if Camel itself creates the thread pool, then it handles the
lifecycle, and will shutdown the thread pool itself,
but that happens normally only when Camel is shutdown itself.
We could possible improve this to detect in the EIPs whether the
thread pool can safely be shutdown when the route
is removed or not. I will log a JIRA about this.
> I'm using Camel 2.9.0.
> thanks
> SJ
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/WireTap-threads-number-keeps-on-growing-tp5557827p5560267.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: WireTap threads number keeps on growing
Posted by ShlomiJ <sh...@gmail.com>.
OK, I'm a bit more knowledgeable now...
The first issue is that I thought I created a pool of 1 thread.
ShlomiJ wrote
> ThreadPoolBuilder tpBuilder = new ThreadPoolBuilder(this.context);
> ExecutorService wiretapThreadpool =
> tpBuilder.poolSize(*1*).maxPoolSize(*1*).maxQueueSize(*1*).build("WireTap");
> registry.put(WIRETAP_THREADPOOL, wiretapThreadpool);
But I did not. The pool created was actually of 10 threads.
So once 10 messages passed in the route, all 10 threads have been created
and reused.
Another issue is the number of calls to wireTap and the way the pool is
used.
ShlomiJ wrote
> from("quartz://myScheduler?cron=0+*+*+*+*+?")
> .process(new MyMessageSetterProcessor())
> .wireTap("seda:wireTapQueue").onPrepare(new
> MyCloneProcessor()).*executorServiceRef*(WIRETAP_THREADPOOL)
> .log("${body}")
I now understand that each route has its own pool. So I have (10 *
#schedules) threads of wiretap.
*But the real issue is that the wiretap threads (maybe others also?) are not
removed when the route is removed.*
I remove a route with CamelContext.getRoute(String) ->
CamelContext.stopRoute(String) -> CamelContext.removeRoute(String).
The last method returns true, but the wiretap threads are still alive.
I'm using Camel 2.9.0.
thanks
SJ
--
View this message in context: http://camel.465427.n5.nabble.com/WireTap-threads-number-keeps-on-growing-tp5557827p5560267.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: WireTap threads number keeps on growing
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
What version of Camel are you using?
On Mon, Mar 12, 2012 at 2:57 PM, ShlomiJ <sh...@gmail.com> wrote:
> hi,
>
> I'm having problems using the Wire Tap endpoint. *Every time a message
> passes in the route, a new thread of wire tap is created, stays in "wait"
> state and never dies*. Eventually my application is full of threads.
>
> I'm creating the following ExecutorService in the registry:
>
> ThreadPoolBuilder tpBuilder = new ThreadPoolBuilder(this.context);
>> ExecutorService wiretapThreadpool =
>> tpBuilder.poolSize(1).maxPoolSize(1).maxQueueSize(1).build("WireTap");
>> registry.put(WIRETAP_THREADPOOL, wiretapThreadpool);
> My route is simple, triggered by quartz, adds properties to the message,
> wire-taps and logs:
>
> from("quartz://myScheduler?cron=0+*+*+*+*+?")
>> .process(new MyMessageSetterProcessor())
>> .wireTap("seda:wireTapQueue").onPrepare(new
>> MyCloneProcessor()).executorServiceRef(WIRETAP_THREADPOOL)
>> .log("${body}")
>
> Am I using the WireTap wrong?
> Am I not creating the thread pool correctly?
>
> thanks
> ShlomiJ
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/WireTap-threads-number-keeps-on-growing-tp5557827p5557827.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/