You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Aida <ai...@gmail.com> on 2012/03/08 19:04:28 UTC
[FTP + ConsumerTemplate] Threads remaining alive
Hi all.
I am using the FTP component (Camel 2.9.0) and a Consumer Template for
reading files. The thing is that, while doing stress tests (reading 1100
files in parallel) and using the JConsole, I realised that there are one
thread per file that is not killed (in this case, 1100 threads remain
alive).
A little more information:
- Each file is different because I am trying to simulate a long running
production enviroment (in the “real world”, I have to read 20 differents
files per day, so I set up this scenario for 505 days))
- The consumer template is started before reading and stoped after:
try{
//start the template
consumerTemplate.start();
if(timeout == 0){
dataExchange =
consumerTemplate.receiveNoWait(endpoint2Consume);
}
else{
dataExchange =
consumerTemplate.receive(endpoint2Consume, timeout);
}
}catch (Exception e) {
//null the exchange to throw the exception after (dirty
trick, I know)
dataExchange = null;
}
finally{
//stops the template (release threads)
consumerTemplate.stop();
}
I also "make the unit of work done": consumerTemplate.doneUoW(dataExchange
);
I have been investigating both, the ftp component and the consumer template
usage but I'm not able to see where/what the problem is.
Thanks in advance.
Aida.
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5548338.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Hi,
I have been investigating more deeply, and I have this stacktrace. The
problem is solved for me removing the "disconnect" option from the endpoints
consumed by the consumerTemplate (but I add here the stacktrace in case it
can help to somebody else). Otherwise, sometimes a NullPointerException is
thrown and a thread remains alive.
Thanks.
-------------------------------------------------------------------------------------
012-12-07 14:09:05,487 FATAL [myPackage.processor.MyProcessor]
(pool-27-thread-8) Exception launched while stopping consumerTemplate: null
java.lang.NullPointerException
at org.apache.commons.net.ftp.FTP.__send(FTP.java:496)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:470)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:547)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:596)
at org.apache.commons.net.ftp.FTP.quit(FTP.java:804)
at org.apache.commons.net.ftp.FTPClient.logout(FTPClient.java:941)
at
org.apache.camel.component.file.remote.FtpOperations.disconnect(FtpOperations.java:197)
at
org.apache.camel.component.file.remote.RemoteFileConsumer.disconnect(RemoteFileConsumer.java:113)
at
org.apache.camel.component.file.remote.RemoteFileConsumer.doStop(RemoteFileConsumer.java:100)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:109)
at
org.apache.camel.impl.EventDrivenPollingConsumer.doStop(EventDrivenPollingConsumer.java:153)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:109)
at
org.apache.camel.util.ServiceHelper.stopAndShutdownServices(ServiceHelper.java:177)
at org.apache.camel.impl.ConsumerCache.doStop(ConsumerCache.java:238)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:109)
at
org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:155)
at
org.apache.camel.impl.DefaultConsumerTemplate.doStop(DefaultConsumerTemplate.java:268)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
at
...
some of my classes
...
at sun.reflect.GeneratedMethodAccessor716.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:340)
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:237)
at
org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:166)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:304)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:85)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:304)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122)
at
org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:50)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:117)
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:291)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:112)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:304)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:81)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:304)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122)
at
org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:50)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.impl.InterceptSendToEndpoint$1.process(InterceptSendToEndpoint.java:154)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:117)
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:291)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:112)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:304)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:703)
at
org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:80)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:289)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:274)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
-------------------------------------------------------------------------------------
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5723821.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Hi,
Thank you very much, as you said doing that the threads are cleaned up :) I
should have tried that before, my apologies ...
I'm sorry to bother you, but after using that approach and consuming files
intensively I have seen that sometimes the consumerTemplate cannot be
stoppped, and, in that case the thread created for that consumption is not
released, neither in the moment to do the stop nor forward in the time (when
the next time the consumer template is used a start/stop is done).
I have seen that a NullpointerException is thrown (with no trace and no
localized message). I think this happen when calling service.shutdown();
from the method stopAndShutdownService of the ServiceHelper class. I haven´t
been able to delimit more the problem for the moment.
Thanks in advance.
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5723622.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Dec 3, 2012 at 4:57 PM, Aida <ai...@gmail.com> wrote:
> Hi again,
>
> I fear that after upgrading to Camel 2.9.4 this problem still remains, at
> least for me (I see that the issue was fixed for 2.9.2 and 2.10.0). I
> couldn´t test it with Camel 2.10.0 because of cxf incompatibilities.
>
> I have been doing some tests in real and I have been able to "produce" 5000
> threads living at the same time (and never dying), one per endpoint that the
> consumerTemplate tried to consume. At first I thought that the threads could
> have something to do with the LRUCache that the DefaultConsumerTemplate has,
> but since its maximun size is 1000 by default and I had about 5000 threads I
> discarded that option and I'm looking for other solution (without success
> for the moment).
>
> As a reminder of this forum thread, my problem was that when consuming a
> file (from a file/ftp endpoint) with the consumerTemplate, a thread remained
> alive for each different endpoint that was consumed.
>
> I'm using JBoss running in a Windows machine, and the JConsole to see the
> threads that are alive.
>
> If anyone could help or tell me what I can try, or what would be the next
> step to keep on looking for the solution would be great.
>
> Thanks in advance.
>
You need to stop the consumer template if you are not going to re-use
any endpoints.
Then any running threads etc is cleaned up.
> Aida.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5723551.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Hi again,
I fear that after upgrading to Camel 2.9.4 this problem still remains, at
least for me (I see that the issue was fixed for 2.9.2 and 2.10.0). I
couldn´t test it with Camel 2.10.0 because of cxf incompatibilities.
I have been doing some tests in real and I have been able to "produce" 5000
threads living at the same time (and never dying), one per endpoint that the
consumerTemplate tried to consume. At first I thought that the threads could
have something to do with the LRUCache that the DefaultConsumerTemplate has,
but since its maximun size is 1000 by default and I had about 5000 threads I
discarded that option and I'm looking for other solution (without success
for the moment).
As a reminder of this forum thread, my problem was that when consuming a
file (from a file/ftp endpoint) with the consumerTemplate, a thread remained
alive for each different endpoint that was consumed.
I'm using JBoss running in a Windows machine, and the JConsole to see the
threads that are alive.
If anyone could help or tell me what I can try, or what would be the next
step to keep on looking for the solution would be great.
Thanks in advance.
Aida.
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5723551.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Mar 9, 2012 at 1:31 PM, Aida <ai...@gmail.com> wrote:
> Ok, thank you Claus.
>
> I am looking forward to the improvement. Meanwhile I will use a filter class
> as you suggested to minimize the impact.
>
The fix/improvement has been committed to trunk and the 2.9.x branch.
> Thanks again.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550436.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Ok, thank you Claus.
I am looking forward to the improvement. Meanwhile I will use a filter class
as you suggested to minimize the impact.
Thanks again.
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550436.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I logged a JIRA to improve this to shutdown the thread pools more eagerly
https://issues.apache.org/jira/browse/CAMEL-5072
On Fri, Mar 9, 2012 at 12:39 PM, Aida <ai...@gmail.com> wrote:
> Here is:
>
> --------------------------------------------------------------------------------------------------
> Name: Camel (processesCamelContext) thread #22 -
> ftp://ftp_user@test_host.com:21/readFileDir
> State: WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36bc16b2
> Total blocked: 9 Total waited: 4
>
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> java.lang.Thread.run(Thread.java:662)
> --------------------------------------------------------------------------------------------------
>
> Thanks
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550328.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Here is:
--------------------------------------------------------------------------------------------------
Name: Camel (processesCamelContext) thread #22 -
ftp://ftp_user@test_host.com:21/readFileDir
State: WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36bc16b2
Total blocked: 9 Total waited: 4
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
--------------------------------------------------------------------------------------------------
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550328.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Mar 9, 2012 at 11:58 AM, Aida <ai...@gmail.com> wrote:
> Hi again,
>
> I have upgraded to camel 2.9.1 in order to avoid CAMEL-4976 but there is no
> changes.
>
> I also try to stop the endpoint (no changes again) and use the
> ServiceHelper.stopAndShutdownService(enpoint) with no success.
>
> The EndpointRegistry's size remains stable although I can still see the
> created threads (it seems that they are not registred).
>
> The idea you suggest (using a filter strategy and one common endpoint) may
> be an option, but the remote folder that I have to read has lots (thousands)
> of files and I’m worried about performance.
>
> Do you have any other idea?
>
Can you post a thread dump of one of the threads that you see?
> I fear that maybe I have to change my approach to workaround the issue.
>
> Thanks
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550245.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Hi again,
I have upgraded to camel 2.9.1 in order to avoid CAMEL-4976 but there is no
changes.
I also try to stop the endpoint (no changes again) and use the
ServiceHelper.stopAndShutdownService(enpoint) with no success.
The EndpointRegistry's size remains stable although I can still see the
created threads (it seems that they are not registred).
The idea you suggest (using a filter strategy and one common endpoint) may
be an option, but the remote folder that I have to read has lots (thousands)
of files and I’m worried about performance.
Do you have any other idea?
I fear that maybe I have to change my approach to workaround the issue.
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550245.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Mar 9, 2012 at 10:50 AM, Aida <ai...@gmail.com> wrote:
> Hi Claus,
>
> This is quite weird but calling ”camelContext.removeEndpoints(endpointUri)”
> has no effect, the threads still there (from now, the inmortal thread, ;-D)
>
Can you try to stop the endpoint before removing it.
Also see CAMEL-4976 as there may was a problem with this.
> Furthermore, if I call “camelContext.getEndpoints()” these endpoints are not
> in the list.
>
Instead of using different endpoints all the time. You can use the
same endpoint.
And then use a filter class, where you in the filter class can check
if you accept the file or not.
Also see this FAQ
http://camel.apache.org/how-do-i-configure-the-maximum-endpoint-cache-size-for-camelcontext.html
> If it helps, I am using a spring based camel context.
>
> Moreover, I see that the GenericFileEndpoint are “not lenient” by default
> (line 326 in DefaultEnpoint) so when I get the endpoint key, the properties
> (filename among others) are used to create the key, so I always get a
> different endpoint for each file read, isn’t it?
>
> Thanks again for your time
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550086.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Hi Claus,
This is quite weird but calling ”camelContext.removeEndpoints(endpointUri)”
has no effect, the threads still there (from now, the inmortal thread, ;-D)
Furthermore, if I call “camelContext.getEndpoints()” these endpoints are not
in the list.
If it helps, I am using a spring based camel context.
Moreover, I see that the GenericFileEndpoint are “not lenient” by default
(line 326 in DefaultEnpoint) so when I get the endpoint key, the properties
(filename among others) are used to create the key, so I always get a
different endpoint for each file read, isn’t it?
Thanks again for your time
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5550086.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Claus Ibsen <cl...@gmail.com>.
CamelContext has API to remove endpoint(s). You can use that.
On Fri, Mar 9, 2012 at 10:15 AM, Aida <ai...@gmail.com> wrote:
> Hi,
>
> First of all, thanks for the quick response.
>
> I will try to explain with more details what I am doing:
> 1) I build the endpoint uri to retrieve the file from the FTP server. Each
> day, the file has a different name (based on date), so the endpoint has to
> be created dynamically.
> 2) Save the retrieved file by sending the exchange to a local directory
> (using a file endpoint).
> 3) Finally, I done the UoW on the original retrieved exchange.
>
> I am using camel 2.9.0.
>
> Here is my theory:
>
> Each file has a different name (is based on the current date time) so, for
> each file I have to read, I have to build a different endpoint uri.
>
> I think that the alive threads are these endpoints, because with JConsole I
> can see lots of threads with this “name”:
>
> Name: Camel (processesCamelContext) thread #22 -
> ftp://ftp_user@test_host.com:21/readFileDir
> State: TIMED_WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4279f121
> Total blocked: 4 Total waited: 942
>
> When calling the “consumerTemplate.receive(endpointUri)”, a different
> Endpoint is created for each file, and this endpoint is added to the
> registry and is never removed (calling the “doneUoW” seem has no effect).
>
> I also try to done the UoW before sending the exchange to the file endpoint
> but the alive threads are still there.
>
> makes it sense for you or I’m missing something?
>
> If it is correct, Is there any way to remove these created endpoints? Maybe
> using placeholders for the properties (host, password, filename…) I can
> avoid to create a different endpoint each time? (I can live with one thread
> living always, but no one for each read)
>
> Thanks for your help
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5549997.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Aida <ai...@gmail.com>.
Hi,
First of all, thanks for the quick response.
I will try to explain with more details what I am doing:
1) I build the endpoint uri to retrieve the file from the FTP server. Each
day, the file has a different name (based on date), so the endpoint has to
be created dynamically.
2) Save the retrieved file by sending the exchange to a local directory
(using a file endpoint).
3) Finally, I done the UoW on the original retrieved exchange.
I am using camel 2.9.0.
Here is my theory:
Each file has a different name (is based on the current date time) so, for
each file I have to read, I have to build a different endpoint uri.
I think that the alive threads are these endpoints, because with JConsole I
can see lots of threads with this “name”:
Name: Camel (processesCamelContext) thread #22 -
ftp://ftp_user@test_host.com:21/readFileDir
State: TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4279f121
Total blocked: 4 Total waited: 942
When calling the “consumerTemplate.receive(endpointUri)”, a different
Endpoint is created for each file, and this endpoint is added to the
registry and is never removed (calling the “doneUoW” seem has no effect).
I also try to done the UoW before sending the exchange to the file endpoint
but the alive threads are still there.
makes it sense for you or I’m missing something?
If it is correct, Is there any way to remove these created endpoints? Maybe
using placeholders for the properties (host, password, filename…) I can
avoid to create a different endpoint each time? (I can live with one thread
living always, but no one for each read)
Thanks for your help
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5549997.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Read the java doc of the consumerTemplate
http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/ConsumerTemplate.html
You need to done the UoW on the exchange.
On Thu, Mar 8, 2012 at 7:04 PM, Aida <ai...@gmail.com> wrote:
> Hi all.
>
> I am using the FTP component (Camel 2.9.0) and a Consumer Template for
> reading files. The thing is that, while doing stress tests (reading 1100
> files in parallel) and using the JConsole, I realised that there are one
> thread per file that is not killed (in this case, 1100 threads remain
> alive).
>
> A little more information:
> - Each file is different because I am trying to simulate a long running
> production enviroment (in the “real world”, I have to read 20 differents
> files per day, so I set up this scenario for 505 days))
> - The consumer template is started before reading and stoped after:
> try{
> //start the template
> consumerTemplate.start();
> if(timeout == 0){
> dataExchange =
> consumerTemplate.receiveNoWait(endpoint2Consume);
> }
> else{
> dataExchange =
> consumerTemplate.receive(endpoint2Consume, timeout);
> }
> }catch (Exception e) {
> //null the exchange to throw the exception after (dirty
> trick, I know)
> dataExchange = null;
> }
> finally{
> //stops the template (release threads)
> consumerTemplate.stop();
> }
>
> I also "make the unit of work done": consumerTemplate.doneUoW(dataExchange
> );
>
>
> I have been investigating both, the ftp component and the consumer template
> usage but I'm not able to see where/what the problem is.
>
> Thanks in advance.
>
> Aida.
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5548338.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [FTP + ConsumerTemplate] Threads remaining alive
Posted by Ashwin Karpe <ak...@fusesource.com>.
Hi,
Can you please provide further details such as
- Camel version
- Use-case details (Camel route details and where/how the consumer
Template is being utilized).
- Any exceptions being thrown
Unfortunately, it is not very straightforward to diagnose the issue based on
the limited information available via your post.
Also I assume you use DefaultConsumerTemplate instead of rolling your own
code... I looked at the DefaultConsumerTemplate code and do not see anything
suspect about it.
Please provide enough data to allow us to raise a Jira issue and tackle the
problem if it exists.
Cheers,
Ashwin...
-----
---------------------------------------------------------
Ashwin Karpe
Apache Camel Committer & Sr Principal Consultant
FUSESource (a Progress Software Corporation subsidiary)
http://fusesource.com
Blog: http://opensourceknowledge.blogspot.com
---------------------------------------------------------
--
View this message in context: http://camel.465427.n5.nabble.com/FTP-ConsumerTemplate-Threads-remaining-alive-tp5548338p5548620.html
Sent from the Camel - Users mailing list archive at Nabble.com.