You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Chirag Dewan <ch...@yahoo.in> on 2014/02/04 12:33:18 UTC
Large Number of threads getting spawned in File Producer
Hi All,
I am using Camel 2.12.1. I have a route which consumes files from a FTP Server and places them on the local disk.
from("ftp:admin@ip:port/source_dir&throwExceptionOnConnectFailed=true&stepwise=false&readLock=changed&fastExistsCheck=true&localWorkDirectory=destination_dir/.temp").to("file:destination_dir").process(some processor);
Now I have 10000 files(around 10kbs in size) on the FTP source directory. When I start my route,I see a lot of "file:destination_dir" threads created and in TIME_WAITING state during profiling. Roughly,there is a thread for each file that the FTP Consumer picks from the FTP server(2000 threads for around 2000 files) after which the JVM crashes due to OOM.
It seems the file producer creates a new thread for every file that is consumed from its default thread pool.
How can I restrict the number of threads? Or why are all the threads are in a TIME_WAIT state?
Any help is highly appreciated.
Thanks!
Chirag
Re: Large Number of threads getting spawned in File Producer
Posted by Kondalarv <ko...@gmail.com>.
Hi Chirag,
Did you resolve this issue? is so what is the root cause of this problem?.
--
View this message in context: http://camel.465427.n5.nabble.com/Large-Number-of-threads-getting-spawned-in-File-Producer-tp5746781p5767667.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Large Number of threads getting spawned in File Producer
Posted by Kondalarv <ko...@gmail.com>.
Hi Chirag,
Did you resolve this issue? If so, what is the root cause of this problem?.
--
View this message in context: http://camel.465427.n5.nabble.com/Large-Number-of-threads-getting-spawned-in-File-Producer-tp5746781p5767668.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Large Number of threads getting spawned in File Producer
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Its maybe storm then.
All threads created by Camel is prefixed with Camel by default. There
is a naming syntax you can configure
http://camel.apache.org/threading-model.html
On Wed, Feb 5, 2014 at 10:49 AM, Chirag Dewan <ch...@yahoo.in> wrote:
> Hi Claus,
>
> My route is running in a Storm cluster. And all I do in my processor is call a listener to emit the File name and the produced file path to the Spout which passes it on to the Bolts.
>
> The thread name which I can see in my Profiler is "file:///destination_dir" which gives an impression that its the File Producer threads that are continously growing in number.
>
> I will quickly run a simple test from ftp -> file without any processing and post the results here.
>
> Thanks a lot!
>
> Chirag
>
>
>
>
>
>
> ________________________________
> From: Claus Ibsen <cl...@gmail.com>
> To: "users@camel.apache.org" <us...@camel.apache.org>
> Sent: Wednesday, 5 February 2014 1:27 PM
> Subject: Re: Large Number of threads getting spawned in File Producer
>
>
> The ftp consumer in Camel is a single threaded scheduled thread pool.
>
> So if threads is spawned it happens somewhere else. Maybe you do
> something in that processor that causes threads to be spawned. You can
> try to just do from ftp -> to file or from ftp -> to log and see if
> there is some changes.
>
> All the threads pool used by Camel are enlisted in JMX which you can
> see. Maybe that can also give some pointers.
>
>
>
>
>
> On Tue, Feb 4, 2014 at 12:33 PM, Chirag Dewan <ch...@yahoo.in> wrote:
>> Hi All,
>>
>> I am using Camel 2.12.1. I have a route which consumes files from a FTP Server and places them on the local disk.
>>
>> from("ftp:admin@ip:port/source_dir&throwExceptionOnConnectFailed=true&stepwise=false&readLock=changed&fastExistsCheck=true&localWorkDirectory=destination_dir/.temp").to("file:destination_dir").process(some processor);
>>
>> Now I have 10000 files(around 10kbs in size) on the FTP source directory. When I start my route,I see a lot of "file:destination_dir" threads created and in TIME_WAITING state during profiling. Roughly,there is a thread for each file that the FTP Consumer picks from the FTP server(2000 threads for around 2000 files) after which the JVM crashes due to OOM.
>>
>> It seems the file producer creates a new thread for every file that is consumed from its default thread pool.
>>
>> How can I restrict the number of threads? Or why are all the threads are in a TIME_WAIT state?
>>
>> Any help is highly appreciated.
>>
>> Thanks!
>>
>> Chirag
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Re: Large Number of threads getting spawned in File Producer
Posted by Chirag Dewan <ch...@yahoo.in>.
Hi Claus,
My route is running in a Storm cluster. And all I do in my processor is call a listener to emit the File name and the produced file path to the Spout which passes it on to the Bolts.
The thread name which I can see in my Profiler is "file:///destination_dir" which gives an impression that its the File Producer threads that are continously growing in number.
I will quickly run a simple test from ftp -> file without any processing and post the results here.
Thanks a lot!
Chirag
________________________________
From: Claus Ibsen <cl...@gmail.com>
To: "users@camel.apache.org" <us...@camel.apache.org>
Sent: Wednesday, 5 February 2014 1:27 PM
Subject: Re: Large Number of threads getting spawned in File Producer
The ftp consumer in Camel is a single threaded scheduled thread pool.
So if threads is spawned it happens somewhere else. Maybe you do
something in that processor that causes threads to be spawned. You can
try to just do from ftp -> to file or from ftp -> to log and see if
there is some changes.
All the threads pool used by Camel are enlisted in JMX which you can
see. Maybe that can also give some pointers.
On Tue, Feb 4, 2014 at 12:33 PM, Chirag Dewan <ch...@yahoo.in> wrote:
> Hi All,
>
> I am using Camel 2.12.1. I have a route which consumes files from a FTP Server and places them on the local disk.
>
> from("ftp:admin@ip:port/source_dir&throwExceptionOnConnectFailed=true&stepwise=false&readLock=changed&fastExistsCheck=true&localWorkDirectory=destination_dir/.temp").to("file:destination_dir").process(some processor);
>
> Now I have 10000 files(around 10kbs in size) on the FTP source directory. When I start my route,I see a lot of "file:destination_dir" threads created and in TIME_WAITING state during profiling. Roughly,there is a thread for each file that the FTP Consumer picks from the FTP server(2000 threads for around 2000 files) after which the JVM crashes due to OOM.
>
> It seems the file producer creates a new thread for every file that is consumed from its default thread pool.
>
> How can I restrict the number of threads? Or why are all the threads are in a TIME_WAIT state?
>
> Any help is highly appreciated.
>
> Thanks!
>
> Chirag
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Re: Large Number of threads getting spawned in File Producer
Posted by Claus Ibsen <cl...@gmail.com>.
The ftp consumer in Camel is a single threaded scheduled thread pool.
So if threads is spawned it happens somewhere else. Maybe you do
something in that processor that causes threads to be spawned. You can
try to just do from ftp -> to file or from ftp -> to log and see if
there is some changes.
All the threads pool used by Camel are enlisted in JMX which you can
see. Maybe that can also give some pointers.
On Tue, Feb 4, 2014 at 12:33 PM, Chirag Dewan <ch...@yahoo.in> wrote:
> Hi All,
>
> I am using Camel 2.12.1. I have a route which consumes files from a FTP Server and places them on the local disk.
>
> from("ftp:admin@ip:port/source_dir&throwExceptionOnConnectFailed=true&stepwise=false&readLock=changed&fastExistsCheck=true&localWorkDirectory=destination_dir/.temp").to("file:destination_dir").process(some processor);
>
> Now I have 10000 files(around 10kbs in size) on the FTP source directory. When I start my route,I see a lot of "file:destination_dir" threads created and in TIME_WAITING state during profiling. Roughly,there is a thread for each file that the FTP Consumer picks from the FTP server(2000 threads for around 2000 files) after which the JVM crashes due to OOM.
>
> It seems the file producer creates a new thread for every file that is consumed from its default thread pool.
>
> How can I restrict the number of threads? Or why are all the threads are in a TIME_WAIT state?
>
> Any help is highly appreciated.
>
> Thanks!
>
> Chirag
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Re: Large Number of threads getting spawned in File Producer
Posted by Chirag Dewan <ch...@yahoo.in>.
Hi Richard,
Its not the FTP Consumer threads. I can see in my Profiler,a single threaded FTP Consumer. Its the File Producer which has an increasing number of threads. Can I somehow configure a thread pool for the File Producer?
Thanks a lot!
Chirag
________________________________
From: Richard Kettelerij <ri...@gmail.com>
To: "users@camel.apache.org" <us...@camel.apache.org>
Sent: Wednesday, 5 February 2014 1:31 PM
Subject: Re: Large Number of threads getting spawned in File Producer
Hi,
I'd be surprised if a thread was started for every file. By default the
file consumer is using a thread pool of size 1.
Anyway you can customize this by specifying a ScheduledExecutorService on
the endpoint (e.g. ftp://.....&scheduledExecutorService=#myExecutorService)
where myExecutorService is a threadPoolProfile which you specify on the
CamelContext. See http://camel.apache.org/threading-model.html for details.
Regards,
Richard
On Wed, Feb 5, 2014 at 5:23 AM, Chirag Dewan <ch...@yahoo.in>wrote:
> Hi All,
>
> Any views on this?
>
> BR,
>
> Chirag
>
>
Re: Large Number of threads getting spawned in File Producer
Posted by Richard Kettelerij <ri...@gmail.com>.
Hi,
I'd be surprised if a thread was started for every file. By default the
file consumer is using a thread pool of size 1.
Anyway you can customize this by specifying a ScheduledExecutorService on
the endpoint (e.g. ftp://.....&scheduledExecutorService=#myExecutorService)
where myExecutorService is a threadPoolProfile which you specify on the
CamelContext. See http://camel.apache.org/threading-model.html for details.
Regards,
Richard
On Wed, Feb 5, 2014 at 5:23 AM, Chirag Dewan <ch...@yahoo.in>wrote:
> Hi All,
>
> Any views on this?
>
> BR,
>
> Chirag
>
>
Re: Large Number of threads getting spawned in File Producer
Posted by Chirag Dewan <ch...@yahoo.in>.
Hi All,
Any views on this?
BR,
Chirag