Re: FTP performance with hundreds of concurrent routes.

[domenec <ds...@nextret.net>]

Claus Ibsen-2 wrote
> 
> What version of Camel do you use?
> And do you download or upload files to the FTP server? (consumer or
> producer in EIP terms)
> The Camel FTP consumer is by default single threaded, eg
> So if you have a route that does: (pseudo)
> from ftp
>   to bean
> Then only 1 thread is active.
> 

Thanks for the response (you and Rocco). There have been holidays in
between, please excuse the time gone so far.

Camel version in WEB-INF/lib is *-2.7.2.jar (although they left an strange
WEB-INF/lib/camel/*-2.7.1.jar subdir).

Concerning routes there are:

- 140 with from uri="ftp: to a file
- Another 300 of a local file to an ftp uri

I have no idea of what average file size is. Someone else developed, I was
asked to take a look.

Is it possible that being single threaded and having many FTP transfers
waiting in parallel makes the CPU go that high? See the bursts in the slides
I made next.

I did a JMX capture of one night of activity which includes some coments and
thread count over time and their state:
http://www.slideshare.net/domenex/jmx-capture

Thanks for any hint. Beer tokens in Barcelona offered in exchange :)


--
View this message in context: http://camel.465427.n5.nabble.com/FTP-performance-with-hundreds-of-concurrent-routes-tp5697247p5715757.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: FTP performance with hundreds of concurrent routes.

[domenec <ds...@nextret.net>]

Thanks for the tips, Claus.

--
View this message in context: http://camel.465427.n5.nabble.com/FTP-performance-with-hundreds-of-concurrent-routes-tp5697247p5715812.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: FTP performance with hundreds of concurrent routes.

[Claus Ibsen <cl...@gmail.com>]

On Mon, Jul 9, 2012 at 6:31 PM, domenec <ds...@nextret.net> wrote:
>
> Claus Ibsen-2 wrote
>>
>> What version of Camel do you use?
>> And do you download or upload files to the FTP server? (consumer or
>> producer in EIP terms)
>> The Camel FTP consumer is by default single threaded, eg
>> So if you have a route that does: (pseudo)
>> from ftp
>>   to bean
>> Then only 1 thread is active.
>>
>
> Thanks for the response (you and Rocco). There have been holidays in
> between, please excuse the time gone so far.
>
> Camel version in WEB-INF/lib is *-2.7.2.jar (although they left an strange
> WEB-INF/lib/camel/*-2.7.1.jar subdir).
>
> Concerning routes there are:
>
> - 140 with from uri="ftp: to a file
> - Another 300 of a local file to an ftp uri
>
> I have no idea of what average file size is. Someone else developed, I was
> asked to take a look.
>
> Is it possible that being single threaded and having many FTP transfers
> waiting in parallel makes the CPU go that high? See the bursts in the slides
> I made next.
>

Yeah each from has its own thread, so you have 140 + 300 = 440 threads.

Although they dont run at the same time. You can control how often
they run, and by default they run 2 times per second, which is very
fast. So for production usage you should raise this delay, by setting
delay to a higher value . You can have different values, so some
routes can run more frequently if they are more important to pickup
new files asap.

Also you should set the option useFixedDelay=true, when you have so
many routes. You can read about this option here, and the JDK javadoc
what it means.
http://camel.apache.org/file2

And from Camel 2.10 onwards you can configure a shared thread pool for
all your file/ftp consumers. So instead of having 440 threads, you can
have a thread pool with a max limit. Also with a pool the threads can
be shared among the consumers etc. Its the new
scheduledExecutorService option


> I did a JMX capture of one night of activity which includes some coments and
> thread count over time and their state:
> http://www.slideshare.net/domenex/jmx-capture
>
> Thanks for any hint. Beer tokens in Barcelona offered in exchange :)
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-performance-with-hundreds-of-concurrent-routes-tp5697247p5715757.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen