You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by ravishankarhassain <ra...@gmail.com> on 2017/01/09 11:55:49 UTC
How to increase the performance for SFTP file polling consumer
I am trying to create a file polling route which will poll the source SFTP
server and transfer the file to the destination SFTP server.
The file count is huge and the source can generate upto 270 - 300 files per
second where as the file size is in KB.
I am planing to have 3 instance (say A, B, C) of the same route in active
mode where each route is capable of handling and transfering 100 files per
each poll.
Once a file is picked for processing by say intance A then the same file
should not be picked for processing by the rest 2 active instance Intance B
& C.
I have created the below route and this route is transfering files ar the
rate of 5-6 files per second.
from("sftp://user:password@source.sftp.server.com/input"
+ "?readLock=changed"
+ "&readLockMinAge=1m"
+ "&readLockTimeout=70000"
+ "&readLockCheckInterval=5000"
+ "&delay=1000"
+ "&preMove=processing"
+ "&maxMessagesPerPoll=100"
+ "&move=../archive"
+ "&localWorkDirectory=Invoice_ProgressESB"
+ "&stepwise=false"
+ "&include=.*(txt)$")
.threads(30, 35)
.log(LoggingLevel.INFO, "downloading files from Source SFTP Server")
.to("sftp://user:password@destination.sftp.server.com/output")
.end()
Is there any other configuration that needs to be applied to achieve the
processing speed of 100 files per poll and to process the file between
instance in an
intelligent and efficient manner.
Any help suggestion or pointers is much appreciated .
--
View this message in context: http://camel.465427.n5.nabble.com/How-to-increase-the-performance-for-SFTP-file-polling-consumer-tp5792312.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to increase the performance for SFTP file polling consumer
Posted by Zoran Regvart <zo...@regvart.com>.
Hi,
there was a blog post recently addressing just that by Josh Reagan:
http://joshdreagan.github.io/2017/01/05/faster_file_consumption_with_camel/
see if that helps,
zoran
On Mon, Jan 9, 2017 at 12:55 PM, ravishankarhassain
<ra...@gmail.com> wrote:
> I am trying to create a file polling route which will poll the source SFTP
> server and transfer the file to the destination SFTP server.
> The file count is huge and the source can generate upto 270 - 300 files per
> second where as the file size is in KB.
>
> I am planing to have 3 instance (say A, B, C) of the same route in active
> mode where each route is capable of handling and transfering 100 files per
> each poll.
> Once a file is picked for processing by say intance A then the same file
> should not be picked for processing by the rest 2 active instance Intance B
> & C.
>
> I have created the below route and this route is transfering files ar the
> rate of 5-6 files per second.
>
>
> from("sftp://user:password@source.sftp.server.com/input"
> + "?readLock=changed"
> + "&readLockMinAge=1m"
> + "&readLockTimeout=70000"
> + "&readLockCheckInterval=5000"
> + "&delay=1000"
> + "&preMove=processing"
> + "&maxMessagesPerPoll=100"
> + "&move=../archive"
> + "&localWorkDirectory=Invoice_ProgressESB"
> + "&stepwise=false"
> + "&include=.*(txt)$")
> .threads(30, 35)
> .log(LoggingLevel.INFO, "downloading files from Source SFTP Server")
> .to("sftp://user:password@destination.sftp.server.com/output")
> .end()
>
> Is there any other configuration that needs to be applied to achieve the
> processing speed of 100 files per poll and to process the file between
> instance in an
> intelligent and efficient manner.
>
> Any help suggestion or pointers is much appreciated .
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-increase-the-performance-for-SFTP-file-polling-consumer-tp5792312.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Zoran Regvart