You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by wiseotherwise <mu...@gmail.com> on 2013/11/12 17:09:30 UTC
Parallel processing of File Component on clustered environment
Hi,
I am looking for someone to help me out on this issue.
Our implementation is to read files placed in an INPUT folder and convert
those to respective XMLs on OUTPUT folder. We have 2 instances of TOMCAT
server running to listen to this INPUT Folder for load balancing and
parallel processing.
<from
uri="file://{{initial.input.dir.path}}?move={{archive.dir.path}}/inbox/initial&maxMessagesPerPoll=5&delay=10000&readLock=changed&readLockTimeout=5000&readLockLoggingLevel=INFO"
/>
The issue we are facing is, even though the polling size is 5 we have only 2
files that are getting picked and processed. For Example: File1.csv and
File2.csv are picked in the first polling. If File2.csv is processed in 2
min and lets say File1.csv is taking a long time(20 min). Once the File2.csv
is processed, CAMEL is not picking the File3.csv. Its waiting all the way
until File1.csv is processed and then picks only File3.csv (Where as it
should pick 2 files as there are 2 instances).
What we want to achieve is: We have 6 instances on higher environment and we
want all the instances to pick files and process them parallel so that it
wont pool one and the other wait to finish.
Appreciate your help.
--
View this message in context: http://camel.465427.n5.nabble.com/Parallel-processing-of-File-Component-on-clustered-environment-tp5743115.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Parallel processing of File Component on clustered environment
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You can use threads in your route to hand over the processing to a
thread pool, then the file consumer can pickup the next file.
I wrote an old blog about that
http://www.davsclaus.com/2009/05/on-road-to-camel-20-concurrency-with.html
Though back then the we named it async, but its threads.
On Tue, Nov 12, 2013 at 5:09 PM, wiseotherwise <mu...@gmail.com> wrote:
> Hi,
> I am looking for someone to help me out on this issue.
>
> Our implementation is to read files placed in an INPUT folder and convert
> those to respective XMLs on OUTPUT folder. We have 2 instances of TOMCAT
> server running to listen to this INPUT Folder for load balancing and
> parallel processing.
>
> <from
> uri="file://{{initial.input.dir.path}}?move={{archive.dir.path}}/inbox/initial&maxMessagesPerPoll=5&delay=10000&readLock=changed&readLockTimeout=5000&readLockLoggingLevel=INFO"
> />
>
> The issue we are facing is, even though the polling size is 5 we have only 2
> files that are getting picked and processed. For Example: File1.csv and
> File2.csv are picked in the first polling. If File2.csv is processed in 2
> min and lets say File1.csv is taking a long time(20 min). Once the File2.csv
> is processed, CAMEL is not picking the File3.csv. Its waiting all the way
> until File1.csv is processed and then picks only File3.csv (Where as it
> should pick 2 files as there are 2 instances).
>
> What we want to achieve is: We have 6 instances on higher environment and we
> want all the instances to pick files and process them parallel so that it
> wont pool one and the other wait to finish.
>
> Appreciate your help.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Parallel-processing-of-File-Component-on-clustered-environment-tp5743115.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen