You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Raf <ra...@persgroep.be> on 2012/05/31 16:04:06 UTC
Input directory with multiple file consumers
Hello,
Is there a way to monitor the same input folder from camel applications on
different machines? I am wondering what the best practice is on handling
this situation.
I have a working camel (2.7.1) application that processes zip files placed
in an input folder. The application runs on a single Unix machine.
<from
uri="file://${file.input}/?doneFileName=${file:name}.done&readLock=changed&delete=true"/>
Since the workload is gradually increasing and the processing is quite cpu
and memory intensive, it doesn't make sense to loadbalance within the camel
context. Rather I want to deploy the application on multiple machines.
But testing this setup I see that the different machines start processing
the same files and ignoring the .camelLock from the other application. Or
errors that files are missing (meaning already processed by the other
machine). Is there a combination of options to prevent this or do I need to
put some sort of broker between the file consumer and the processing?
Kind regards,
Raf
--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Input directory with multiple file consumers
Posted by Henryk Konsek <he...@gmail.com>.
Hi Raf,
> In enterprise applications it is common to have 2 nodes running the same
> application for load balancing and failover. And since camel does enterprise
> integration I just assumed the file component would have this functionality
> built in.
It certainly does. :) If I find some time tomorrow, I'll try to
reproduce your problem. Reading lock should not behave in that way.
Regarding load balancing - I only suggesting that load balancing
reading from the local file system, as long as you only fan-out the
messages from the file system to JMS, is often unnecessary. That
happens because reading from local file system is extremely fast and
because file systems are not usually used to handle so much payloads,
that they can hit the bottleneck of dedicated Camel fan-out router.
Regarding the failover - you can for example make your ServiceMix
instance to be backed by failover instance. So if one ServiceMix is
dead, then the second one will handle the file system processing and
continue to fan-out messages to the JMS.
I'm not telling that load balancing file system is bad or unsupported
by Camel. I'm just sharing my point of view regarding clustering of
this endpoint.
--
Henryk Konsek
http://henryk-konsek.blogspot.com
RE: Input directory with multiple file consumers
Posted by "ravi.4indra" <ra...@gmail.com>.
I ran into a similar issue and exploring options for this.
does reducing the delay between polls to very low value (<10) increases with
file consumption on both the nodes?
--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5762241.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Input directory with multiple file consumers
Posted by Raf <ra...@persgroep.be>.
I am no camel expert, but this is my experience.
If you monitor a folder from multiple camel instances (for instance when you deploy on multiple servers) then you can only poll for 1 .
If that is not the case you can poll for multiple messages.
By maxMessagesPerPoll=10 to poll for 10 messages
Or maxMessagesPerPoll=0 to poll for all messages.
Regards,
Raf
From: Kondalarv [via Camel] [mailto:ml-node+s465427n5761967h2@n5.nabble.com]
Sent: woensdag 21 januari 2015 0:34
To: Raf Lenaerts
Subject: RE: Input directory with multiple file consumers
I tried below option and it is working
<camel:from uri="file:/app/test/data/inbox?readLock=changed&delete=true&preMove=processing&maxMessagesPerPoll=1/>
But I would like to go for multiple messages for poll. do you have any solution for this?
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5761967.html
To unsubscribe from Input directory with multiple file consumers, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713788&code=cmFmLmxlbmFlcnRzQHBlcnNncm9lcC5iZXw1NzEzNzg4fDExOTY3MzYyNzU=>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5761998.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Input directory with multiple file consumers
Posted by Kondalarv <ko...@gmail.com>.
I tried below option and it is working
<camel:from
uri="file:/app/test/data/inbox?readLock=changed&delete=true&preMove=processing&maxMessagesPerPoll=1/>
But I would like to go for multiple messages for poll. do you have any
solution for this?
--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5761967.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Input directory with multiple file consumers
Posted by Raf <ra...@persgroep.be>.
> Is there a way to monitor the same input folder from camel
applications on different machines? I am wondering what the best
practice is on handling this situation.
Since you logged an issue for this, you might want to know that I found
a workaround.
If you add the parameters preMove=processing&maxMessagesPerPoll=1 then
multiple servers can work on the same camel folder.
Without maxMessagesPerPoll=1, camel will scan all files in the folder
and then process them one by one. This will give errors when another
server has already processed the file while this server still thinks it
needs to be processed. This is no longer possible when
maxMessagesPerPoll=1 is used.
Without preMove=processing, the different servers will want to lock the
same file, only one gets the lock and the others are waiting to lock the
same file. With preMove=processing, each server will pick up a different
file because they no longer see the files locked by other servers.
Regards,
Raf
--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5716340.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Input directory with multiple file consumers
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, May 31, 2012 at 5:29 PM, Raf <ra...@persgroep.be> wrote:
> Hi Henryk,
>
> hekonsek wrote
>> You share the directory using distributed file system, don't you? :)
>
> During testing I was just running multiple instances on my local machine on
> a local directory, but in production you are right about the distributed
> file system :)
>
> Local filesystems are fixed in size so that's not an option. The files are
> read now from the same distributed file system without a problem.
>
> In enterprise applications it is common to have 2 nodes running the same
> application for load balancing and failover. And since camel does enterprise
> integration I just assumed the file component would have this functionality
> built in.
>
Yeah you hit an issue because you use readLock=changed, and then the
marker files is not created.
I have logged a ticket about this
https://issues.apache.org/jira/browse/CAMEL-5324
> Raf
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5713794.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Input directory with multiple file consumers
Posted by Raf <ra...@persgroep.be>.
Hi Henryk,
hekonsek wrote
> You share the directory using distributed file system, don't you? :)
During testing I was just running multiple instances on my local machine on
a local directory, but in production you are right about the distributed
file system :)
Local filesystems are fixed in size so that's not an option. The files are
read now from the same distributed file system without a problem.
In enterprise applications it is common to have 2 nodes running the same
application for load balancing and failover. And since camel does enterprise
integration I just assumed the file component would have this functionality
built in.
Raf
--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5713794.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Input directory with multiple file consumers
Posted by Henryk Konsek <he...@gmail.com>.
Hi Raf,
> it doesn't make sense to loadbalance within the camel
> context. Rather I want to deploy the application on multiple machines.
You share the directory using distributed file system, don't you? :)
Reading from the shared file system may generate problems. It is
highly dependent on the particular implementation of the file system.
I personally prefer to stick to the local file consumer in such case
and then forward messages from the file system to the JMS. It is
usually hard to hit the bottleneck when reading messages from the
local file system, if you just fan-out them to the JMS queue.
I prefer to create the solutions I can embrace. One local file system
plus JMS queue are easy to embrace. All possible corner cases of
distributed file systems are not :) .
--
Henryk Konsek
http://henryk-konsek.blogspot.com