Re: Input directory with multiple file consumers

[Claus Ibsen <cl...@gmail.com>]

On Thu, May 31, 2012 at 5:29 PM, Raf <ra...@persgroep.be> wrote:
> Hi Henryk,
>
> hekonsek wrote
>> You share the directory using distributed file system, don't you? :)
>
> During testing I was just running multiple instances on my local machine on
> a local directory, but in production you are right about the distributed
> file system :)
>
> Local filesystems are fixed in size so that's not an option. The files are
> read now from the same distributed file system without a problem.
>
> In enterprise applications it is common to have 2 nodes running the same
> application for load balancing and failover. And since camel does enterprise
> integration I just assumed the file component would have this functionality
> built in.
>

Yeah you hit an issue because you use readLock=changed, and then the
marker files is not created.

I have logged a ticket about this
https://issues.apache.org/jira/browse/CAMEL-5324



> Raf
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5713794.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

RE: Input directory with multiple file consumers

["ravi.4indra" <ra...@gmail.com>]

I ran into a similar issue and exploring options for this.
does reducing the delay between polls to very low value (<10) increases with
file consumption on both the nodes?



--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5762241.html
Sent from the Camel - Users mailing list archive at Nabble.com.

RE: Input directory with multiple file consumers

[Raf <ra...@persgroep.be>]

I am no camel expert, but this is my experience.

If you monitor a folder from multiple camel instances (for instance when you deploy on multiple servers) then you can only poll for 1 .
If that is not the case you can poll for multiple messages.
By maxMessagesPerPoll=10 to poll for 10 messages
Or maxMessagesPerPoll=0 to poll for all messages.

Regards,
Raf

From: Kondalarv [via Camel] [mailto:ml-node+s465427n5761967h2@n5.nabble.com]
Sent: woensdag 21 januari 2015 0:34
To: Raf Lenaerts
Subject: RE: Input directory with multiple file consumers

I tried below option and it is working

<camel:from uri="file:/app/test/data/inbox?readLock=changed&amp;delete=true&amp;preMove=processing&amp;maxMessagesPerPoll=1/>

But I would like to go for multiple messages for poll. do you have any solution for this?
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5761967.html
To unsubscribe from Input directory with multiple file consumers, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713788&code=cmFmLmxlbmFlcnRzQHBlcnNncm9lcC5iZXw1NzEzNzg4fDExOTY3MzYyNzU=>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5761998.html
Sent from the Camel - Users mailing list archive at Nabble.com.

RE: Input directory with multiple file consumers

[Kondalarv <ko...@gmail.com>]

I tried below option and it is working 

<camel:from
uri="file:/app/test/data/inbox?readLock=changed&amp;delete=true&amp;preMove=processing&amp;maxMessagesPerPoll=1/>

But I would like to go for multiple messages for poll. do you have any
solution for this?



--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5761967.html
Sent from the Camel - Users mailing list archive at Nabble.com.

RE: Input directory with multiple file consumers

[Raf <ra...@persgroep.be>]

> Is there a way to monitor the same input folder from camel
applications on different machines? I am wondering what the best
practice is on handling this situation. 

 

Since you logged an issue for this, you might want to know that I found
a workaround.

 

If you add the parameters preMove=processing&maxMessagesPerPoll=1 then
multiple servers can work on the same camel folder.

 

Without maxMessagesPerPoll=1, camel will scan all files in the folder
and then process them one by one. This will give errors when another
server has already processed the file while this server still thinks it
needs to be processed. This is no longer possible when
maxMessagesPerPoll=1 is used.

 

Without preMove=processing, the different servers will want to lock the
same file, only one gets the lock and the others are waiting to lock the
same file. With preMove=processing, each server will pick up a different
file because they no longer see the files locked by other servers.

 

 

Regards,

Raf





--
View this message in context: http://camel.465427.n5.nabble.com/Input-directory-with-multiple-file-consumers-tp5713788p5716340.html
Sent from the Camel - Users mailing list archive at Nabble.com.