You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by vss123 <vs...@gmail.com> on 2013/05/31 15:12:53 UTC
File 'markerFile' strategy race condition
Hi,We use camel 2.11 version on Linux.We run two instances of camel which
watch on the a directory to pick, process and move files to processed
directory on successful completion. On error it gets moved to error
directory. We use 'markerFile' as a strategy for read lock.What we observe
is that sometimes a file gets picked by both the instances, resulting in
error.
File named /A.txt/ gets picked by /Camel 1/ instance and lock file
/A.txt.camelLock/ gets created.
/ Camel 1/ instance reads and finishes processing the file /A.txt/
/ Camel 1/ instance deletes /A.txt.camelLock/
/ Camel 2/ instance picks /A.txt/ as there is not lock to it now and starts
processing. It creates /A.txt.camelLock/
/ Camel 1/ instance moves /A.txt/ to processed dir.
/ Camel 2/ instance deletes /A.txt.camelLock/ as it finished processing
/ Camel 2/ faces error as it tries to move /A.txt/ to processed dir as
/A.txt/ is not available anymore
/ Camel 2/ faces further error as it tries to move /A.txt/ to error dir
because of previous error and obviously it cannot find /A.txt/
The problem is /.camelLock/ file gets deleted before the actual files gets
moved to processed dir. And if we flip the order, this race condition should
not happen.Can you help us to understand the reasoning behind this design
and how to avoid this error?
--
View this message in context: http://camel.465427.n5.nabble.com/File-markerFile-strategy-race-condition-tp5733561.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File 'markerFile' strategy race condition
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Well spotted. I have logged a ticket to get this fixed
https://issues.apache.org/jira/browse/CAMEL-6413
Though I guess you are really unlucky to see this as there is few CPU
cycles between moving the files, and deleting the .camelLock file. So
it ought to be hard to reproduce or see. You can possible have
different poll delays in the 2 nodes. So they poll for files at
different times, and has less chance for them to have a race
condition.
You can also build you own custom read lock strategy etc.
On Fri, May 31, 2013 at 3:12 PM, vss123 <vs...@gmail.com> wrote:
> Hi,We use camel 2.11 version on Linux.We run two instances of camel which
> watch on the a directory to pick, process and move files to processed
> directory on successful completion. On error it gets moved to error
> directory. We use 'markerFile' as a strategy for read lock.What we observe
> is that sometimes a file gets picked by both the instances, resulting in
> error.
> File named /A.txt/ gets picked by /Camel 1/ instance and lock file
> /A.txt.camelLock/ gets created.
> / Camel 1/ instance reads and finishes processing the file /A.txt/
> / Camel 1/ instance deletes /A.txt.camelLock/
> / Camel 2/ instance picks /A.txt/ as there is not lock to it now and starts
> processing. It creates /A.txt.camelLock/
> / Camel 1/ instance moves /A.txt/ to processed dir.
> / Camel 2/ instance deletes /A.txt.camelLock/ as it finished processing
> / Camel 2/ faces error as it tries to move /A.txt/ to processed dir as
> /A.txt/ is not available anymore
> / Camel 2/ faces further error as it tries to move /A.txt/ to error dir
> because of previous error and obviously it cannot find /A.txt/
> The problem is /.camelLock/ file gets deleted before the actual files gets
> moved to processed dir. And if we flip the order, this race condition should
> not happen.Can you help us to understand the reasoning behind this design
> and how to avoid this error?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-markerFile-strategy-race-condition-tp5733561.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen