You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by swwyatt <st...@sungard.com> on 2013/05/09 20:05:04 UTC
GenericFileConsumer - Idempotent and InProgress repositories
The implementations of the GenericFileConsumer (FileConsumer,
SftpConsumer,...) method, pollDirectory(), make calls to isValidFile() and
if that returns true, calls isInProgress(). This process fails when there
are multiple processes polling. For example when process P0 calls
isValidFile() and the idempotent repository does not have the file entry, it
then calls isInProgress(). But, if process P1, who is in progress for the
file, commits to the idempotent repository and subsequently releases its
entry in the in progress repository, then P0 may obtain an in progress
status and process a file that is now in the idempotent repository.
Should the isInProgress() call be made before the isValidFile() call? This
may have additional overhead of checking the in progress repository, but the
chance for duplicate processing would be removed.
--
View this message in context: http://camel.465427.n5.nabble.com/GenericFileConsumer-Idempotent-and-InProgress-repositories-tp5732237.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: GenericFileConsumer - Idempotent and InProgress repositories
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah we can check the in progress repo a bit more eager in case we
have a file candidate
https://issues.apache.org/jira/browse/CAMEL-6355
Though we should do the isValidFile first as it skips unwanted file
patterns etc.
On Thu, May 9, 2013 at 8:05 PM, swwyatt <st...@sungard.com> wrote:
> The implementations of the GenericFileConsumer (FileConsumer,
> SftpConsumer,...) method, pollDirectory(), make calls to isValidFile() and
> if that returns true, calls isInProgress(). This process fails when there
> are multiple processes polling. For example when process P0 calls
> isValidFile() and the idempotent repository does not have the file entry, it
> then calls isInProgress(). But, if process P1, who is in progress for the
> file, commits to the idempotent repository and subsequently releases its
> entry in the in progress repository, then P0 may obtain an in progress
> status and process a file that is now in the idempotent repository.
>
> Should the isInProgress() call be made before the isValidFile() call? This
> may have additional overhead of checking the in progress repository, but the
> chance for duplicate processing would be removed.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/GenericFileConsumer-Idempotent-and-InProgress-repositories-tp5732237.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen