You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by vss123 <vs...@gmail.com> on 2013/03/05 11:16:38 UTC
Re: File component/language: preMove only
Hi,
We also tried using preMove to avoid multiple instances of Camel picking the
same file. But that does not seem to work. When we run with more than one
instance, We get following error.
cannot begin processing file: GenericFile[sample1.xml] due to: Cannot rename
file: GenericFile[sample1.xml] to: GenericFile[inProcess/sample1.xml].
Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file: GenericFile[sample1.xml] to:
GenericFile[inProcess/sample1.xml]]
The route we have is.
from("file://files/inbox/?preMove=inprogress").log("processing
${file:name}").to("file://files/outbox?fileName=${file:name}+${exchangeId}")
Are we missing anything obvious in using Camel this way?
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728565.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component/language: preMove only
Posted by vss123 <vs...@gmail.com>.
> And the other Camel is picking up that file. Eg it get processed?
> So the failed to rename is a race condition because the other Camel
> grabbed the same file, but was faster to do the rename?
We believe this is what is happening. Even with readLock option two
instances pick the same file and one moves it before the other and hence the
other throws 'cannot rename' exception.
Any way to avoid this race condition?
--
View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5730477.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Mar 6, 2013 at 6:40 AM, vss123 <vs...@gmail.com> wrote:
> Sorry. The Camel version we use is 2.10.3.
>
> We have a drop-zone in a file system where the other systems drop xml files.
>
> With this project we use camel to pickup these files and do processing to
> communicate with few other systems. We want to run multiple instances of our
> project, so that we can process many xml files in parallel.
>
> We used 'preMove' as a mechanism so that two instances don't start
> processing the same file. So the files move from 'files/inbox' to
> 'files/inbox/inProgress' and then after processing it moves to
> 'files/inProgress/.camel'
>
You can just remove preMove and let the default readLock do its business.
It writes a .camelLock file when a consumer is picking up a file.
> When we run multiple instances we get following kinds of errors with one or
> more of them.
>
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot rename file: GenericFile[sample1.xml] to:
> GenericFile[inProcess/sample1.xml]]
>
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot rename file: GenericFile[inProcess/sample2.xml] to:
> GenericFile[inProcess/.camel/sample2.xml]]
>
> Because of this, the processed files go missing in 'inProcess' folder or
> 'inProcess/.camel'.
>
Are you using Windows? If you are they make sure you close all streams
when you acces the file.
Otherwise you cannot move the file after use.
You can also let Camel read the file content into memory first, then
you do not have that problem
from file
convertBodyTo(String.class)
> Also, will using SFTP/FTP will help in this kind of scenarios? If so how to
> do the same with FTP/SFTP?
>
> Thanks for your response and happy to provide more details. :)
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728629.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: File component/language: preMove only
Posted by vss123 <vs...@gmail.com>.
Sorry. The Camel version we use is 2.10.3.
We have a drop-zone in a file system where the other systems drop xml files.
With this project we use camel to pickup these files and do processing to
communicate with few other systems. We want to run multiple instances of our
project, so that we can process many xml files in parallel.
We used 'preMove' as a mechanism so that two instances don't start
processing the same file. So the files move from 'files/inbox' to
'files/inbox/inProgress' and then after processing it moves to
'files/inProgress/.camel'
When we run multiple instances we get following kinds of errors with one or
more of them.
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file: GenericFile[sample1.xml] to:
GenericFile[inProcess/sample1.xml]]
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file: GenericFile[inProcess/sample2.xml] to:
GenericFile[inProcess/.camel/sample2.xml]]
Because of this, the processed files go missing in 'inProcess' folder or
'inProcess/.camel'.
Also, will using SFTP/FTP will help in this kind of scenarios? If so how to
do the same with FTP/SFTP?
Thanks for your response and happy to provide more details. :)
--
View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728629.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Mar 5, 2013 at 11:16 AM, vss123 <vs...@gmail.com> wrote:
> Hi,
>
> We also tried using preMove to avoid multiple instances of Camel picking the
> same file. But that does not seem to work. When we run with more than one
> instance, We get following error.
>
> cannot begin processing file: GenericFile[sample1.xml] due to: Cannot rename
> file: GenericFile[sample1.xml] to: GenericFile[inProcess/sample1.xml].
> Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot rename file: GenericFile[sample1.xml] to:
> GenericFile[inProcess/sample1.xml]]
>
> The route we have is.
>
> from("file://files/inbox/?preMove=inprogress").log("processing
> ${file:name}").to("file://files/outbox?fileName=${file:name}+${exchangeId}")
>
>
> Are we missing anything obvious in using Camel this way?
>
And the other Camel is picking up that file. Eg it get processed?
So the failed to rename is a race condition because the other Camel
grabbed the same file, but was faster to do the rename?
And as usual people should mention which Camel version you use. As we
have many releases, and its important to know what version you use.
> Thanks.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728565.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen