You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Pavel <pa...@gmail.com> on 2010/07/23 09:03:17 UTC
File component/language: preMove only
Hi,
Is it possible to pre-move files, but not move them once processing
complete? I use "preMove" to ensure camel from 2 different cluster nodes
don't process the same file twice - file is moved into different directory
and not scanned anymore.
"move" is not necessary, as custom components performs logic on file
archival, stores metadata in the database, etc.
Seems like setting "move" to null should do what I need, but I'm not quite
sure how to do that. The "&move=null" will create directory called "null",
not specifying anything will default to ".camel", and adding "&noop=true"
disables preMove.
Is there something in file language that allows setting nulls? Or is there a
way to have preMove without move for file component?
--
Best regards,
Pavel
Re: File component/language: preMove only
Posted by vss123 <vs...@gmail.com>.
> And the other Camel is picking up that file. Eg it get processed?
> So the failed to rename is a race condition because the other Camel
> grabbed the same file, but was faster to do the rename?
We believe this is what is happening. Even with readLock option two
instances pick the same file and one moves it before the other and hence the
other throws 'cannot rename' exception.
Any way to avoid this race condition?
--
View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5730477.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Mar 6, 2013 at 6:40 AM, vss123 <vs...@gmail.com> wrote:
> Sorry. The Camel version we use is 2.10.3.
>
> We have a drop-zone in a file system where the other systems drop xml files.
>
> With this project we use camel to pickup these files and do processing to
> communicate with few other systems. We want to run multiple instances of our
> project, so that we can process many xml files in parallel.
>
> We used 'preMove' as a mechanism so that two instances don't start
> processing the same file. So the files move from 'files/inbox' to
> 'files/inbox/inProgress' and then after processing it moves to
> 'files/inProgress/.camel'
>
You can just remove preMove and let the default readLock do its business.
It writes a .camelLock file when a consumer is picking up a file.
> When we run multiple instances we get following kinds of errors with one or
> more of them.
>
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot rename file: GenericFile[sample1.xml] to:
> GenericFile[inProcess/sample1.xml]]
>
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot rename file: GenericFile[inProcess/sample2.xml] to:
> GenericFile[inProcess/.camel/sample2.xml]]
>
> Because of this, the processed files go missing in 'inProcess' folder or
> 'inProcess/.camel'.
>
Are you using Windows? If you are they make sure you close all streams
when you acces the file.
Otherwise you cannot move the file after use.
You can also let Camel read the file content into memory first, then
you do not have that problem
from file
convertBodyTo(String.class)
> Also, will using SFTP/FTP will help in this kind of scenarios? If so how to
> do the same with FTP/SFTP?
>
> Thanks for your response and happy to provide more details. :)
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728629.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: File component/language: preMove only
Posted by vss123 <vs...@gmail.com>.
Sorry. The Camel version we use is 2.10.3.
We have a drop-zone in a file system where the other systems drop xml files.
With this project we use camel to pickup these files and do processing to
communicate with few other systems. We want to run multiple instances of our
project, so that we can process many xml files in parallel.
We used 'preMove' as a mechanism so that two instances don't start
processing the same file. So the files move from 'files/inbox' to
'files/inbox/inProgress' and then after processing it moves to
'files/inProgress/.camel'
When we run multiple instances we get following kinds of errors with one or
more of them.
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file: GenericFile[sample1.xml] to:
GenericFile[inProcess/sample1.xml]]
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file: GenericFile[inProcess/sample2.xml] to:
GenericFile[inProcess/.camel/sample2.xml]]
Because of this, the processed files go missing in 'inProcess' folder or
'inProcess/.camel'.
Also, will using SFTP/FTP will help in this kind of scenarios? If so how to
do the same with FTP/SFTP?
Thanks for your response and happy to provide more details. :)
--
View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728629.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Mar 5, 2013 at 11:16 AM, vss123 <vs...@gmail.com> wrote:
> Hi,
>
> We also tried using preMove to avoid multiple instances of Camel picking the
> same file. But that does not seem to work. When we run with more than one
> instance, We get following error.
>
> cannot begin processing file: GenericFile[sample1.xml] due to: Cannot rename
> file: GenericFile[sample1.xml] to: GenericFile[inProcess/sample1.xml].
> Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Cannot rename file: GenericFile[sample1.xml] to:
> GenericFile[inProcess/sample1.xml]]
>
> The route we have is.
>
> from("file://files/inbox/?preMove=inprogress").log("processing
> ${file:name}").to("file://files/outbox?fileName=${file:name}+${exchangeId}")
>
>
> Are we missing anything obvious in using Camel this way?
>
And the other Camel is picking up that file. Eg it get processed?
So the failed to rename is a race condition because the other Camel
grabbed the same file, but was faster to do the rename?
And as usual people should mention which Camel version you use. As we
have many releases, and its important to know what version you use.
> Thanks.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728565.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: File component/language: preMove only
Posted by vss123 <vs...@gmail.com>.
Hi,
We also tried using preMove to avoid multiple instances of Camel picking the
same file. But that does not seem to work. When we run with more than one
instance, We get following error.
cannot begin processing file: GenericFile[sample1.xml] due to: Cannot rename
file: GenericFile[sample1.xml] to: GenericFile[inProcess/sample1.xml].
Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file: GenericFile[sample1.xml] to:
GenericFile[inProcess/sample1.xml]]
The route we have is.
from("file://files/inbox/?preMove=inprogress").log("processing
${file:name}").to("file://files/outbox?fileName=${file:name}+${exchangeId}")
Are we missing anything obvious in using Camel this way?
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/File-component-language-preMove-only-tp1843111p5728565.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah the file component is starting to be overloaded with
move/delete/noop options :)
Maybe the noop should have been an action option, so you could specify
what you want to do
action=preMove
action=preMove,move
action=preMove,delete
action=noop
action=delete
action=move (* default)
Let's thinker for a minute what a good option name would be for your use case.
Using preMove is kinda nice in a clustered as a IO move is atomic.
Then you don't need to mess with lock files etc.
On Fri, Jul 23, 2010 at 12:05 PM, Pavel <pa...@gmail.com> wrote:
> That doesn't work, since by the time route completes, file is moved
> elsewhere.
>
> Do you think preMove-only could be useful extension? I know it is useful for
> me, but not sure how typical such requirement is.
>
> If so, I could e.g. add ${null} simple expressions, or "noMove=true" to file
> component.
>
> Thanks,
> Pavel
>
> On Fri, Jul 23, 2010 at 10:09 AM, Claus Ibsen <cl...@gmail.com> wrote:
>
>> Hi
>>
>> Have you tried with move=. to indicate current folder?
>>
>> But no its, not a feature of camel to only pre move and then noop.
>>
>>
>>
>> On Fri, Jul 23, 2010 at 9:03 AM, Pavel <pa...@gmail.com> wrote:
>> > Hi,
>> >
>> > Is it possible to pre-move files, but not move them once processing
>> > complete? I use "preMove" to ensure camel from 2 different cluster nodes
>> > don't process the same file twice - file is moved into different
>> directory
>> > and not scanned anymore.
>> > "move" is not necessary, as custom components performs logic on file
>> > archival, stores metadata in the database, etc.
>> > Seems like setting "move" to null should do what I need, but I'm not
>> quite
>> > sure how to do that. The "&move=null" will create directory called
>> "null",
>> > not specifying anything will default to ".camel", and adding "&noop=true"
>> > disables preMove.
>> >
>> > Is there something in file language that allows setting nulls? Or is
>> there a
>> > way to have preMove without move for file component?
>> >
>> >
>> > --
>> > Best regards,
>> > Pavel
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component/language: preMove only
Posted by Pavel <pa...@gmail.com>.
Cool - thanks a lot. Yes, seems like preMove is a bit special indeed - it
addresses technical issue of concurrency as opposed to functional
requirements of file placement after processing...
Pavel
On Wed, Jul 28, 2010 at 12:31 PM, Claus Ibsen <cl...@gmail.com> wrote:
> I created a ticket to track this
> https://issues.apache.org/activemq/browse/CAMEL-3003
>
> And you can now use preMove together with noop|delete.
>
>
> On Tue, Jul 27, 2010 at 2:50 PM, Claus Ibsen <cl...@gmail.com>
> wrote:
> > Hi
> >
> > What if preMove is a bit special. So you can do
> > preMove=moveMe&noop=true. Which then tells Camel to pre move the file.
> > But on commit it should just leave the file as is.
> >
> >
> >
> > On Fri, Jul 23, 2010 at 12:05 PM, Pavel <pa...@gmail.com> wrote:
> >> That doesn't work, since by the time route completes, file is moved
> >> elsewhere.
> >>
> >> Do you think preMove-only could be useful extension? I know it is useful
> for
> >> me, but not sure how typical such requirement is.
> >>
> >> If so, I could e.g. add ${null} simple expressions, or "noMove=true" to
> file
> >> component.
> >>
> >> Thanks,
> >> Pavel
> >>
> >> On Fri, Jul 23, 2010 at 10:09 AM, Claus Ibsen <cl...@gmail.com>
> wrote:
> >>
> >>> Hi
> >>>
> >>> Have you tried with move=. to indicate current folder?
> >>>
> >>> But no its, not a feature of camel to only pre move and then noop.
> >>>
> >>>
> >>>
> >>> On Fri, Jul 23, 2010 at 9:03 AM, Pavel <pa...@gmail.com> wrote:
> >>> > Hi,
> >>> >
> >>> > Is it possible to pre-move files, but not move them once processing
> >>> > complete? I use "preMove" to ensure camel from 2 different cluster
> nodes
> >>> > don't process the same file twice - file is moved into different
> >>> directory
> >>> > and not scanned anymore.
> >>> > "move" is not necessary, as custom components performs logic on file
> >>> > archival, stores metadata in the database, etc.
> >>> > Seems like setting "move" to null should do what I need, but I'm not
> >>> quite
> >>> > sure how to do that. The "&move=null" will create directory called
> >>> "null",
> >>> > not specifying anything will default to ".camel", and adding
> "&noop=true"
> >>> > disables preMove.
> >>> >
> >>> > Is there something in file language that allows setting nulls? Or is
> >>> there a
> >>> > way to have preMove without move for file component?
> >>> >
> >>> >
> >>> > --
> >>> > Best regards,
> >>> > Pavel
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Claus Ibsen
> >>> Apache Camel Committer
> >>>
> >>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>> Open Source Integration: http://fusesource.com
> >>> Blog: http://davsclaus.blogspot.com/
> >>> Twitter: http://twitter.com/davsclaus
> >>>
> >>
> >
> >
> >
> > --
> > Claus Ibsen
> > Apache Camel Committer
> >
> > Author of Camel in Action: http://www.manning.com/ibsen/
> > Open Source Integration: http://fusesource.com
> > Blog: http://davsclaus.blogspot.com/
> > Twitter: http://twitter.com/davsclaus
> >
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
I created a ticket to track this
https://issues.apache.org/activemq/browse/CAMEL-3003
And you can now use preMove together with noop|delete.
On Tue, Jul 27, 2010 at 2:50 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> What if preMove is a bit special. So you can do
> preMove=moveMe&noop=true. Which then tells Camel to pre move the file.
> But on commit it should just leave the file as is.
>
>
>
> On Fri, Jul 23, 2010 at 12:05 PM, Pavel <pa...@gmail.com> wrote:
>> That doesn't work, since by the time route completes, file is moved
>> elsewhere.
>>
>> Do you think preMove-only could be useful extension? I know it is useful for
>> me, but not sure how typical such requirement is.
>>
>> If so, I could e.g. add ${null} simple expressions, or "noMove=true" to file
>> component.
>>
>> Thanks,
>> Pavel
>>
>> On Fri, Jul 23, 2010 at 10:09 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> Have you tried with move=. to indicate current folder?
>>>
>>> But no its, not a feature of camel to only pre move and then noop.
>>>
>>>
>>>
>>> On Fri, Jul 23, 2010 at 9:03 AM, Pavel <pa...@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > Is it possible to pre-move files, but not move them once processing
>>> > complete? I use "preMove" to ensure camel from 2 different cluster nodes
>>> > don't process the same file twice - file is moved into different
>>> directory
>>> > and not scanned anymore.
>>> > "move" is not necessary, as custom components performs logic on file
>>> > archival, stores metadata in the database, etc.
>>> > Seems like setting "move" to null should do what I need, but I'm not
>>> quite
>>> > sure how to do that. The "&move=null" will create directory called
>>> "null",
>>> > not specifying anything will default to ".camel", and adding "&noop=true"
>>> > disables preMove.
>>> >
>>> > Is there something in file language that allows setting nulls? Or is
>>> there a
>>> > way to have preMove without move for file component?
>>> >
>>> >
>>> > --
>>> > Best regards,
>>> > Pavel
>>> >
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
What if preMove is a bit special. So you can do
preMove=moveMe&noop=true. Which then tells Camel to pre move the file.
But on commit it should just leave the file as is.
On Fri, Jul 23, 2010 at 12:05 PM, Pavel <pa...@gmail.com> wrote:
> That doesn't work, since by the time route completes, file is moved
> elsewhere.
>
> Do you think preMove-only could be useful extension? I know it is useful for
> me, but not sure how typical such requirement is.
>
> If so, I could e.g. add ${null} simple expressions, or "noMove=true" to file
> component.
>
> Thanks,
> Pavel
>
> On Fri, Jul 23, 2010 at 10:09 AM, Claus Ibsen <cl...@gmail.com> wrote:
>
>> Hi
>>
>> Have you tried with move=. to indicate current folder?
>>
>> But no its, not a feature of camel to only pre move and then noop.
>>
>>
>>
>> On Fri, Jul 23, 2010 at 9:03 AM, Pavel <pa...@gmail.com> wrote:
>> > Hi,
>> >
>> > Is it possible to pre-move files, but not move them once processing
>> > complete? I use "preMove" to ensure camel from 2 different cluster nodes
>> > don't process the same file twice - file is moved into different
>> directory
>> > and not scanned anymore.
>> > "move" is not necessary, as custom components performs logic on file
>> > archival, stores metadata in the database, etc.
>> > Seems like setting "move" to null should do what I need, but I'm not
>> quite
>> > sure how to do that. The "&move=null" will create directory called
>> "null",
>> > not specifying anything will default to ".camel", and adding "&noop=true"
>> > disables preMove.
>> >
>> > Is there something in file language that allows setting nulls? Or is
>> there a
>> > way to have preMove without move for file component?
>> >
>> >
>> > --
>> > Best regards,
>> > Pavel
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File component/language: preMove only
Posted by Pavel <pa...@gmail.com>.
That doesn't work, since by the time route completes, file is moved
elsewhere.
Do you think preMove-only could be useful extension? I know it is useful for
me, but not sure how typical such requirement is.
If so, I could e.g. add ${null} simple expressions, or "noMove=true" to file
component.
Thanks,
Pavel
On Fri, Jul 23, 2010 at 10:09 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Have you tried with move=. to indicate current folder?
>
> But no its, not a feature of camel to only pre move and then noop.
>
>
>
> On Fri, Jul 23, 2010 at 9:03 AM, Pavel <pa...@gmail.com> wrote:
> > Hi,
> >
> > Is it possible to pre-move files, but not move them once processing
> > complete? I use "preMove" to ensure camel from 2 different cluster nodes
> > don't process the same file twice - file is moved into different
> directory
> > and not scanned anymore.
> > "move" is not necessary, as custom components performs logic on file
> > archival, stores metadata in the database, etc.
> > Seems like setting "move" to null should do what I need, but I'm not
> quite
> > sure how to do that. The "&move=null" will create directory called
> "null",
> > not specifying anything will default to ".camel", and adding "&noop=true"
> > disables preMove.
> >
> > Is there something in file language that allows setting nulls? Or is
> there a
> > way to have preMove without move for file component?
> >
> >
> > --
> > Best regards,
> > Pavel
> >
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
Re: File component/language: preMove only
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Have you tried with move=. to indicate current folder?
But no its, not a feature of camel to only pre move and then noop.
On Fri, Jul 23, 2010 at 9:03 AM, Pavel <pa...@gmail.com> wrote:
> Hi,
>
> Is it possible to pre-move files, but not move them once processing
> complete? I use "preMove" to ensure camel from 2 different cluster nodes
> don't process the same file twice - file is moved into different directory
> and not scanned anymore.
> "move" is not necessary, as custom components performs logic on file
> archival, stores metadata in the database, etc.
> Seems like setting "move" to null should do what I need, but I'm not quite
> sure how to do that. The "&move=null" will create directory called "null",
> not specifying anything will default to ".camel", and adding "&noop=true"
> disables preMove.
>
> Is there something in file language that allows setting nulls? Or is there a
> way to have preMove without move for file component?
>
>
> --
> Best regards,
> Pavel
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus