You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by sagy <sa...@gmail.com> on 2010/04/13 14:33:57 UTC
File consumer with noop=true&recursive=true
Hi,
I'm using Camel 2.2 and have the following route:
from("file://test?recursive=true&noop=true").process(...
When I drop a file f1.txt into the test folder the processor gets called.
When I drop the file f2.txt into a sub folder of test folder for example
test/sub/f2.txt ,again the processor gets called.
But when I drop the file f1.txt into test/sub the processor doesn't get
called.
This is happening because of line 115 in camel-core
org.apache.camel.component.file.GenericFileOnCompletion:
// only add to idempotent repository if we could process the
file
// only use the filename as the key as the file could be moved
into a done folder
endpoint.getIdempotentRepository().add(file.getFileName());
The repository stores just the file name without the path.
Therefore if 2 files with the same name are dropped to 2 different sub
directories only the first one will be processed.
Is this the intended behavior?
Is there a way to work around it?
--
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer with noop=true&recursive=true
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Apr 15, 2010 at 6:55 PM, sagy <sa...@gmail.com> wrote:
>
> I've tested it and it seems to be working fine now.
> Thanks so much for the quick fix
>
Thanks for taking your time to test and report back.
> Sagy
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Tue, Apr 13, 2010 at 6:35 PM, sagy <sa...@gmail.com> wrote:
>>>
>>> Thanks Claus!
>>>
>>
>> You are welcome to test 2.3-SNAPSHOT as I have committed a fix just now.
>> http://camel.apache.org/download.html
>>
>> Mind that it takes a bit time for Apache to deploy a new SNAPSHOT on
>> their servers.
>>
>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Tue, Apr 13, 2010 at 5:49 PM, sagy <sa...@gmail.com> wrote:
>>>>>
>>>>> exactly. Lets assume a directory root called /test
>>>>> the writing application will write the following files:
>>>>> /test/file1.txt
>>>>> /test/file2.txt
>>>>> /test/sub-folder/file1.txt
>>>>>
>>>>> etc..
>>>>> My application will get an event for
>>>>> /test/file1.txt
>>>>> /test/file2.txt
>>>>>
>>>>> but not for
>>>>> /test/sub-folder/file1.txt
>>>>>
>>>>> since the IdempotentRepository already contains an entry called
>>>>> "file1.txt".
>>>>> It was created for the /test/file1.txt event.
>>>>>
>>>>
>>>> Noted. I have created a ticket to allow you define this with an option
>>>> on the file component
>>>> https://issues.apache.org/activemq/browse/CAMEL-2640
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>
>>>>>>> I'm trying to consume files from it not skip it.
>>>>>>> This is a general description of the situation:
>>>>>>>
>>>>>>> There is another application with write permissions which create
>>>>>>> files
>>>>>>> in
>>>>>>> that directory and sub directories.
>>>>>>> My application only has read permissions but I still need to consume
>>>>>>> the
>>>>>>> files from that directory structure.
>>>>>>>
>>>>>>> That's why I'm using the "noop=true" flag. I can't move the files or
>>>>>>> delete
>>>>>>> them.
>>>>>>> It seems from the documentation that noop is my only choice.
>>>>>>> But a combination of "recursive=true" and "noop=true" causes the
>>>>>>> problem
>>>>>>> I've described in the original post.
>>>>>>>
>>>>>>
>>>>>> And that app can write files with the same name?
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> I'm actually trying to consume files from a directory recursively
>>>>>>>>> and
>>>>>>>>> the
>>>>>>>>> directory has read-only permissions.
>>>>>>>>> Is there another way to do it without using
>>>>>>>>> noop=true&recursive=true
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> You can filter out the read-only directory if you should "skip" it.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Sagy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Yeah its on purpose for some reason, which as I can't really
>>>>>>>>>> recall
>>>>>>>>>> right
>>>>>>>>>> now.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>>>>>>
>>>>>>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>>>>>>
>>>>>>>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>>>>>>>> called.
>>>>>>>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>>>>>>>> example
>>>>>>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>>>>>>> But when I drop the file f1.txt into test/sub the processor
>>>>>>>>>>> doesn't
>>>>>>>>>>> get
>>>>>>>>>>> called.
>>>>>>>>>>> This is happening because of line 115 in camel-core
>>>>>>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>>>>>>
>>>>>>>>>>> // only add to idempotent repository if we could
>>>>>>>>>>> process
>>>>>>>>>>> the
>>>>>>>>>>> file
>>>>>>>>>>> // only use the filename as the key as the file could
>>>>>>>>>>> be
>>>>>>>>>>> moved
>>>>>>>>>>> into a done folder
>>>>>>>>>>>
>>>>>>>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>>>>>>
>>>>>>>>>>> The repository stores just the file name without the path.
>>>>>>>>>>> Therefore if 2 files with the same name are dropped to 2
>>>>>>>>>>> different
>>>>>>>>>>> sub
>>>>>>>>>>> directories only the first one will be processed.
>>>>>>>>>>> Is this the intended behavior?
>>>>>>>>>>> Is there a way to work around it?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232170.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232762.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28257864.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File consumer with noop=true&recursive=true
Posted by sagy <sa...@gmail.com>.
I've tested it and it seems to be working fine now.
Thanks so much for the quick fix
Sagy
Claus Ibsen-2 wrote:
>
> On Tue, Apr 13, 2010 at 6:35 PM, sagy <sa...@gmail.com> wrote:
>>
>> Thanks Claus!
>>
>
> You are welcome to test 2.3-SNAPSHOT as I have committed a fix just now.
> http://camel.apache.org/download.html
>
> Mind that it takes a bit time for Apache to deploy a new SNAPSHOT on
> their servers.
>
>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Tue, Apr 13, 2010 at 5:49 PM, sagy <sa...@gmail.com> wrote:
>>>>
>>>> exactly. Lets assume a directory root called /test
>>>> the writing application will write the following files:
>>>> /test/file1.txt
>>>> /test/file2.txt
>>>> /test/sub-folder/file1.txt
>>>>
>>>> etc..
>>>> My application will get an event for
>>>> /test/file1.txt
>>>> /test/file2.txt
>>>>
>>>> but not for
>>>> /test/sub-folder/file1.txt
>>>>
>>>> since the IdempotentRepository already contains an entry called
>>>> "file1.txt".
>>>> It was created for the /test/file1.txt event.
>>>>
>>>
>>> Noted. I have created a ticket to allow you define this with an option
>>> on the file component
>>> https://issues.apache.org/activemq/browse/CAMEL-2640
>>>
>>>>
>>>>
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>>>>>>
>>>>>> I'm trying to consume files from it not skip it.
>>>>>> This is a general description of the situation:
>>>>>>
>>>>>> There is another application with write permissions which create
>>>>>> files
>>>>>> in
>>>>>> that directory and sub directories.
>>>>>> My application only has read permissions but I still need to consume
>>>>>> the
>>>>>> files from that directory structure.
>>>>>>
>>>>>> That's why I'm using the "noop=true" flag. I can't move the files or
>>>>>> delete
>>>>>> them.
>>>>>> It seems from the documentation that noop is my only choice.
>>>>>> But a combination of "recursive=true" and "noop=true" causes the
>>>>>> problem
>>>>>> I've described in the original post.
>>>>>>
>>>>>
>>>>> And that app can write files with the same name?
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> I'm actually trying to consume files from a directory recursively
>>>>>>>> and
>>>>>>>> the
>>>>>>>> directory has read-only permissions.
>>>>>>>> Is there another way to do it without using
>>>>>>>> noop=true&recursive=true
>>>>>>>> ?
>>>>>>>>
>>>>>>>
>>>>>>> You can filter out the read-only directory if you should "skip" it.
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sagy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Yeah its on purpose for some reason, which as I can't really
>>>>>>>>> recall
>>>>>>>>> right
>>>>>>>>> now.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>>>>>
>>>>>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>>>>>
>>>>>>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>>>>>>> called.
>>>>>>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>>>>>>> example
>>>>>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>>>>>> But when I drop the file f1.txt into test/sub the processor
>>>>>>>>>> doesn't
>>>>>>>>>> get
>>>>>>>>>> called.
>>>>>>>>>> This is happening because of line 115 in camel-core
>>>>>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>>>>>
>>>>>>>>>> // only add to idempotent repository if we could
>>>>>>>>>> process
>>>>>>>>>> the
>>>>>>>>>> file
>>>>>>>>>> // only use the filename as the key as the file could
>>>>>>>>>> be
>>>>>>>>>> moved
>>>>>>>>>> into a done folder
>>>>>>>>>>
>>>>>>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>>>>>
>>>>>>>>>> The repository stores just the file name without the path.
>>>>>>>>>> Therefore if 2 files with the same name are dropped to 2
>>>>>>>>>> different
>>>>>>>>>> sub
>>>>>>>>>> directories only the first one will be processed.
>>>>>>>>>> Is this the intended behavior?
>>>>>>>>>> Is there a way to work around it?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claus Ibsen
>>>>>>>>> Apache Camel Committer
>>>>>>>>>
>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232170.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232762.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28257864.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer with noop=true&recursive=true
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Apr 13, 2010 at 6:35 PM, sagy <sa...@gmail.com> wrote:
>
> Thanks Claus!
>
You are welcome to test 2.3-SNAPSHOT as I have committed a fix just now.
http://camel.apache.org/download.html
Mind that it takes a bit time for Apache to deploy a new SNAPSHOT on
their servers.
>
> Claus Ibsen-2 wrote:
>>
>> On Tue, Apr 13, 2010 at 5:49 PM, sagy <sa...@gmail.com> wrote:
>>>
>>> exactly. Lets assume a directory root called /test
>>> the writing application will write the following files:
>>> /test/file1.txt
>>> /test/file2.txt
>>> /test/sub-folder/file1.txt
>>>
>>> etc..
>>> My application will get an event for
>>> /test/file1.txt
>>> /test/file2.txt
>>>
>>> but not for
>>> /test/sub-folder/file1.txt
>>>
>>> since the IdempotentRepository already contains an entry called
>>> "file1.txt".
>>> It was created for the /test/file1.txt event.
>>>
>>
>> Noted. I have created a ticket to allow you define this with an option
>> on the file component
>> https://issues.apache.org/activemq/browse/CAMEL-2640
>>
>>>
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>>>>>
>>>>> I'm trying to consume files from it not skip it.
>>>>> This is a general description of the situation:
>>>>>
>>>>> There is another application with write permissions which create files
>>>>> in
>>>>> that directory and sub directories.
>>>>> My application only has read permissions but I still need to consume
>>>>> the
>>>>> files from that directory structure.
>>>>>
>>>>> That's why I'm using the "noop=true" flag. I can't move the files or
>>>>> delete
>>>>> them.
>>>>> It seems from the documentation that noop is my only choice.
>>>>> But a combination of "recursive=true" and "noop=true" causes the
>>>>> problem
>>>>> I've described in the original post.
>>>>>
>>>>
>>>> And that app can write files with the same name?
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>
>>>>>>> I'm actually trying to consume files from a directory recursively and
>>>>>>> the
>>>>>>> directory has read-only permissions.
>>>>>>> Is there another way to do it without using noop=true&recursive=true
>>>>>>> ?
>>>>>>>
>>>>>>
>>>>>> You can filter out the read-only directory if you should "skip" it.
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Sagy
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Yeah its on purpose for some reason, which as I can't really recall
>>>>>>>> right
>>>>>>>> now.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>>>>
>>>>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>>>>
>>>>>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>>>>>> called.
>>>>>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>>>>>> example
>>>>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>>>>> But when I drop the file f1.txt into test/sub the processor doesn't
>>>>>>>>> get
>>>>>>>>> called.
>>>>>>>>> This is happening because of line 115 in camel-core
>>>>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>>>>
>>>>>>>>> // only add to idempotent repository if we could process
>>>>>>>>> the
>>>>>>>>> file
>>>>>>>>> // only use the filename as the key as the file could be
>>>>>>>>> moved
>>>>>>>>> into a done folder
>>>>>>>>>
>>>>>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>>>>
>>>>>>>>> The repository stores just the file name without the path.
>>>>>>>>> Therefore if 2 files with the same name are dropped to 2 different
>>>>>>>>> sub
>>>>>>>>> directories only the first one will be processed.
>>>>>>>>> Is this the intended behavior?
>>>>>>>>> Is there a way to work around it?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232170.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232762.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File consumer with noop=true&recursive=true
Posted by sagy <sa...@gmail.com>.
Thanks Claus!
Claus Ibsen-2 wrote:
>
> On Tue, Apr 13, 2010 at 5:49 PM, sagy <sa...@gmail.com> wrote:
>>
>> exactly. Lets assume a directory root called /test
>> the writing application will write the following files:
>> /test/file1.txt
>> /test/file2.txt
>> /test/sub-folder/file1.txt
>>
>> etc..
>> My application will get an event for
>> /test/file1.txt
>> /test/file2.txt
>>
>> but not for
>> /test/sub-folder/file1.txt
>>
>> since the IdempotentRepository already contains an entry called
>> "file1.txt".
>> It was created for the /test/file1.txt event.
>>
>
> Noted. I have created a ticket to allow you define this with an option
> on the file component
> https://issues.apache.org/activemq/browse/CAMEL-2640
>
>>
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>>>>
>>>> I'm trying to consume files from it not skip it.
>>>> This is a general description of the situation:
>>>>
>>>> There is another application with write permissions which create files
>>>> in
>>>> that directory and sub directories.
>>>> My application only has read permissions but I still need to consume
>>>> the
>>>> files from that directory structure.
>>>>
>>>> That's why I'm using the "noop=true" flag. I can't move the files or
>>>> delete
>>>> them.
>>>> It seems from the documentation that noop is my only choice.
>>>> But a combination of "recursive=true" and "noop=true" causes the
>>>> problem
>>>> I've described in the original post.
>>>>
>>>
>>> And that app can write files with the same name?
>>>
>>>
>>>
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>
>>>>>> I'm actually trying to consume files from a directory recursively and
>>>>>> the
>>>>>> directory has read-only permissions.
>>>>>> Is there another way to do it without using noop=true&recursive=true
>>>>>> ?
>>>>>>
>>>>>
>>>>> You can filter out the read-only directory if you should "skip" it.
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Sagy
>>>>>>
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Yeah its on purpose for some reason, which as I can't really recall
>>>>>>> right
>>>>>>> now.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>>>
>>>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>>>
>>>>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>>>>> called.
>>>>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>>>>> example
>>>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>>>> But when I drop the file f1.txt into test/sub the processor doesn't
>>>>>>>> get
>>>>>>>> called.
>>>>>>>> This is happening because of line 115 in camel-core
>>>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>>>
>>>>>>>> // only add to idempotent repository if we could process
>>>>>>>> the
>>>>>>>> file
>>>>>>>> // only use the filename as the key as the file could be
>>>>>>>> moved
>>>>>>>> into a done folder
>>>>>>>>
>>>>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>>>
>>>>>>>> The repository stores just the file name without the path.
>>>>>>>> Therefore if 2 files with the same name are dropped to 2 different
>>>>>>>> sub
>>>>>>>> directories only the first one will be processed.
>>>>>>>> Is this the intended behavior?
>>>>>>>> Is there a way to work around it?
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232170.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232762.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer with noop=true&recursive=true
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Apr 13, 2010 at 5:49 PM, sagy <sa...@gmail.com> wrote:
>
> exactly. Lets assume a directory root called /test
> the writing application will write the following files:
> /test/file1.txt
> /test/file2.txt
> /test/sub-folder/file1.txt
>
> etc..
> My application will get an event for
> /test/file1.txt
> /test/file2.txt
>
> but not for
> /test/sub-folder/file1.txt
>
> since the IdempotentRepository already contains an entry called "file1.txt".
> It was created for the /test/file1.txt event.
>
Noted. I have created a ticket to allow you define this with an option
on the file component
https://issues.apache.org/activemq/browse/CAMEL-2640
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>>>
>>> I'm trying to consume files from it not skip it.
>>> This is a general description of the situation:
>>>
>>> There is another application with write permissions which create files in
>>> that directory and sub directories.
>>> My application only has read permissions but I still need to consume the
>>> files from that directory structure.
>>>
>>> That's why I'm using the "noop=true" flag. I can't move the files or
>>> delete
>>> them.
>>> It seems from the documentation that noop is my only choice.
>>> But a combination of "recursive=true" and "noop=true" causes the problem
>>> I've described in the original post.
>>>
>>
>> And that app can write files with the same name?
>>
>>
>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>
>>>>> I'm actually trying to consume files from a directory recursively and
>>>>> the
>>>>> directory has read-only permissions.
>>>>> Is there another way to do it without using noop=true&recursive=true ?
>>>>>
>>>>
>>>> You can filter out the read-only directory if you should "skip" it.
>>>>
>>>>
>>>>> Thanks,
>>>>> Sagy
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Yeah its on purpose for some reason, which as I can't really recall
>>>>>> right
>>>>>> now.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>>
>>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>>
>>>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>>>> called.
>>>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>>>> example
>>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>>> But when I drop the file f1.txt into test/sub the processor doesn't
>>>>>>> get
>>>>>>> called.
>>>>>>> This is happening because of line 115 in camel-core
>>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>>
>>>>>>> // only add to idempotent repository if we could process
>>>>>>> the
>>>>>>> file
>>>>>>> // only use the filename as the key as the file could be
>>>>>>> moved
>>>>>>> into a done folder
>>>>>>>
>>>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>>
>>>>>>> The repository stores just the file name without the path.
>>>>>>> Therefore if 2 files with the same name are dropped to 2 different
>>>>>>> sub
>>>>>>> directories only the first one will be processed.
>>>>>>> Is this the intended behavior?
>>>>>>> Is there a way to work around it?
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232170.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File consumer with noop=true&recursive=true
Posted by sagy <sa...@gmail.com>.
Here is a small junit test which explains the problem:
public void testIdempotentConsumerDoesntIgnoreRecursiveDuplicates() throws
Exception {
String newFile1 = "some_dir/some_file3";
String newFile2 = "some_file3";
mockRecognizerService.setExpectedMessageCount(2);
dropFileInTargetDir(newFile1);
dropFileInTargetDir(newFile2);
Thread.sleep(4000);
mockRecognizerService.assertIsSatisfied();
}
dropFileInTargetDir - creates a file under the directory which the file
component is listening to.
This test fails.
sagy wrote:
>
> exactly. Lets assume a directory root called /test
> the writing application will write the following files:
> /test/file1.txt
> /test/file2.txt
> /test/sub-folder/file1.txt
>
> etc..
> My application will get an event for
> /test/file1.txt
> /test/file2.txt
>
> but not for
> /test/sub-folder/file1.txt
>
> since the IdempotentRepository already contains an entry called
> "file1.txt". It was created for the /test/file1.txt event.
>
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>>>
>>> I'm trying to consume files from it not skip it.
>>> This is a general description of the situation:
>>>
>>> There is another application with write permissions which create files
>>> in
>>> that directory and sub directories.
>>> My application only has read permissions but I still need to consume the
>>> files from that directory structure.
>>>
>>> That's why I'm using the "noop=true" flag. I can't move the files or
>>> delete
>>> them.
>>> It seems from the documentation that noop is my only choice.
>>> But a combination of "recursive=true" and "noop=true" causes the problem
>>> I've described in the original post.
>>>
>>
>> And that app can write files with the same name?
>>
>>
>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>
>>>>> I'm actually trying to consume files from a directory recursively and
>>>>> the
>>>>> directory has read-only permissions.
>>>>> Is there another way to do it without using noop=true&recursive=true ?
>>>>>
>>>>
>>>> You can filter out the read-only directory if you should "skip" it.
>>>>
>>>>
>>>>> Thanks,
>>>>> Sagy
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Yeah its on purpose for some reason, which as I can't really recall
>>>>>> right
>>>>>> now.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>>
>>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>>
>>>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>>>> called.
>>>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>>>> example
>>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>>> But when I drop the file f1.txt into test/sub the processor doesn't
>>>>>>> get
>>>>>>> called.
>>>>>>> This is happening because of line 115 in camel-core
>>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>>
>>>>>>> // only add to idempotent repository if we could process
>>>>>>> the
>>>>>>> file
>>>>>>> // only use the filename as the key as the file could be
>>>>>>> moved
>>>>>>> into a done folder
>>>>>>>
>>>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>>
>>>>>>> The repository stores just the file name without the path.
>>>>>>> Therefore if 2 files with the same name are dropped to 2 different
>>>>>>> sub
>>>>>>> directories only the first one will be processed.
>>>>>>> Is this the intended behavior?
>>>>>>> Is there a way to work around it?
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
>
--
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232225.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer with noop=true&recursive=true
Posted by sagy <sa...@gmail.com>.
exactly. Lets assume a directory root called /test
the writing application will write the following files:
/test/file1.txt
/test/file2.txt
/test/sub-folder/file1.txt
etc..
My application will get an event for
/test/file1.txt
/test/file2.txt
but not for
/test/sub-folder/file1.txt
since the IdempotentRepository already contains an entry called "file1.txt".
It was created for the /test/file1.txt event.
Claus Ibsen-2 wrote:
>
> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>>
>> I'm trying to consume files from it not skip it.
>> This is a general description of the situation:
>>
>> There is another application with write permissions which create files in
>> that directory and sub directories.
>> My application only has read permissions but I still need to consume the
>> files from that directory structure.
>>
>> That's why I'm using the "noop=true" flag. I can't move the files or
>> delete
>> them.
>> It seems from the documentation that noop is my only choice.
>> But a combination of "recursive=true" and "noop=true" causes the problem
>> I've described in the original post.
>>
>
> And that app can write files with the same name?
>
>
>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>>
>>>> I'm actually trying to consume files from a directory recursively and
>>>> the
>>>> directory has read-only permissions.
>>>> Is there another way to do it without using noop=true&recursive=true ?
>>>>
>>>
>>> You can filter out the read-only directory if you should "skip" it.
>>>
>>>
>>>> Thanks,
>>>> Sagy
>>>>
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> Yeah its on purpose for some reason, which as I can't really recall
>>>>> right
>>>>> now.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>
>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>
>>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>>> called.
>>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>>> example
>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>> But when I drop the file f1.txt into test/sub the processor doesn't
>>>>>> get
>>>>>> called.
>>>>>> This is happening because of line 115 in camel-core
>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>
>>>>>> // only add to idempotent repository if we could process
>>>>>> the
>>>>>> file
>>>>>> // only use the filename as the key as the file could be
>>>>>> moved
>>>>>> into a done folder
>>>>>>
>>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>
>>>>>> The repository stores just the file name without the path.
>>>>>> Therefore if 2 files with the same name are dropped to 2 different
>>>>>> sub
>>>>>> directories only the first one will be processed.
>>>>>> Is this the intended behavior?
>>>>>> Is there a way to work around it?
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232170.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer with noop=true&recursive=true
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Apr 13, 2010 at 5:10 PM, sagy <sa...@gmail.com> wrote:
>
> I'm trying to consume files from it not skip it.
> This is a general description of the situation:
>
> There is another application with write permissions which create files in
> that directory and sub directories.
> My application only has read permissions but I still need to consume the
> files from that directory structure.
>
> That's why I'm using the "noop=true" flag. I can't move the files or delete
> them.
> It seems from the documentation that noop is my only choice.
> But a combination of "recursive=true" and "noop=true" causes the problem
> I've described in the original post.
>
And that app can write files with the same name?
>
>
> Claus Ibsen-2 wrote:
>>
>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>>
>>> I'm actually trying to consume files from a directory recursively and the
>>> directory has read-only permissions.
>>> Is there another way to do it without using noop=true&recursive=true ?
>>>
>>
>> You can filter out the read-only directory if you should "skip" it.
>>
>>
>>> Thanks,
>>> Sagy
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> Yeah its on purpose for some reason, which as I can't really recall
>>>> right
>>>> now.
>>>>
>>>>
>>>>
>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>> I'm using Camel 2.2 and have the following route:
>>>>>
>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>
>>>>> When I drop a file f1.txt into the test folder the processor gets
>>>>> called.
>>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>>> example
>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>> But when I drop the file f1.txt into test/sub the processor doesn't get
>>>>> called.
>>>>> This is happening because of line 115 in camel-core
>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>
>>>>> // only add to idempotent repository if we could process the
>>>>> file
>>>>> // only use the filename as the key as the file could be
>>>>> moved
>>>>> into a done folder
>>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>
>>>>> The repository stores just the file name without the path.
>>>>> Therefore if 2 files with the same name are dropped to 2 different sub
>>>>> directories only the first one will be processed.
>>>>> Is this the intended behavior?
>>>>> Is there a way to work around it?
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File consumer with noop=true&recursive=true
Posted by sagy <sa...@gmail.com>.
I'm trying to consume files from it not skip it.
This is a general description of the situation:
There is another application with write permissions which create files in
that directory and sub directories.
My application only has read permissions but I still need to consume the
files from that directory structure.
That's why I'm using the "noop=true" flag. I can't move the files or delete
them.
It seems from the documentation that noop is my only choice.
But a combination of "recursive=true" and "noop=true" causes the problem
I've described in the original post.
Claus Ibsen-2 wrote:
>
> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>>
>> I'm actually trying to consume files from a directory recursively and the
>> directory has read-only permissions.
>> Is there another way to do it without using noop=true&recursive=true ?
>>
>
> You can filter out the read-only directory if you should "skip" it.
>
>
>> Thanks,
>> Sagy
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> Yeah its on purpose for some reason, which as I can't really recall
>>> right
>>> now.
>>>
>>>
>>>
>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>> I'm using Camel 2.2 and have the following route:
>>>>
>>>> from("file://test?recursive=true&noop=true").process(...
>>>>
>>>> When I drop a file f1.txt into the test folder the processor gets
>>>> called.
>>>> When I drop the file f2.txt into a sub folder of test folder for
>>>> example
>>>> test/sub/f2.txt ,again the processor gets called.
>>>> But when I drop the file f1.txt into test/sub the processor doesn't get
>>>> called.
>>>> This is happening because of line 115 in camel-core
>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>
>>>> // only add to idempotent repository if we could process the
>>>> file
>>>> // only use the filename as the key as the file could be
>>>> moved
>>>> into a done folder
>>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>>
>>>> The repository stores just the file name without the path.
>>>> Therefore if 2 files with the same name are dropped to 2 different sub
>>>> directories only the first one will be processed.
>>>> Is this the intended behavior?
>>>> Is there a way to work around it?
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer with noop=true&recursive=true
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Apr 13, 2010 at 3:33 PM, sagy <sa...@gmail.com> wrote:
>
> I'm actually trying to consume files from a directory recursively and the
> directory has read-only permissions.
> Is there another way to do it without using noop=true&recursive=true ?
>
You can filter out the read-only directory if you should "skip" it.
> Thanks,
> Sagy
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Yeah its on purpose for some reason, which as I can't really recall right
>> now.
>>
>>
>>
>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>>
>>> Hi,
>>> I'm using Camel 2.2 and have the following route:
>>>
>>> from("file://test?recursive=true&noop=true").process(...
>>>
>>> When I drop a file f1.txt into the test folder the processor gets called.
>>> When I drop the file f2.txt into a sub folder of test folder for example
>>> test/sub/f2.txt ,again the processor gets called.
>>> But when I drop the file f1.txt into test/sub the processor doesn't get
>>> called.
>>> This is happening because of line 115 in camel-core
>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>
>>> // only add to idempotent repository if we could process the
>>> file
>>> // only use the filename as the key as the file could be moved
>>> into a done folder
>>> endpoint.getIdempotentRepository().add(file.getFileName());
>>>
>>> The repository stores just the file name without the path.
>>> Therefore if 2 files with the same name are dropped to 2 different sub
>>> directories only the first one will be processed.
>>> Is this the intended behavior?
>>> Is there a way to work around it?
>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: File consumer with noop=true&recursive=true
Posted by sagy <sa...@gmail.com>.
I'm actually trying to consume files from a directory recursively and the
directory has read-only permissions.
Is there another way to do it without using noop=true&recursive=true ?
Thanks,
Sagy
Claus Ibsen-2 wrote:
>
> Hi
>
> Yeah its on purpose for some reason, which as I can't really recall right
> now.
>
>
>
> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>>
>> Hi,
>> I'm using Camel 2.2 and have the following route:
>>
>> from("file://test?recursive=true&noop=true").process(...
>>
>> When I drop a file f1.txt into the test folder the processor gets called.
>> When I drop the file f2.txt into a sub folder of test folder for example
>> test/sub/f2.txt ,again the processor gets called.
>> But when I drop the file f1.txt into test/sub the processor doesn't get
>> called.
>> This is happening because of line 115 in camel-core
>> org.apache.camel.component.file.GenericFileOnCompletion:
>>
>> // only add to idempotent repository if we could process the
>> file
>> // only use the filename as the key as the file could be moved
>> into a done folder
>> endpoint.getIdempotentRepository().add(file.getFileName());
>>
>> The repository stores just the file name without the path.
>> Therefore if 2 files with the same name are dropped to 2 different sub
>> directories only the first one will be processed.
>> Is this the intended behavior?
>> Is there a way to work around it?
>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer with noop=true&recursive=true
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah its on purpose for some reason, which as I can't really recall right now.
On Tue, Apr 13, 2010 at 2:33 PM, sagy <sa...@gmail.com> wrote:
>
> Hi,
> I'm using Camel 2.2 and have the following route:
>
> from("file://test?recursive=true&noop=true").process(...
>
> When I drop a file f1.txt into the test folder the processor gets called.
> When I drop the file f2.txt into a sub folder of test folder for example
> test/sub/f2.txt ,again the processor gets called.
> But when I drop the file f1.txt into test/sub the processor doesn't get
> called.
> This is happening because of line 115 in camel-core
> org.apache.camel.component.file.GenericFileOnCompletion:
>
> // only add to idempotent repository if we could process the
> file
> // only use the filename as the key as the file could be moved
> into a done folder
> endpoint.getIdempotentRepository().add(file.getFileName());
>
> The repository stores just the file name without the path.
> Therefore if 2 files with the same name are dropped to 2 different sub
> directories only the first one will be processed.
> Is this the intended behavior?
> Is there a way to work around it?
>
>
> --
> View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus