You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Bhuvan Gupta <bh...@gmail.com> on 2018/01/02 05:02:10 UTC
Fwd: Camel processing single file twice in 'file' endpoint
Hello nice people,
We are using camel 2.17 with java 1.7 in our sample project and we defined
our route as:
from("file:stageDummy")
.threads(1)
.process(myProcess);
We first add 1000 file in stageDummy folder, then we run the code and we
get exception as below
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
at org.apache.camel.component.file.strategy.GenericFileProcessS
trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
at org.apache.camel.component.file.strategy.GenericFileRenamePr
ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
at org.apache.camel.component.file.GenericFileOnCompletion.proc
essStrategyCommit(GenericFileOnCompletion.java:127)
at org.apache.camel.component.file.GenericFileOnCompletion.onCo
mpletion(GenericFileOnCompletion.java:83)
at org.apache.camel.component.file.GenericFileOnCompletion.onCo
mplete(GenericFileOnCompletion.java:57)
at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
UnitOfWorkHelper.java:104)
at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
ProcessorAdvice.after(CamelInternalProcessor.java:654)
at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
ProcessorAdvice.after(CamelInternalProcessor.java:609)
at org.apache.camel.processor.CamelInternalProcessor$InternalCa
llback.done(CamelInternalProcessor.java:239)
at org.apache.camel.processor.CamelInternalProcessor$InternalCa
llback.done(CamelInternalProcessor.java:250)
at org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
eliveryErrorHandler.java:480)
at org.apache.camel.management.InstrumentationProcessor$1.done(
InstrumentationProcessor.java:86)
at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
ThreadsProcessor.java:85)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
Executor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
lExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
As per our understanding camel picks up the file from the mentioned
directory i.e in my case-stageDummy and processed by process-myProcess
and moves it to .camel sub folder.We noticed that if we were having more
than or around 1000 files in the folder -stageDummy ,there were some
exceptions
thrown
This exception is thrown randomly for n number of files,where n changes
everytime we execute our main.Note that we were not trying to access the
exchange
instead we kept our process simple.We could see that if there are 1000
files in our folder then the processor is getting called 1000+x times
,where x changes everytime example it gets called 1024 times,and we could
see that camel is trying to process these x number of files again
We are attaching the Main class as well as the project zip
AND HAPPY NEW YEAR.
hoping to resolve it soon
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
I have created a bug for it as we are able to see the same for Oracle 1.8
and 2.20.1 version as well
https://issues.apache.org/jira/browse/CAMEL-12112
Thanks
On Wed, Jan 3, 2018 at 10:27 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> Onder,
>
> I am running camel in default mode. i.e
> [1] setup pom
> [2] run the sample shared.
>
> I am not doing any specific configuration
>
> On Tue, Jan 2, 2018 at 8:38 PM, Onder SEZGIN <on...@apache.org> wrote:
>
>> hi, did you check default behaviour of file consumer depending on your
>> environment and readLock option on the endpoint as well?
>>
>> On Tue, Jan 2, 2018 at 8:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
>>
>> > Hello nice people,
>> >
>> > We are using camel 2.17 with java 1.7 in our sample project and we
>> defined
>> > our route as:
>> >
>> > from("file:stageDummy")
>> > .threads(1)
>> > .process(myProcess);
>> >
>> > We first add 1000 file in stageDummy folder, then we run the code and
>> > we get exception as below
>> >
>> >
>> > org.apache.camel.component.file.GenericFileOperationFailedException:
>> > Cannot rename file: GenericFile[newFile319] to:
>> > GenericFile[.camel/newFile319]
>> > at org.apache.camel.component.file.strategy.GenericFileProcessS
>> > trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
>> > at org.apache.camel.component.file.strategy.GenericFileRenamePr
>> > ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>> > at org.apache.camel.component.file.GenericFileOnCompletion.proc
>> > essStrategyCommit(GenericFileOnCompletion.java:127)
>> > at org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> > mpletion(GenericFileOnCompletion.java:83)
>> > at org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> > mplete(GenericFileOnCompletion.java:57)
>> > at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
>> > UnitOfWorkHelper.java:104)
>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
>> > rk.java:229)
>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
>> > per.java:65)
>> > at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> > ProcessorAdvice.after(CamelInternalProcessor.java:654)
>> > at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> > ProcessorAdvice.after(CamelInternalProcessor.java:609)
>> > at org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> > llback.done(CamelInternalProcessor.java:239)
>> > at org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> > llback.done(CamelInternalProcessor.java:250)
>> > at org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
>> > eliveryErrorHandler.java:480)
>> > at org.apache.camel.management.InstrumentationProcessor$1.done(
>> > InstrumentationProcessor.java:86)
>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
>> > at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
>> > ThreadsProcessor.java:85)
>> > at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> > Executor.java:1145)
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> > lExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > As per our understanding camel picks up the file from the mentioned
>> > directory i.e in my case-stageDummy and processed by process-myProcess
>> > and moves it to .camel sub folder.We noticed that if we were having more
>> > than or around 1000 files in the folder -stageDummy ,there were some
>> > exceptions
>> > thrown
>> >
>> > This exception is thrown randomly for n number of files,where n changes
>> > everytime we execute our main.Note that we were not trying to access the
>> > exchange
>> > instead we kept our process simple.We could see that if there are 1000
>> > files in our folder then the processor is getting called 1000+x times
>> > ,where x changes everytime example it gets called 1024 times,and we
>> could
>> > see that camel is trying to process these x number of files again
>> >
>> > We are attaching the Main class as well as the project zip
>> >
>> > AND HAPPY NEW YEAR.
>> >
>> > hoping to resolve it soon
>> >
>> >
>> >
>> >
>> >
>>
>
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
Onder,
I am running camel in default mode. i.e
[1] setup pom
[2] run the sample shared.
I am not doing any specific configuration
On Tue, Jan 2, 2018 at 8:38 PM, Onder SEZGIN <on...@apache.org> wrote:
> hi, did you check default behaviour of file consumer depending on your
> environment and readLock option on the endpoint as well?
>
> On Tue, Jan 2, 2018 at 8:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
>
> > Hello nice people,
> >
> > We are using camel 2.17 with java 1.7 in our sample project and we
> defined
> > our route as:
> >
> > from("file:stageDummy")
> > .threads(1)
> > .process(myProcess);
> >
> > We first add 1000 file in stageDummy folder, then we run the code and
> > we get exception as below
> >
> >
> > org.apache.camel.component.file.GenericFileOperationFailedException:
> > Cannot rename file: GenericFile[newFile319] to:
> > GenericFile[.camel/newFile319]
> > at org.apache.camel.component.file.strategy.GenericFileProcessS
> > trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
> > at org.apache.camel.component.file.strategy.GenericFileRenamePr
> > ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> > at org.apache.camel.component.file.GenericFileOnCompletion.proc
> > essStrategyCommit(GenericFileOnCompletion.java:127)
> > at org.apache.camel.component.file.GenericFileOnCompletion.onCo
> > mpletion(GenericFileOnCompletion.java:83)
> > at org.apache.camel.component.file.GenericFileOnCompletion.onCo
> > mplete(GenericFileOnCompletion.java:57)
> > at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
> > UnitOfWorkHelper.java:104)
> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
> > rk.java:229)
> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
> > per.java:65)
> > at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> > ProcessorAdvice.after(CamelInternalProcessor.java:654)
> > at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> > ProcessorAdvice.after(CamelInternalProcessor.java:609)
> > at org.apache.camel.processor.CamelInternalProcessor$InternalCa
> > llback.done(CamelInternalProcessor.java:239)
> > at org.apache.camel.processor.CamelInternalProcessor$InternalCa
> > llback.done(CamelInternalProcessor.java:250)
> > at org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
> > eliveryErrorHandler.java:480)
> > at org.apache.camel.management.InstrumentationProcessor$1.done(
> > InstrumentationProcessor.java:86)
> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> > at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
> > ThreadsProcessor.java:85)
> > at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > Executor.java:1145)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > lExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > As per our understanding camel picks up the file from the mentioned
> > directory i.e in my case-stageDummy and processed by process-myProcess
> > and moves it to .camel sub folder.We noticed that if we were having more
> > than or around 1000 files in the folder -stageDummy ,there were some
> > exceptions
> > thrown
> >
> > This exception is thrown randomly for n number of files,where n changes
> > everytime we execute our main.Note that we were not trying to access the
> > exchange
> > instead we kept our process simple.We could see that if there are 1000
> > files in our folder then the processor is getting called 1000+x times
> > ,where x changes everytime example it gets called 1024 times,and we could
> > see that camel is trying to process these x number of files again
> >
> > We are attaching the Main class as well as the project zip
> >
> > AND HAPPY NEW YEAR.
> >
> > hoping to resolve it soon
> >
> >
> >
> >
> >
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Onder SEZGIN <on...@apache.org>.
hi, did you check default behaviour of file consumer depending on your
environment and readLock option on the endpoint as well?
On Tue, Jan 2, 2018 at 8:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> Hello nice people,
>
> We are using camel 2.17 with java 1.7 in our sample project and we defined
> our route as:
>
> from("file:stageDummy")
> .threads(1)
> .process(myProcess);
>
> We first add 1000 file in stageDummy folder, then we run the code and
> we get exception as below
>
>
> org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot rename file: GenericFile[newFile319] to:
> GenericFile[.camel/newFile319]
> at org.apache.camel.component.file.strategy.GenericFileProcessS
> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
> at org.apache.camel.component.file.strategy.GenericFileRenamePr
> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> at org.apache.camel.component.file.GenericFileOnCompletion.proc
> essStrategyCommit(GenericFileOnCompletion.java:127)
> at org.apache.camel.component.file.GenericFileOnCompletion.onCo
> mpletion(GenericFileOnCompletion.java:83)
> at org.apache.camel.component.file.GenericFileOnCompletion.onCo
> mplete(GenericFileOnCompletion.java:57)
> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
> UnitOfWorkHelper.java:104)
> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
> rk.java:229)
> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
> per.java:65)
> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> ProcessorAdvice.after(CamelInternalProcessor.java:654)
> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> ProcessorAdvice.after(CamelInternalProcessor.java:609)
> at org.apache.camel.processor.CamelInternalProcessor$InternalCa
> llback.done(CamelInternalProcessor.java:239)
> at org.apache.camel.processor.CamelInternalProcessor$InternalCa
> llback.done(CamelInternalProcessor.java:250)
> at org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
> eliveryErrorHandler.java:480)
> at org.apache.camel.management.InstrumentationProcessor$1.done(
> InstrumentationProcessor.java:86)
> at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
> ThreadsProcessor.java:85)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> As per our understanding camel picks up the file from the mentioned
> directory i.e in my case-stageDummy and processed by process-myProcess
> and moves it to .camel sub folder.We noticed that if we were having more
> than or around 1000 files in the folder -stageDummy ,there were some
> exceptions
> thrown
>
> This exception is thrown randomly for n number of files,where n changes
> everytime we execute our main.Note that we were not trying to access the
> exchange
> instead we kept our process simple.We could see that if there are 1000
> files in our folder then the processor is getting called 1000+x times
> ,where x changes everytime example it gets called 1024 times,and we could
> see that camel is trying to process these x number of files again
>
> We are attaching the Main class as well as the project zip
>
> AND HAPPY NEW YEAR.
>
> hoping to resolve it soon
>
>
>
>
>
Re: Camel processing single file twice in 'file' endpoint
Posted by si...@yahoo.com.INVALID.
I suspect using “File consumer + threads(…) + default readLock” is like having multiple File consumer watching the same folder - they’ll screw up each other.
I offer this as a possible proof: if you add an `idempotent=true` to your URI, the problem should go away.
p/s: I think you’re trying to process your files parallel-ly. If that is your _actual_ question, I have no idea if the above is what you’re supposed to do even if it works.
From: Claus Ibsen
Sent: Wednesday, 3 January, 2018 9:56 PM
To: users@camel.apache.org
We cannot reproduce your problem.
There can be several reasons why a file cannot be renamed, but
unfortunately the file api just returns a boolean if it cannot rename
a file.
A typical problem is that your do some custom code where you read the
file or do something with a input stream you do not close properly.
Also using threads(1) does not really make sense.
On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> Hello nice people,
>
> We are using camel 2.17 with java 1.7 in our sample project and we defined
> our route as:
>
> from("file:stageDummy")
> .threads(1)
> .process(myProcess);
>
> We first add 1000 file in stageDummy folder, then we run the code and we
> get exception as below
>
>
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
> rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
> at
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
> at
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
> at
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
> at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:654)
> at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:609)
> at
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
> at
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:250)
> at
> org.apache.camel.processor.RedeliveryErrorHandler$1.done(RedeliveryErrorHandler.java:480)
> at
> org.apache.camel.management.InstrumentationProcessor$1.done(InstrumentationProcessor.java:86)
> at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> at
> org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:85)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> As per our understanding camel picks up the file from the mentioned
> directory i.e in my case-stageDummy and processed by process-myProcess
> and moves it to .camel sub folder.We noticed that if we were having more
> than or around 1000 files in the folder -stageDummy ,there were some
> exceptions
> thrown
>
> This exception is thrown randomly for n number of files,where n changes
> everytime we execute our main.Note that we were not trying to access the
> exchange
> instead we kept our process simple.We could see that if there are 1000 files
> in our folder then the processor is getting called 1000+x times ,where x
> changes everytime example it gets called 1024 times,and we could see that
> camel is trying to process these x number of files again
>
> We are attaching the Main class as well as the project zip
>
> AND HAPPY NEW YEAR.
>
> hoping to resolve it soon
>
>
>
>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
Hello,
>>A typical problem is that your do some custom code where you read the
>>file or do something with a input stream you do not close properly.
In the sample project shared our processor is doing nothing with file.
Here is the log attached.
Key Observations from log
[1] At end of the log I am printing the file which is handled twice. An
example is "newFile887"
[2] Also as soon as we enter the processor I print the
*"System.out.println("inside
processor"+fileName);"*
[2.1] We can see that *"inside processornewFile887" *is printed first at
line 7220 and then again at line8271
[3] I tried it on 2.19.0 with java 1.8
Note : the issue happen in 2 out of 3 time
PFA:
[1] log file [i have removed the other logs at mail dont go with large file]
[2] updated App.java with better sysouts
On Wed, Jan 3, 2018 at 8:19 PM, Bhuvan Gupta <bh...@gmail.com> wrote:
> I am working more on it prove it.
>
> meanwhile issue video https://drive.google.com/file/d/
> 1znGkVn2JtpFvtlUnwifPKCTWujAiFFPo/view
> See @00.05
>
> On Wed, Jan 3, 2018 at 7:26 PM, Claus Ibsen <cl...@gmail.com> wrote:
>
>> We cannot reproduce your problem.
>>
>> There can be several reasons why a file cannot be renamed, but
>> unfortunately the file api just returns a boolean if it cannot rename
>> a file.
>> A typical problem is that your do some custom code where you read the
>> file or do something with a input stream you do not close properly.
>>
>> Also using threads(1) does not really make sense.
>>
>>
>>
>> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
>> > Hello nice people,
>> >
>> > We are using camel 2.17 with java 1.7 in our sample project and we
>> defined
>> > our route as:
>> >
>> > from("file:stageDummy")
>> > .threads(1)
>> > .process(myProcess);
>> >
>> > We first add 1000 file in stageDummy folder, then we run the code
>> and we
>> > get exception as below
>> >
>> >
>> > org.apache.camel.component.file.GenericFileOperationFailedException:
>> Cannot
>> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
>> > at
>> > org.apache.camel.component.file.strategy.GenericFileProcessS
>> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
>> > at
>> > org.apache.camel.component.file.strategy.GenericFileRenamePr
>> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.proc
>> essStrategyCommit(GenericFileOnCompletion.java:127)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> mpletion(GenericFileOnCompletion.java:83)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> mplete(GenericFileOnCompletion.java:57)
>> > at
>> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
>> UnitOfWorkHelper.java:104)
>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
>> rk.java:229)
>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
>> per.java:65)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> ProcessorAdvice.after(CamelInternalProcessor.java:654)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> ProcessorAdvice.after(CamelInternalProcessor.java:609)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> llback.done(CamelInternalProcessor.java:239)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> llback.done(CamelInternalProcessor.java:250)
>> > at
>> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
>> eliveryErrorHandler.java:480)
>> > at
>> > org.apache.camel.management.InstrumentationProcessor$1.done(
>> InstrumentationProcessor.java:86)
>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
>> > at
>> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
>> ThreadsProcessor.java:85)
>> > at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > As per our understanding camel picks up the file from the mentioned
>> > directory i.e in my case-stageDummy and processed by process-myProcess
>> > and moves it to .camel sub folder.We noticed that if we were having more
>> > than or around 1000 files in the folder -stageDummy ,there were some
>> > exceptions
>> > thrown
>> >
>> > This exception is thrown randomly for n number of files,where n changes
>> > everytime we execute our main.Note that we were not trying to access the
>> > exchange
>> > instead we kept our process simple.We could see that if there are 1000
>> files
>> > in our folder then the processor is getting called 1000+x times ,where x
>> > changes everytime example it gets called 1024 times,and we could see
>> that
>> > camel is trying to process these x number of files again
>> >
>> > We are attaching the Main class as well as the project zip
>> >
>> > AND HAPPY NEW YEAR.
>> >
>> > hoping to resolve it soon
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>>
>
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
With TRUNCATED logs
[2.1] We can see that *"inside processornewFile887" *is printed first at
line 10 and then again at line 18
On Wed, Jan 3, 2018 at 8:39 PM, Bhuvan Gupta <bh...@gmail.com> wrote:
> Hello,
>
> >>A typical problem is that your do some custom code where you read the
> >>file or do something with a input stream you do not close properly.
> In the sample project shared our processor is doing nothing with file.
>
> Here is the log attached.
> Key Observations from log
> [1] At end of the log I am printing the file which is handled twice. An
> example is "newFile887"
> [2] Also as soon as we enter the processor I print the *"System.out.println("inside
> processor"+fileName);"*
> [2.1] We can see that *"inside processornewFile887" *is printed first at
> line 7220 and then again at line8271
> [3] I tried it on 2.19.0 with java 1.8
>
> Note : the issue happen in 2 out of 3 time
>
>
> PFA:
> [1] log file
> [2] updated App.java with better sysouts
>
>
>
>
>
>
> On Wed, Jan 3, 2018 at 8:19 PM, Bhuvan Gupta <bh...@gmail.com> wrote:
>
>> I am working more on it prove it.
>>
>> meanwhile issue video https://drive.google.com
>> /file/d/1znGkVn2JtpFvtlUnwifPKCTWujAiFFPo/view
>> See @00.05
>>
>> On Wed, Jan 3, 2018 at 7:26 PM, Claus Ibsen <cl...@gmail.com>
>> wrote:
>>
>>> We cannot reproduce your problem.
>>>
>>> There can be several reasons why a file cannot be renamed, but
>>> unfortunately the file api just returns a boolean if it cannot rename
>>> a file.
>>> A typical problem is that your do some custom code where you read the
>>> file or do something with a input stream you do not close properly.
>>>
>>> Also using threads(1) does not really make sense.
>>>
>>>
>>>
>>> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
>>> > Hello nice people,
>>> >
>>> > We are using camel 2.17 with java 1.7 in our sample project and we
>>> defined
>>> > our route as:
>>> >
>>> > from("file:stageDummy")
>>> > .threads(1)
>>> > .process(myProcess);
>>> >
>>> > We first add 1000 file in stageDummy folder, then we run the code
>>> and we
>>> > get exception as below
>>> >
>>> >
>>> > org.apache.camel.component.file.GenericFileOperationFailedException:
>>> Cannot
>>> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
>>> > at
>>> > org.apache.camel.component.file.strategy.GenericFileProcessS
>>> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
>>> > at
>>> > org.apache.camel.component.file.strategy.GenericFileRenamePr
>>> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>>> > at
>>> > org.apache.camel.component.file.GenericFileOnCompletion.proc
>>> essStrategyCommit(GenericFileOnCompletion.java:127)
>>> > at
>>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>>> mpletion(GenericFileOnCompletion.java:83)
>>> > at
>>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>>> mplete(GenericFileOnCompletion.java:57)
>>> > at
>>> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
>>> UnitOfWorkHelper.java:104)
>>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
>>> rk.java:229)
>>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
>>> per.java:65)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>>> ProcessorAdvice.after(CamelInternalProcessor.java:654)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>>> ProcessorAdvice.after(CamelInternalProcessor.java:609)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>>> llback.done(CamelInternalProcessor.java:239)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>>> llback.done(CamelInternalProcessor.java:250)
>>> > at
>>> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
>>> eliveryErrorHandler.java:480)
>>> > at
>>> > org.apache.camel.management.InstrumentationProcessor$1.done(
>>> InstrumentationProcessor.java:86)
>>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
>>> > at
>>> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
>>> ThreadsProcessor.java:85)
>>> > at java.util.concurrent.Executors$RunnableAdapter.call(Executor
>>> s.java:471)
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1145)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:615)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> >
>>> > As per our understanding camel picks up the file from the mentioned
>>> > directory i.e in my case-stageDummy and processed by process-myProcess
>>> > and moves it to .camel sub folder.We noticed that if we were having
>>> more
>>> > than or around 1000 files in the folder -stageDummy ,there were some
>>> > exceptions
>>> > thrown
>>> >
>>> > This exception is thrown randomly for n number of files,where n changes
>>> > everytime we execute our main.Note that we were not trying to access
>>> the
>>> > exchange
>>> > instead we kept our process simple.We could see that if there are 1000
>>> files
>>> > in our folder then the processor is getting called 1000+x times ,where
>>> x
>>> > changes everytime example it gets called 1024 times,and we could see
>>> that
>>> > camel is trying to process these x number of files again
>>> >
>>> > We are attaching the Main class as well as the project zip
>>> >
>>> > AND HAPPY NEW YEAR.
>>> >
>>> > hoping to resolve it soon
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> http://davsclaus.com @davsclaus
>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>
>>
>>
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
I am working more on it prove it.
meanwhile issue video
https://drive.google.com/file/d/1znGkVn2JtpFvtlUnwifPKCTWujAiFFPo/view
See @00.05
On Wed, Jan 3, 2018 at 7:26 PM, Claus Ibsen <cl...@gmail.com> wrote:
> We cannot reproduce your problem.
>
> There can be several reasons why a file cannot be renamed, but
> unfortunately the file api just returns a boolean if it cannot rename
> a file.
> A typical problem is that your do some custom code where you read the
> file or do something with a input stream you do not close properly.
>
> Also using threads(1) does not really make sense.
>
>
>
> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> > Hello nice people,
> >
> > We are using camel 2.17 with java 1.7 in our sample project and we
> defined
> > our route as:
> >
> > from("file:stageDummy")
> > .threads(1)
> > .process(myProcess);
> >
> > We first add 1000 file in stageDummy folder, then we run the code and
> we
> > get exception as below
> >
> >
> > org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot
> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
> > at
> > org.apache.camel.component.file.strategy.GenericFileProcessStrategySupp
> ort.renameFile(GenericFileProcessStrategySupport.java:115)
> > at
> > org.apache.camel.component.file.strategy.GenericFileRenameProcessStrate
> gy.commit(GenericFileRenameProcessStrategy.java:88)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.
> processStrategyCommit(GenericFileOnCompletion.java:127)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(
> GenericFileOnCompletion.java:83)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.onComplete(
> GenericFileOnCompletion.java:57)
> > at
> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
> UnitOfWorkHelper.java:104)
> > at org.apache.camel.impl.DefaultUnitOfWork.done(
> DefaultUnitOfWork.java:229)
> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(
> UnitOfWorkHelper.java:65)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$
> UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:654)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$
> UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:609)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(
> CamelInternalProcessor.java:239)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(
> CamelInternalProcessor.java:250)
> > at
> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(
> RedeliveryErrorHandler.java:480)
> > at
> > org.apache.camel.management.InstrumentationProcessor$1.
> done(InstrumentationProcessor.java:86)
> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> > at
> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.
> run(ThreadsProcessor.java:85)
> > at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > As per our understanding camel picks up the file from the mentioned
> > directory i.e in my case-stageDummy and processed by process-myProcess
> > and moves it to .camel sub folder.We noticed that if we were having more
> > than or around 1000 files in the folder -stageDummy ,there were some
> > exceptions
> > thrown
> >
> > This exception is thrown randomly for n number of files,where n changes
> > everytime we execute our main.Note that we were not trying to access the
> > exchange
> > instead we kept our process simple.We could see that if there are 1000
> files
> > in our folder then the processor is getting called 1000+x times ,where x
> > changes everytime example it gets called 1024 times,and we could see that
> > camel is trying to process these x number of files again
> >
> > We are attaching the Main class as well as the project zip
> >
> > AND HAPPY NEW YEAR.
> >
> > hoping to resolve it soon
> >
> >
> >
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
thanks
On Thu, Jan 11, 2018 at 3:09 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Yes its a little bug, its easier to reproduce if you make the
> processing slower, and speedup the file polling consumer to scan
> faster.
>
>
>
> On Tue, Jan 9, 2018 at 8:26 AM, Claus Ibsen <cl...@gmail.com> wrote:
> > Hi
> >
> > Have patience, we will take a look at this later when we get time.
> >
> >
> >
> > On Tue, Jan 9, 2018 at 1:52 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> >> Humble request to validate my findings and confirming that it is a bug
> >>
> >> On 06-Jan-2018 5:06 PM, "Bhuvan Gupta" <bh...@gmail.com> wrote:
> >>
> >> Camel Users,
> >>
> >> I have found the issue
> >> First lets go over the code routines that are executed for a route like
> >>
> >> from("file:stageDummy")
> >> .threads(1)
> >> .process(myProcess)
> >>
> >> [1] In the above route there will be one polling thread (Let call it as
> "
> >> *pollingThread*",) which poll the route directory and fetch all the
> files
> >> in it. (link
> >> <https://github.com/apache/camel/blob/20936b173f332090c6fc5903737152
> e7d26eb60b/camel-core/src/main/java/org/apache/camel/
> component/file/FileConsumer.java#L75>
> >> )
> >> [2] Then fo reach file *pollingThread *does a
> GenericFileConsumer.isValid
> >> check , inside isValid it checks if the file is processed using
> LRUcache of
> >> size 1000 (link)
> >> <https://github.com/apache/camel/blob/bb39dc44d9603c0e6fce2460ddf35a
> 7b94ca30d4/camel-core/src/main/java/org/apache/camel/component/file/
> GenericFileConsumer.java#L541>
> >> [3] If file is Valid then *pollingThread *submit the task to
> >> executorService having 1 thread(Lets Name is it CamelThread#1)
> >> [4] Inside CamelThread#1 does following things
> >> [1] Call the processor
> >> [1] moving the file to .camel
> >> [2] clear up the .camelLock file
> >> [3] Remove the entry from LRU cache ----- IMPORTANT Point (link
> >> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4
> 610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/
> GenericFileOnCompletion.java#L95>
> >> )
> >>
> >>
> >> Now if the two thread execution in the following timeline fashion THEN
> we
> >> will end up with the same file been processed twice.
> >>
> >> [1] *pollingThread*: poll and fetch all the files in it let say we get
> >> [File1]
> >> [2] *pollingThread*: send the file for isValid Check
> >> [3] *pollingThread*: isValid Check passes for File1 as it is not
> present in
> >> LRU cahce
> >> [4] *pollingThread*: Submit the File1 exchange to CamelThread#1
> >> [5] *pollingThread*: poll and fetch all the files in it let say we get
> >> [File1, File1.camelLock]
> >> [6] *CamelThread#1*: call processor for File1 and then remove File1
> entry
> >> from LRU cache
> >> [7] *pollingThread*: send the file for isValid Check
> >> [8] *pollingThread*: isValid check passes for the File1 as it is not is
> LRU
> >> cache.
> >> NOTE: isValid checked passed for File1 twice....HENCE THE EXCEPTION
> >>
> >>
> >> Solution I see:
> >> [1] Dont remove entry from cache after onCompletion (link
> >> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4
> 610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/
> GenericFileOnCompletion.java#L95>
> >> )
> >>
> >> *Humble request to validate my findings and confirming it as a bug :)*
> >> Feel free to ask for more explanatory text if above text is not clean
> >> enough.
> >>
> >> Thanks
> >> Bhuvan
> >>
> >> On Wed, Jan 3, 2018 at 11:18 PM, <si...@yahoo.com.invalid>
> wrote:
> >>
> >>> I suspect using “File consumer + threads(…) + default readLock” is like
> >>> having multiple File consumer watching the same folder - they’ll screw
> up
> >>> each other.
> >>>
> >>>
> >>> I offer this as a possible proof: if you add an `idempotent=true` to
> your
> >>> URI, the problem should go away.
> >>>
> >>>
> >>>
> >>> p/s: I think you’re trying to process your files parallel-ly. If that
> is
> >>> your _actual_ question, I have no idea if the above is what you’re
> supposed
> >>> to do even if it works.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From: Claus Ibsen
> >>> Sent: Wednesday, 3 January, 2018 9:56 PM
> >>> To: users@camel.apache.org
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> We cannot reproduce your problem.
> >>>
> >>> There can be several reasons why a file cannot be renamed, but
> >>> unfortunately the file api just returns a boolean if it cannot rename
> >>> a file.
> >>> A typical problem is that your do some custom code where you read the
> >>> file or do something with a input stream you do not close properly.
> >>>
> >>> Also using threads(1) does not really make sense.
> >>>
> >>>
> >>>
> >>> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com>
> wrote:
> >>> > Hello nice people,
> >>> >
> >>> > We are using camel 2.17 with java 1.7 in our sample project and we
> >>> defined
> >>> > our route as:
> >>> >
> >>> > from("file:stageDummy")
> >>> > .threads(1)
> >>> > .process(myProcess);
> >>> >
> >>> > We first add 1000 file in stageDummy folder, then we run the code
> and
> >>> we
> >>> > get exception as below
> >>> >
> >>> >
> >>> > org.apache.camel.component.file.GenericFileOperationFailedException:
> >>> Cannot
> >>> > rename file: GenericFile[newFile319] to:
> GenericFile[.camel/newFile319]
> >>> > at
> >>> > org.apache.camel.component.file.strategy.GenericFileProcessS
> >>> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
> >>> > at
> >>> > org.apache.camel.component.file.strategy.GenericFileRenamePr
> >>> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> >>> > at
> >>> > org.apache.camel.component.file.GenericFileOnCompletion.proc
> >>> essStrategyCommit(GenericFileOnCompletion.java:127)
> >>> > at
> >>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
> >>> mpletion(GenericFileOnCompletion.java:83)
> >>> > at
> >>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
> >>> mplete(GenericFileOnCompletion.java:57)
> >>> > at
> >>> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
> >>> UnitOfWorkHelper.java:104)
> >>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
> >>> rk.java:229)
> >>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
> >>> per.java:65)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> >>> ProcessorAdvice.after(CamelInternalProcessor.java:654)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> >>> ProcessorAdvice.after(CamelInternalProcessor.java:609)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
> >>> llback.done(CamelInternalProcessor.java:239)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
> >>> llback.done(CamelInternalProcessor.java:250)
> >>> > at
> >>> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
> >>> eliveryErrorHandler.java:480)
> >>> > at
> >>> > org.apache.camel.management.InstrumentationProcessor$1.done(
> >>> InstrumentationProcessor.java:86)
> >>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> >>> > at
> >>> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
> >>> ThreadsProcessor.java:85)
> >>> > at java.util.concurrent.Executors$RunnableAdapter.call(
> >>> Executors.java:471)
> >>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>> > at
> >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> >>> Executor.java:1145)
> >>> > at
> >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> >>> lExecutor.java:615)
> >>> > at java.lang.Thread.run(Thread.java:745)
> >>> >
> >>> > As per our understanding camel picks up the file from the mentioned
> >>> > directory i.e in my case-stageDummy and processed by
> process-myProcess
> >>> > and moves it to .camel sub folder.We noticed that if we were having
> more
> >>> > than or around 1000 files in the folder -stageDummy ,there were some
> >>> > exceptions
> >>> > thrown
> >>> >
> >>> > This exception is thrown randomly for n number of files,where n
> changes
> >>> > everytime we execute our main.Note that we were not trying to access
> the
> >>> > exchange
> >>> > instead we kept our process simple.We could see that if there are
> 1000
> >>> files
> >>> > in our folder then the processor is getting called 1000+x times
> ,where x
> >>> > changes everytime example it gets called 1024 times,and we could see
> that
> >>> > camel is trying to process these x number of files again
> >>> >
> >>> > We are attaching the Main class as well as the project zip
> >>> >
> >>> > AND HAPPY NEW YEAR.
> >>> >
> >>> > hoping to resolve it soon
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Claus Ibsen
> >>> -----------------
> >>> http://davsclaus.com @davsclaus
> >>> Camel in Action 2: https://www.manning.com/ibsen2
> >>>
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yes its a little bug, its easier to reproduce if you make the
processing slower, and speedup the file polling consumer to scan
faster.
On Tue, Jan 9, 2018 at 8:26 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Have patience, we will take a look at this later when we get time.
>
>
>
> On Tue, Jan 9, 2018 at 1:52 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
>> Humble request to validate my findings and confirming that it is a bug
>>
>> On 06-Jan-2018 5:06 PM, "Bhuvan Gupta" <bh...@gmail.com> wrote:
>>
>> Camel Users,
>>
>> I have found the issue
>> First lets go over the code routines that are executed for a route like
>>
>> from("file:stageDummy")
>> .threads(1)
>> .process(myProcess)
>>
>> [1] In the above route there will be one polling thread (Let call it as "
>> *pollingThread*",) which poll the route directory and fetch all the files
>> in it. (link
>> <https://github.com/apache/camel/blob/20936b173f332090c6fc5903737152e7d26eb60b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java#L75>
>> )
>> [2] Then fo reach file *pollingThread *does a GenericFileConsumer.isValid
>> check , inside isValid it checks if the file is processed using LRUcache of
>> size 1000 (link)
>> <https://github.com/apache/camel/blob/bb39dc44d9603c0e6fce2460ddf35a7b94ca30d4/camel-core/src/main/java/org/apache/camel/component/file/GenericFileConsumer.java#L541>
>> [3] If file is Valid then *pollingThread *submit the task to
>> executorService having 1 thread(Lets Name is it CamelThread#1)
>> [4] Inside CamelThread#1 does following things
>> [1] Call the processor
>> [1] moving the file to .camel
>> [2] clear up the .camelLock file
>> [3] Remove the entry from LRU cache ----- IMPORTANT Point (link
>> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
>> )
>>
>>
>> Now if the two thread execution in the following timeline fashion THEN we
>> will end up with the same file been processed twice.
>>
>> [1] *pollingThread*: poll and fetch all the files in it let say we get
>> [File1]
>> [2] *pollingThread*: send the file for isValid Check
>> [3] *pollingThread*: isValid Check passes for File1 as it is not present in
>> LRU cahce
>> [4] *pollingThread*: Submit the File1 exchange to CamelThread#1
>> [5] *pollingThread*: poll and fetch all the files in it let say we get
>> [File1, File1.camelLock]
>> [6] *CamelThread#1*: call processor for File1 and then remove File1 entry
>> from LRU cache
>> [7] *pollingThread*: send the file for isValid Check
>> [8] *pollingThread*: isValid check passes for the File1 as it is not is LRU
>> cache.
>> NOTE: isValid checked passed for File1 twice....HENCE THE EXCEPTION
>>
>>
>> Solution I see:
>> [1] Dont remove entry from cache after onCompletion (link
>> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
>> )
>>
>> *Humble request to validate my findings and confirming it as a bug :)*
>> Feel free to ask for more explanatory text if above text is not clean
>> enough.
>>
>> Thanks
>> Bhuvan
>>
>> On Wed, Jan 3, 2018 at 11:18 PM, <si...@yahoo.com.invalid> wrote:
>>
>>> I suspect using “File consumer + threads(…) + default readLock” is like
>>> having multiple File consumer watching the same folder - they’ll screw up
>>> each other.
>>>
>>>
>>> I offer this as a possible proof: if you add an `idempotent=true` to your
>>> URI, the problem should go away.
>>>
>>>
>>>
>>> p/s: I think you’re trying to process your files parallel-ly. If that is
>>> your _actual_ question, I have no idea if the above is what you’re supposed
>>> to do even if it works.
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Claus Ibsen
>>> Sent: Wednesday, 3 January, 2018 9:56 PM
>>> To: users@camel.apache.org
>>>
>>>
>>>
>>>
>>>
>>> We cannot reproduce your problem.
>>>
>>> There can be several reasons why a file cannot be renamed, but
>>> unfortunately the file api just returns a boolean if it cannot rename
>>> a file.
>>> A typical problem is that your do some custom code where you read the
>>> file or do something with a input stream you do not close properly.
>>>
>>> Also using threads(1) does not really make sense.
>>>
>>>
>>>
>>> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
>>> > Hello nice people,
>>> >
>>> > We are using camel 2.17 with java 1.7 in our sample project and we
>>> defined
>>> > our route as:
>>> >
>>> > from("file:stageDummy")
>>> > .threads(1)
>>> > .process(myProcess);
>>> >
>>> > We first add 1000 file in stageDummy folder, then we run the code and
>>> we
>>> > get exception as below
>>> >
>>> >
>>> > org.apache.camel.component.file.GenericFileOperationFailedException:
>>> Cannot
>>> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
>>> > at
>>> > org.apache.camel.component.file.strategy.GenericFileProcessS
>>> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
>>> > at
>>> > org.apache.camel.component.file.strategy.GenericFileRenamePr
>>> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>>> > at
>>> > org.apache.camel.component.file.GenericFileOnCompletion.proc
>>> essStrategyCommit(GenericFileOnCompletion.java:127)
>>> > at
>>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>>> mpletion(GenericFileOnCompletion.java:83)
>>> > at
>>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>>> mplete(GenericFileOnCompletion.java:57)
>>> > at
>>> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
>>> UnitOfWorkHelper.java:104)
>>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
>>> rk.java:229)
>>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
>>> per.java:65)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>>> ProcessorAdvice.after(CamelInternalProcessor.java:654)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>>> ProcessorAdvice.after(CamelInternalProcessor.java:609)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>>> llback.done(CamelInternalProcessor.java:239)
>>> > at
>>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>>> llback.done(CamelInternalProcessor.java:250)
>>> > at
>>> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
>>> eliveryErrorHandler.java:480)
>>> > at
>>> > org.apache.camel.management.InstrumentationProcessor$1.done(
>>> InstrumentationProcessor.java:86)
>>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
>>> > at
>>> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
>>> ThreadsProcessor.java:85)
>>> > at java.util.concurrent.Executors$RunnableAdapter.call(
>>> Executors.java:471)
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1145)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:615)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> >
>>> > As per our understanding camel picks up the file from the mentioned
>>> > directory i.e in my case-stageDummy and processed by process-myProcess
>>> > and moves it to .camel sub folder.We noticed that if we were having more
>>> > than or around 1000 files in the folder -stageDummy ,there were some
>>> > exceptions
>>> > thrown
>>> >
>>> > This exception is thrown randomly for n number of files,where n changes
>>> > everytime we execute our main.Note that we were not trying to access the
>>> > exchange
>>> > instead we kept our process simple.We could see that if there are 1000
>>> files
>>> > in our folder then the processor is getting called 1000+x times ,where x
>>> > changes everytime example it gets called 1024 times,and we could see that
>>> > camel is trying to process these x number of files again
>>> >
>>> > We are attaching the Main class as well as the project zip
>>> >
>>> > AND HAPPY NEW YEAR.
>>> >
>>> > hoping to resolve it soon
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> http://davsclaus.com @davsclaus
>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel processing single file twice in 'file' endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Have patience, we will take a look at this later when we get time.
On Tue, Jan 9, 2018 at 1:52 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> Humble request to validate my findings and confirming that it is a bug
>
> On 06-Jan-2018 5:06 PM, "Bhuvan Gupta" <bh...@gmail.com> wrote:
>
> Camel Users,
>
> I have found the issue
> First lets go over the code routines that are executed for a route like
>
> from("file:stageDummy")
> .threads(1)
> .process(myProcess)
>
> [1] In the above route there will be one polling thread (Let call it as "
> *pollingThread*",) which poll the route directory and fetch all the files
> in it. (link
> <https://github.com/apache/camel/blob/20936b173f332090c6fc5903737152e7d26eb60b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java#L75>
> )
> [2] Then fo reach file *pollingThread *does a GenericFileConsumer.isValid
> check , inside isValid it checks if the file is processed using LRUcache of
> size 1000 (link)
> <https://github.com/apache/camel/blob/bb39dc44d9603c0e6fce2460ddf35a7b94ca30d4/camel-core/src/main/java/org/apache/camel/component/file/GenericFileConsumer.java#L541>
> [3] If file is Valid then *pollingThread *submit the task to
> executorService having 1 thread(Lets Name is it CamelThread#1)
> [4] Inside CamelThread#1 does following things
> [1] Call the processor
> [1] moving the file to .camel
> [2] clear up the .camelLock file
> [3] Remove the entry from LRU cache ----- IMPORTANT Point (link
> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
> )
>
>
> Now if the two thread execution in the following timeline fashion THEN we
> will end up with the same file been processed twice.
>
> [1] *pollingThread*: poll and fetch all the files in it let say we get
> [File1]
> [2] *pollingThread*: send the file for isValid Check
> [3] *pollingThread*: isValid Check passes for File1 as it is not present in
> LRU cahce
> [4] *pollingThread*: Submit the File1 exchange to CamelThread#1
> [5] *pollingThread*: poll and fetch all the files in it let say we get
> [File1, File1.camelLock]
> [6] *CamelThread#1*: call processor for File1 and then remove File1 entry
> from LRU cache
> [7] *pollingThread*: send the file for isValid Check
> [8] *pollingThread*: isValid check passes for the File1 as it is not is LRU
> cache.
> NOTE: isValid checked passed for File1 twice....HENCE THE EXCEPTION
>
>
> Solution I see:
> [1] Dont remove entry from cache after onCompletion (link
> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
> )
>
> *Humble request to validate my findings and confirming it as a bug :)*
> Feel free to ask for more explanatory text if above text is not clean
> enough.
>
> Thanks
> Bhuvan
>
> On Wed, Jan 3, 2018 at 11:18 PM, <si...@yahoo.com.invalid> wrote:
>
>> I suspect using “File consumer + threads(…) + default readLock” is like
>> having multiple File consumer watching the same folder - they’ll screw up
>> each other.
>>
>>
>> I offer this as a possible proof: if you add an `idempotent=true` to your
>> URI, the problem should go away.
>>
>>
>>
>> p/s: I think you’re trying to process your files parallel-ly. If that is
>> your _actual_ question, I have no idea if the above is what you’re supposed
>> to do even if it works.
>>
>>
>>
>>
>>
>>
>> From: Claus Ibsen
>> Sent: Wednesday, 3 January, 2018 9:56 PM
>> To: users@camel.apache.org
>>
>>
>>
>>
>>
>> We cannot reproduce your problem.
>>
>> There can be several reasons why a file cannot be renamed, but
>> unfortunately the file api just returns a boolean if it cannot rename
>> a file.
>> A typical problem is that your do some custom code where you read the
>> file or do something with a input stream you do not close properly.
>>
>> Also using threads(1) does not really make sense.
>>
>>
>>
>> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
>> > Hello nice people,
>> >
>> > We are using camel 2.17 with java 1.7 in our sample project and we
>> defined
>> > our route as:
>> >
>> > from("file:stageDummy")
>> > .threads(1)
>> > .process(myProcess);
>> >
>> > We first add 1000 file in stageDummy folder, then we run the code and
>> we
>> > get exception as below
>> >
>> >
>> > org.apache.camel.component.file.GenericFileOperationFailedException:
>> Cannot
>> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
>> > at
>> > org.apache.camel.component.file.strategy.GenericFileProcessS
>> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
>> > at
>> > org.apache.camel.component.file.strategy.GenericFileRenamePr
>> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.proc
>> essStrategyCommit(GenericFileOnCompletion.java:127)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> mpletion(GenericFileOnCompletion.java:83)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> mplete(GenericFileOnCompletion.java:57)
>> > at
>> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
>> UnitOfWorkHelper.java:104)
>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
>> rk.java:229)
>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
>> per.java:65)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> ProcessorAdvice.after(CamelInternalProcessor.java:654)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> ProcessorAdvice.after(CamelInternalProcessor.java:609)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> llback.done(CamelInternalProcessor.java:239)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> llback.done(CamelInternalProcessor.java:250)
>> > at
>> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
>> eliveryErrorHandler.java:480)
>> > at
>> > org.apache.camel.management.InstrumentationProcessor$1.done(
>> InstrumentationProcessor.java:86)
>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
>> > at
>> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
>> ThreadsProcessor.java:85)
>> > at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > As per our understanding camel picks up the file from the mentioned
>> > directory i.e in my case-stageDummy and processed by process-myProcess
>> > and moves it to .camel sub folder.We noticed that if we were having more
>> > than or around 1000 files in the folder -stageDummy ,there were some
>> > exceptions
>> > thrown
>> >
>> > This exception is thrown randomly for n number of files,where n changes
>> > everytime we execute our main.Note that we were not trying to access the
>> > exchange
>> > instead we kept our process simple.We could see that if there are 1000
>> files
>> > in our folder then the processor is getting called 1000+x times ,where x
>> > changes everytime example it gets called 1024 times,and we could see that
>> > camel is trying to process these x number of files again
>> >
>> > We are attaching the Main class as well as the project zip
>> >
>> > AND HAPPY NEW YEAR.
>> >
>> > hoping to resolve it soon
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
Humble request to validate my findings and confirming that it is a bug
On 06-Jan-2018 5:06 PM, "Bhuvan Gupta" <bh...@gmail.com> wrote:
Camel Users,
I have found the issue
First lets go over the code routines that are executed for a route like
from("file:stageDummy")
.threads(1)
.process(myProcess)
[1] In the above route there will be one polling thread (Let call it as "
*pollingThread*",) which poll the route directory and fetch all the files
in it. (link
<https://github.com/apache/camel/blob/20936b173f332090c6fc5903737152e7d26eb60b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java#L75>
)
[2] Then fo reach file *pollingThread *does a GenericFileConsumer.isValid
check , inside isValid it checks if the file is processed using LRUcache of
size 1000 (link)
<https://github.com/apache/camel/blob/bb39dc44d9603c0e6fce2460ddf35a7b94ca30d4/camel-core/src/main/java/org/apache/camel/component/file/GenericFileConsumer.java#L541>
[3] If file is Valid then *pollingThread *submit the task to
executorService having 1 thread(Lets Name is it CamelThread#1)
[4] Inside CamelThread#1 does following things
[1] Call the processor
[1] moving the file to .camel
[2] clear up the .camelLock file
[3] Remove the entry from LRU cache ----- IMPORTANT Point (link
<https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
)
Now if the two thread execution in the following timeline fashion THEN we
will end up with the same file been processed twice.
[1] *pollingThread*: poll and fetch all the files in it let say we get
[File1]
[2] *pollingThread*: send the file for isValid Check
[3] *pollingThread*: isValid Check passes for File1 as it is not present in
LRU cahce
[4] *pollingThread*: Submit the File1 exchange to CamelThread#1
[5] *pollingThread*: poll and fetch all the files in it let say we get
[File1, File1.camelLock]
[6] *CamelThread#1*: call processor for File1 and then remove File1 entry
from LRU cache
[7] *pollingThread*: send the file for isValid Check
[8] *pollingThread*: isValid check passes for the File1 as it is not is LRU
cache.
NOTE: isValid checked passed for File1 twice....HENCE THE EXCEPTION
Solution I see:
[1] Dont remove entry from cache after onCompletion (link
<https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
)
*Humble request to validate my findings and confirming it as a bug :)*
Feel free to ask for more explanatory text if above text is not clean
enough.
Thanks
Bhuvan
On Wed, Jan 3, 2018 at 11:18 PM, <si...@yahoo.com.invalid> wrote:
> I suspect using “File consumer + threads(…) + default readLock” is like
> having multiple File consumer watching the same folder - they’ll screw up
> each other.
>
>
> I offer this as a possible proof: if you add an `idempotent=true` to your
> URI, the problem should go away.
>
>
>
> p/s: I think you’re trying to process your files parallel-ly. If that is
> your _actual_ question, I have no idea if the above is what you’re supposed
> to do even if it works.
>
>
>
>
>
>
> From: Claus Ibsen
> Sent: Wednesday, 3 January, 2018 9:56 PM
> To: users@camel.apache.org
>
>
>
>
>
> We cannot reproduce your problem.
>
> There can be several reasons why a file cannot be renamed, but
> unfortunately the file api just returns a boolean if it cannot rename
> a file.
> A typical problem is that your do some custom code where you read the
> file or do something with a input stream you do not close properly.
>
> Also using threads(1) does not really make sense.
>
>
>
> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> > Hello nice people,
> >
> > We are using camel 2.17 with java 1.7 in our sample project and we
> defined
> > our route as:
> >
> > from("file:stageDummy")
> > .threads(1)
> > .process(myProcess);
> >
> > We first add 1000 file in stageDummy folder, then we run the code and
> we
> > get exception as below
> >
> >
> > org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot
> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
> > at
> > org.apache.camel.component.file.strategy.GenericFileProcessS
> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
> > at
> > org.apache.camel.component.file.strategy.GenericFileRenamePr
> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.proc
> essStrategyCommit(GenericFileOnCompletion.java:127)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
> mpletion(GenericFileOnCompletion.java:83)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
> mplete(GenericFileOnCompletion.java:57)
> > at
> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
> UnitOfWorkHelper.java:104)
> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
> rk.java:229)
> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
> per.java:65)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> ProcessorAdvice.after(CamelInternalProcessor.java:654)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> ProcessorAdvice.after(CamelInternalProcessor.java:609)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
> llback.done(CamelInternalProcessor.java:239)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
> llback.done(CamelInternalProcessor.java:250)
> > at
> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
> eliveryErrorHandler.java:480)
> > at
> > org.apache.camel.management.InstrumentationProcessor$1.done(
> InstrumentationProcessor.java:86)
> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> > at
> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
> ThreadsProcessor.java:85)
> > at java.util.concurrent.Executors$RunnableAdapter.call(
> Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > As per our understanding camel picks up the file from the mentioned
> > directory i.e in my case-stageDummy and processed by process-myProcess
> > and moves it to .camel sub folder.We noticed that if we were having more
> > than or around 1000 files in the folder -stageDummy ,there were some
> > exceptions
> > thrown
> >
> > This exception is thrown randomly for n number of files,where n changes
> > everytime we execute our main.Note that we were not trying to access the
> > exchange
> > instead we kept our process simple.We could see that if there are 1000
> files
> > in our folder then the processor is getting called 1000+x times ,where x
> > changes everytime example it gets called 1024 times,and we could see that
> > camel is trying to process these x number of files again
> >
> > We are attaching the Main class as well as the project zip
> >
> > AND HAPPY NEW YEAR.
> >
> > hoping to resolve it soon
> >
> >
> >
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Bhuvan Gupta <bh...@gmail.com>.
Camel Users,
I have found the issue
First lets go over the code routines that are executed for a route like
from("file:stageDummy")
.threads(1)
.process(myProcess)
[1] In the above route there will be one polling thread (Let call it as "
*pollingThread*",) which poll the route directory and fetch all the files
in it. (link
<https://github.com/apache/camel/blob/20936b173f332090c6fc5903737152e7d26eb60b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java#L75>
)
[2] Then fo reach file *pollingThread *does a GenericFileConsumer.isValid
check , inside isValid it checks if the file is processed using LRUcache of
size 1000 (link)
<https://github.com/apache/camel/blob/bb39dc44d9603c0e6fce2460ddf35a7b94ca30d4/camel-core/src/main/java/org/apache/camel/component/file/GenericFileConsumer.java#L541>
[3] If file is Valid then *pollingThread *submit the task to
executorService having 1 thread(Lets Name is it CamelThread#1)
[4] Inside CamelThread#1 does following things
[1] Call the processor
[1] moving the file to .camel
[2] clear up the .camelLock file
[3] Remove the entry from LRU cache ----- IMPORTANT Point (link
<https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
)
Now if the two thread execution in the following timeline fashion THEN we
will end up with the same file been processed twice.
[1] *pollingThread*: poll and fetch all the files in it let say we get
[File1]
[2] *pollingThread*: send the file for isValid Check
[3] *pollingThread*: isValid Check passes for File1 as it is not present in
LRU cahce
[4] *pollingThread*: Submit the File1 exchange to CamelThread#1
[5] *pollingThread*: poll and fetch all the files in it let say we get
[File1, File1.camelLock]
[6] *CamelThread#1*: call processor for File1 and then remove File1 entry
from LRU cache
[7] *pollingThread*: send the file for isValid Check
[8] *pollingThread*: isValid check passes for the File1 as it is not is LRU
cache.
NOTE: isValid checked passed for File1 twice....HENCE THE EXCEPTION
Solution I see:
[1] Dont remove entry from cache after onCompletion (link
<https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
)
*Humble request to validate my findings and confirming it as a bug :)*
Feel free to ask for more explanatory text if above text is not clean
enough.
Thanks
Bhuvan
On Wed, Jan 3, 2018 at 11:18 PM, <si...@yahoo.com.invalid> wrote:
> I suspect using “File consumer + threads(…) + default readLock” is like
> having multiple File consumer watching the same folder - they’ll screw up
> each other.
>
>
> I offer this as a possible proof: if you add an `idempotent=true` to your
> URI, the problem should go away.
>
>
>
> p/s: I think you’re trying to process your files parallel-ly. If that is
> your _actual_ question, I have no idea if the above is what you’re supposed
> to do even if it works.
>
>
>
>
>
>
> From: Claus Ibsen
> Sent: Wednesday, 3 January, 2018 9:56 PM
> To: users@camel.apache.org
>
>
>
>
>
> We cannot reproduce your problem.
>
> There can be several reasons why a file cannot be renamed, but
> unfortunately the file api just returns a boolean if it cannot rename
> a file.
> A typical problem is that your do some custom code where you read the
> file or do something with a input stream you do not close properly.
>
> Also using threads(1) does not really make sense.
>
>
>
> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> > Hello nice people,
> >
> > We are using camel 2.17 with java 1.7 in our sample project and we
> defined
> > our route as:
> >
> > from("file:stageDummy")
> > .threads(1)
> > .process(myProcess);
> >
> > We first add 1000 file in stageDummy folder, then we run the code and
> we
> > get exception as below
> >
> >
> > org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot
> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
> > at
> > org.apache.camel.component.file.strategy.GenericFileProcessStrategySupp
> ort.renameFile(GenericFileProcessStrategySupport.java:115)
> > at
> > org.apache.camel.component.file.strategy.GenericFileRenameProcessStrate
> gy.commit(GenericFileRenameProcessStrategy.java:88)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.
> processStrategyCommit(GenericFileOnCompletion.java:127)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(
> GenericFileOnCompletion.java:83)
> > at
> > org.apache.camel.component.file.GenericFileOnCompletion.onComplete(
> GenericFileOnCompletion.java:57)
> > at
> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
> UnitOfWorkHelper.java:104)
> > at org.apache.camel.impl.DefaultUnitOfWork.done(
> DefaultUnitOfWork.java:229)
> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(
> UnitOfWorkHelper.java:65)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$
> UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:654)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$
> UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:609)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(
> CamelInternalProcessor.java:239)
> > at
> > org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(
> CamelInternalProcessor.java:250)
> > at
> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(
> RedeliveryErrorHandler.java:480)
> > at
> > org.apache.camel.management.InstrumentationProcessor$1.
> done(InstrumentationProcessor.java:86)
> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> > at
> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.
> run(ThreadsProcessor.java:85)
> > at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > As per our understanding camel picks up the file from the mentioned
> > directory i.e in my case-stageDummy and processed by process-myProcess
> > and moves it to .camel sub folder.We noticed that if we were having more
> > than or around 1000 files in the folder -stageDummy ,there were some
> > exceptions
> > thrown
> >
> > This exception is thrown randomly for n number of files,where n changes
> > everytime we execute our main.Note that we were not trying to access the
> > exchange
> > instead we kept our process simple.We could see that if there are 1000
> files
> > in our folder then the processor is getting called 1000+x times ,where x
> > changes everytime example it gets called 1024 times,and we could see that
> > camel is trying to process these x number of files again
> >
> > We are attaching the Main class as well as the project zip
> >
> > AND HAPPY NEW YEAR.
> >
> > hoping to resolve it soon
> >
> >
> >
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Re: Camel processing single file twice in 'file' endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
We cannot reproduce your problem.
There can be several reasons why a file cannot be renamed, but
unfortunately the file api just returns a boolean if it cannot rename
a file.
A typical problem is that your do some custom code where you read the
file or do something with a input stream you do not close properly.
Also using threads(1) does not really make sense.
On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bh...@gmail.com> wrote:
> Hello nice people,
>
> We are using camel 2.17 with java 1.7 in our sample project and we defined
> our route as:
>
> from("file:stageDummy")
> .threads(1)
> .process(myProcess);
>
> We first add 1000 file in stageDummy folder, then we run the code and we
> get exception as below
>
>
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
> rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
> at
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
> at
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
> at
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
> at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:654)
> at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:609)
> at
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
> at
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:250)
> at
> org.apache.camel.processor.RedeliveryErrorHandler$1.done(RedeliveryErrorHandler.java:480)
> at
> org.apache.camel.management.InstrumentationProcessor$1.done(InstrumentationProcessor.java:86)
> at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> at
> org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:85)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> As per our understanding camel picks up the file from the mentioned
> directory i.e in my case-stageDummy and processed by process-myProcess
> and moves it to .camel sub folder.We noticed that if we were having more
> than or around 1000 files in the folder -stageDummy ,there were some
> exceptions
> thrown
>
> This exception is thrown randomly for n number of files,where n changes
> everytime we execute our main.Note that we were not trying to access the
> exchange
> instead we kept our process simple.We could see that if there are 1000 files
> in our folder then the processor is getting called 1000+x times ,where x
> changes everytime example it gets called 1024 times,and we could see that
> camel is trying to process these x number of files again
>
> We are attaching the Main class as well as the project zip
>
> AND HAPPY NEW YEAR.
>
> hoping to resolve it soon
>
>
>
>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2