You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by OrackBahama <jd...@metadok.de> on 2012/09/17 18:18:05 UTC
File renaming problems under Windows
Hi,
I'm getting exceptions when polling and processing files (in the same
filesystem). The last camel step is moving the file into a backup or into an
error directory using the parameters 'move' and 'moveFailed' of the file
component.
I'm aware of renaming problems if there's a different filesystem or
concurrent consumers trying to read the file, but I'm worried that this
happens under relatively simple circumstances (Java 6, Camel 2.8.1). Do I
have to use Java 7 for better native filesystem locking behavior ? Or can I
configure several retries ?
This is the exception that occurs - thanks for any recommendation in advance
!
2012-09-15 10:26:13 ERROR : CamelLogger file.GenericFileOnCompletion :
Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException - Error
renaming file from C:\in\00800530.txt to C:\error\2012-09-15\00800530.txt]
org.apache.camel.component.file.GenericFileOperationFailedException: Error
renaming file from C:\in\00800530.txt to C:\error\2012-09-15\00800530.txt
at
org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:68)
at
org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:96)
at
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.rollback(GenericFileRenameProcessStrategy.java:68)
at
org.apache.camel.component.file.GenericFileOnCompletion.processStrategyRollback(GenericFileOnCompletion.java:154)
at
org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:89)
at
org.apache.camel.component.file.GenericFileOnCompletion.onFailure(GenericFileOnCompletion.java:58)
at
org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:53)
at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:208)
at
org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:188)
at
org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:37)
at
org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:146)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:105)
at
org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:139)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:106)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69)
at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:353)
at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:176)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:137)
at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:138)
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:90)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Renaming file from: C:\in\00800530.txt to:
C:\error\2012-09-15\00800530.txt failed due cannot delete from file:
C:\in\00800530.txt after copy succeeded
at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:379)
at
org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:66)
... 31 more
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by bocamel <jo...@gmail.com>.
Christian,
I tested this under Camel 2.12.0 and the issue that I reported did not
occur. Thank you very much for fixing it!
FYI. This is the first time I tried Camel 2.12.0. Eclipse complained about
" The method getTextContent() is undefined for the type Node”
(org.w3c.dom.Node) unless I move JDK ahead of my Maven dependencies in the
buildpath. The only thing changed in the Maven dependencies is Camel
ungraded from 2.11.1 to 2.12.0.
Thanks!
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5739065.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by bocamel <jo...@gmail.com>.
Done... JIRA Camel-6756 was opened.
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5739555.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Sep 12, 2013 at 11:06 PM, bocamel <jo...@gmail.com> wrote:
> Christian,
>
> I am afraid my celebration of the fix might be a little premature. :-).
> This failure of renaming/deleting a consumed file error still occurs when I
> forced a org.apache.camel.component.bean.AmbiguousMethodCallException in the
> route. In other words, if the route receives a file from file consumer and
> sends it to a bean (having similar methods) that may result the above
> exception, then the File component cannot delete/rename the file. I have
> been able to reproduce this issue consistently.
>
Thanks for reporting. Please feel free to log a JIRA ticket.
> Unfortunately in this case, since I am processing Excel files (binary
> format), I cannot try to convert the file into String first to see whether
> the problem will go away.
>
> Here is the route:
>
> <camel:route id="route-process-row">
> <camel:from uri="file:src/data/rowfile" />
> <camel:to uri="bean:converter" />
> <camel:onException>
> <camel:exception>org.apache.camel.component.bean.AmbiguousMethodCallException</camel:exception>
> <camel:handled><constant>true</constant></camel:handled>
> <camel:log message="*** Exception caught ***" />
> </camel:onException>
> <camel:onException>
> <camel:exception>java.lang.RuntimeException</camel:exception>
> <camel:handled><constant>true</constant></camel:handled>
> <camel:log message="*** Exception caught ***" />
> </camel:onException>
> </camel:route>
>
> The two methods of bean:converter are:
> public ExternalDoc processRowExcelDoc(@Body InputStream body,
> @Header("CamelFileNameOnly") String fn)
> and
> public ExternalDoc processColExcelDoc(@Body InputStream body,
> @Header("CamelFileNameOnly") String fn)
>
> Thanks for any help you may provide!
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5739283.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: File renaming problems under Windows
Posted by bocamel <jo...@gmail.com>.
Christian,
I am afraid my celebration of the fix might be a little premature. :-).
This failure of renaming/deleting a consumed file error still occurs when I
forced a org.apache.camel.component.bean.AmbiguousMethodCallException in the
route. In other words, if the route receives a file from file consumer and
sends it to a bean (having similar methods) that may result the above
exception, then the File component cannot delete/rename the file. I have
been able to reproduce this issue consistently.
Unfortunately in this case, since I am processing Excel files (binary
format), I cannot try to convert the file into String first to see whether
the problem will go away.
Here is the route:
<camel:route id="route-process-row">
<camel:from uri="file:src/data/rowfile" />
<camel:to uri="bean:converter" />
<camel:onException>
<camel:exception>org.apache.camel.component.bean.AmbiguousMethodCallException</camel:exception>
<camel:handled><constant>true</constant></camel:handled>
<camel:log message="*** Exception caught ***" />
</camel:onException>
<camel:onException>
<camel:exception>java.lang.RuntimeException</camel:exception>
<camel:handled><constant>true</constant></camel:handled>
<camel:log message="*** Exception caught ***" />
</camel:onException>
</camel:route>
The two methods of bean:converter are:
public ExternalDoc processRowExcelDoc(@Body InputStream body,
@Header("CamelFileNameOnly") String fn)
and
public ExternalDoc processColExcelDoc(@Body InputStream body,
@Header("CamelFileNameOnly") String fn)
Thanks for any help you may provide!
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5739283.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by Christian Müller <ch...@gmail.com>.
I think we fixed an issue in Camel 2.12.0 which could be related to this.
Can you give it a try? Camel 2.12.0 was released one day ago...
Best,
Christian
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Thu, Sep 5, 2013 at 7:34 PM, bocamel <jo...@gmail.com> wrote:
> I ran into a similar problem. My use case is slightly different and I am
> using Camel 2.11.1 and JDK 1.7. But the result is that Camel cannot delete
> the file from the input folder and keeps processing it indefinitely.
>
> I have two routes. The first route reads in XML files from a directory
> then
> puts them on SEDA. The second route retrieves the files from SEDA and sends
> them to a web services (CXF).
>
> <camel:route id="CustomerXmlFileIn" >
> <camel:from uri="file:/xmlInput" />
> <camel:to uri="seda:processCustomerInput" />
> </camel:route>
>
> and
>
> <camel:route id="ProcessCustomerMessage"
> errorHandlerRef="DeadLetterErrorHandler">
> <camel:from uri="seda:processCustomerInput" />
> <camel:to uri="cxf:bean:webSvcEndpoint" />
> <camel:onException>
>
> <camel:exception>java.lang.Exception</camel:exception>
>
> <camel:handled><camel:constant>true</camel:constant></camel:handled>
> <camel:log message="*** Error occurred
> processing file
> ${headers.CamelFileName}. ***" />
> <camel:log
> message="${property.CamelExceptionCaught}" />
> </camel:onException>
> </camel:route>
>
> Everything works fine until a non-XML file (bad.xml) is thrown into the
> xmlInput folder. The CXF endpoint throws the following exception. When
> this happens, route1 cannot delete the bad.xml file from the xmlInput.
> Camel keeps picking it up and the same error repeats indefinitely.
>
> org.apache.camel.InvalidPayloadException: No body available of type:
> org.apache.camel.component.cxf.CxfPayload but has value:
> GenericFile[M:\bad.xml] of type:
> org.apache.camel.component.file.GenericFile
> on: null. Caused by: Error during type conversion from type:
> org.apache.camel.component.file.GenericFile to the required type:
> org.apache.camel.component.cxf.CxfPayload with value
> GenericFile[M:\bad.xml]
> due org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 2; Content
> is not allowed in prolog.. Exchange[null]. Caused by:
> [org.apache.camel.TypeConversionException - Error during type conversion
> from type: org.apache.camel.component.file.GenericFile to the required
> type:
> org.apache.camel.component.cxf.CxfPayload with value
> GenericFile[M:\bad.xml]
> due org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 2; Content
> is not allowed in prolog.]
>
> It is followed by the file deletion error:
>
> 2013-09-05 13:01:53,469 [ssCustomerInput] WARN GenericFileOnCompletion
> - Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Error
> renaming file from M:\bad.xml to M:\.camel\bad.xml]
> org.apache.camel.component.file.GenericFileOperationFailedException: Error
> renaming file from M:\bad.xml to M:\.camel\bad.xml
> at
>
> org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:72)
> at
>
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
> at
>
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> at
>
> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:144)
> at
>
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:82)
> at
>
> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:53)
> at
>
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
> at
> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:226)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:199)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:37)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:157)
> at
>
> org.apache.camel.processor.RouteContextProcessor$1.done(RouteContextProcessor.java:56)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
> at
>
> org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:46)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
> at
>
> org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
> at
>
> org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:294)
> at
> org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:203)
> at
> org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:150)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Renaming file from: M:\bad.xml to:
> M:\.camel\bad.xml failed due cannot delete from file: M:\bad.xml after copy
> succeeded
> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:397)
> at
>
> org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:70)
> ... 29 more
>
> I can avoid this problem by first converting the body to String in route1
> before sending it to SEDA. But if this is something can be fixed once for
> all in Camel/CXF, it will make Camel more robust.
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5738801.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: File renaming problems under Windows
Posted by bocamel <jo...@gmail.com>.
I ran into a similar problem. My use case is slightly different and I am
using Camel 2.11.1 and JDK 1.7. But the result is that Camel cannot delete
the file from the input folder and keeps processing it indefinitely.
I have two routes. The first route reads in XML files from a directory then
puts them on SEDA. The second route retrieves the files from SEDA and sends
them to a web services (CXF).
<camel:route id="CustomerXmlFileIn" >
<camel:from uri="file:/xmlInput" />
<camel:to uri="seda:processCustomerInput" />
</camel:route>
and
<camel:route id="ProcessCustomerMessage"
errorHandlerRef="DeadLetterErrorHandler">
<camel:from uri="seda:processCustomerInput" />
<camel:to uri="cxf:bean:webSvcEndpoint" />
<camel:onException>
<camel:exception>java.lang.Exception</camel:exception>
<camel:handled><camel:constant>true</camel:constant></camel:handled>
<camel:log message="*** Error occurred processing file
${headers.CamelFileName}. ***" />
<camel:log message="${property.CamelExceptionCaught}" />
</camel:onException>
</camel:route>
Everything works fine until a non-XML file (bad.xml) is thrown into the
xmlInput folder. The CXF endpoint throws the following exception. When
this happens, route1 cannot delete the bad.xml file from the xmlInput.
Camel keeps picking it up and the same error repeats indefinitely.
org.apache.camel.InvalidPayloadException: No body available of type:
org.apache.camel.component.cxf.CxfPayload but has value:
GenericFile[M:\bad.xml] of type: org.apache.camel.component.file.GenericFile
on: null. Caused by: Error during type conversion from type:
org.apache.camel.component.file.GenericFile to the required type:
org.apache.camel.component.cxf.CxfPayload with value GenericFile[M:\bad.xml]
due org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 2; Content
is not allowed in prolog.. Exchange[null]. Caused by:
[org.apache.camel.TypeConversionException - Error during type conversion
from type: org.apache.camel.component.file.GenericFile to the required type:
org.apache.camel.component.cxf.CxfPayload with value GenericFile[M:\bad.xml]
due org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 2; Content
is not allowed in prolog.]
It is followed by the file deletion error:
2013-09-05 13:01:53,469 [ssCustomerInput] WARN GenericFileOnCompletion
- Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException - Error
renaming file from M:\bad.xml to M:\.camel\bad.xml]
org.apache.camel.component.file.GenericFileOperationFailedException: Error
renaming file from M:\bad.xml to M:\.camel\bad.xml
at
org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:72)
at
org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
at
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
at
org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:144)
at
org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:82)
at
org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:53)
at
org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:226)
at
org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:199)
at
org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:37)
at
org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:157)
at
org.apache.camel.processor.RouteContextProcessor$1.done(RouteContextProcessor.java:56)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:46)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
at
org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:294)
at
org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:203)
at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Renaming file from: M:\bad.xml to:
M:\.camel\bad.xml failed due cannot delete from file: M:\bad.xml after copy
succeeded
at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:397)
at
org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:70)
... 29 more
I can avoid this problem by first converting the body to String in route1
before sending it to SEDA. But if this is something can be fixed once for
all in Camel/CXF, it will make Camel more robust.
Thanks!
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5738801.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by Christian Müller <ch...@gmail.com>.
I think Claus means the Camel code is "as safe as it could be". If the
source is not closed properly, that it's an issue in the JDK.
It's more likely an issue in the Smooks code. Did you already checked the
code?
Best,
Christian
On Mon, Sep 24, 2012 at 4:54 PM, OrackBahama <jd...@metadok.de> wrote:
> Hi Claus,
> I'm not quite sure if I understand you right - you suspect Smooks for
> eventually not closing the InputStream or Reader internally used by
> StreamSource ?
>
> But that would be a systematic error or leak that happens to *all*
> processed
> files, right ?
> On the other hand I'm getting the rename exception at irregular intervals
> (let's say once in a week or two / import route permanently polling)...
>
> Would it be possible that you give me an example for safely converting a
> file to StreamSource and calling Smooks in a route ?
> Do I need a special errorhandler or .try()/.catch() command in my route ?
>
> Thanks in advance.
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719862.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Re: File renaming problems under Windows
Posted by OrackBahama <jd...@metadok.de>.
Hi Claus,
I'm not quite sure if I understand you right - you suspect Smooks for
eventually not closing the InputStream or Reader internally used by
StreamSource ?
But that would be a systematic error or leak that happens to *all* processed
files, right ?
On the other hand I'm getting the rename exception at irregular intervals
(let's say once in a week or two / import route permanently polling)...
Would it be possible that you give me an example for safely converting a
file to StreamSource and calling Smooks in a route ?
Do I need a special errorhandler or .try()/.catch() command in my route ?
Thanks in advance.
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719862.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Sep 19, 2012 at 6:48 PM, Christian Müller
<ch...@gmail.com> wrote:
> May be Smooks doesn't close the stream in a propper way?
>
Yeah as well double check the Camel converter for File -> StreamSource.
Though its using this code
@Converter
public StreamSource toStreamSource(File in) throws TransformerException {
if (in != null) {
return new StreamSource(in);
}
return null;
}
Which uses the API directly.
> Sent from a mobile device
> Am 19.09.2012 11:00 schrieb "OrackBahama" <jd...@metadok.de>:
>
>> Thanks Mohan, I fully agree with you - besides one point:
>>
>> Always checking the return value of the renameTo(...) function call is of
>> no
>> use if you can't rely on Windows having completed, flushed and unlocked
>> everything to disk before returning this value.
>>
>> My experience is that even if the return value is TRUE, the underlying
>> Windows rename operation (or the unlocking) still is incomplete.
>>
>> The correct call should have an operating system dependent option such as
>> "renameToEx(..., WRITE_THROUGH=TRUE)" or
>> "renameToEx(...,
>> TELL_WINDOWS_TO_NOT_RETURN_UNLESS_EVERYTHING_IS_FLUSHED_AND_UNLOCKED)"
>>
>> Unfortunately - making heavy use of frameworks (Camel ->
>> org.apache.commons...FileUtils, Smooks, ..) - I have no direct influence on
>> what happens with IO calls, retry counts, close operations e.g. ...)
>>
>> My camel route is as simple as this:
>>
>> from( "file://{{w.inbox}}?"
>> + "initialDelay={{w.polling.init.msec}}" //15000
>> + "&delay={{w.polling.msec}}" //25000
>> + "&preMove={{w.inbox.pre}}"
>> + "&move={{w.backup}}/${date:now:yyyy-MM-dd}/${file:onlyname}"
>> + "&moveFailed={{w.error}}/${date:now:yyyy-MM-dd}/${file:onlyname}"
>> + "&maxMessagesPerPoll={{w.inbox.maxMsgPerPoll}}" //10
>> )
>> .log( LoggingLevel.INFO, "imp_protocol", "Importiere Daten :
>> ${file:onlyname}" )
>> .convertBodyTo( StreamSource.class )
>> .to( "smooks://smooks-config.xml" )
>> ;
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719581.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: File renaming problems under Windows
Posted by Christian Müller <ch...@gmail.com>.
May be Smooks doesn't close the stream in a propper way?
Sent from a mobile device
Am 19.09.2012 11:00 schrieb "OrackBahama" <jd...@metadok.de>:
> Thanks Mohan, I fully agree with you - besides one point:
>
> Always checking the return value of the renameTo(...) function call is of
> no
> use if you can't rely on Windows having completed, flushed and unlocked
> everything to disk before returning this value.
>
> My experience is that even if the return value is TRUE, the underlying
> Windows rename operation (or the unlocking) still is incomplete.
>
> The correct call should have an operating system dependent option such as
> "renameToEx(..., WRITE_THROUGH=TRUE)" or
> "renameToEx(...,
> TELL_WINDOWS_TO_NOT_RETURN_UNLESS_EVERYTHING_IS_FLUSHED_AND_UNLOCKED)"
>
> Unfortunately - making heavy use of frameworks (Camel ->
> org.apache.commons...FileUtils, Smooks, ..) - I have no direct influence on
> what happens with IO calls, retry counts, close operations e.g. ...)
>
> My camel route is as simple as this:
>
> from( "file://{{w.inbox}}?"
> + "initialDelay={{w.polling.init.msec}}" //15000
> + "&delay={{w.polling.msec}}" //25000
> + "&preMove={{w.inbox.pre}}"
> + "&move={{w.backup}}/${date:now:yyyy-MM-dd}/${file:onlyname}"
> + "&moveFailed={{w.error}}/${date:now:yyyy-MM-dd}/${file:onlyname}"
> + "&maxMessagesPerPoll={{w.inbox.maxMsgPerPoll}}" //10
> )
> .log( LoggingLevel.INFO, "imp_protocol", "Importiere Daten :
> ${file:onlyname}" )
> .convertBodyTo( StreamSource.class )
> .to( "smooks://smooks-config.xml" )
> ;
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719581.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: File renaming problems under Windows
Posted by OrackBahama <jd...@metadok.de>.
Thanks Mohan, I fully agree with you - besides one point:
Always checking the return value of the renameTo(...) function call is of no
use if you can't rely on Windows having completed, flushed and unlocked
everything to disk before returning this value.
My experience is that even if the return value is TRUE, the underlying
Windows rename operation (or the unlocking) still is incomplete.
The correct call should have an operating system dependent option such as
"renameToEx(..., WRITE_THROUGH=TRUE)" or
"renameToEx(...,
TELL_WINDOWS_TO_NOT_RETURN_UNLESS_EVERYTHING_IS_FLUSHED_AND_UNLOCKED)"
Unfortunately - making heavy use of frameworks (Camel ->
org.apache.commons...FileUtils, Smooks, ..) - I have no direct influence on
what happens with IO calls, retry counts, close operations e.g. ...)
My camel route is as simple as this:
from( "file://{{w.inbox}}?"
+ "initialDelay={{w.polling.init.msec}}" //15000
+ "&delay={{w.polling.msec}}" //25000
+ "&preMove={{w.inbox.pre}}"
+ "&move={{w.backup}}/${date:now:yyyy-MM-dd}/${file:onlyname}"
+ "&moveFailed={{w.error}}/${date:now:yyyy-MM-dd}/${file:onlyname}"
+ "&maxMessagesPerPoll={{w.inbox.maxMsgPerPoll}}" //10
)
.log( LoggingLevel.INFO, "imp_protocol", "Importiere Daten :
${file:onlyname}" )
.convertBodyTo( StreamSource.class )
.to( "smooks://smooks-config.xml" )
;
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719581.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by projectnash1994 <pr...@gmail.com>.
Hi,
I don't have any solution, but would like to add a note about File api
rename issue.
If Camel code make use of File API for renaming, then it would fail
regardless of JDK. I like to paste snippet from JDK-7 and JDK-6
document -
http://docs.oracle.com/javase/7/docs/api/java/io/File.html#renameTo%28java.io.File%29.
/"public boolean renameTo(File <http://docs.oracle.com/javase/7/docs/api/java/io/File.html> dest)/
/Renames the file denoted by this abstract pathname. /
/Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful."
/
I am a Camel beginer, but very much affected by above bug in windows
platform multiple times in other java projects.
Regards
Mohan/
/
On 19/9/2012 4:40 AM, Christian Müller wrote:
> Please find my comments inline.
>
> Best,
> Christian
>
> On Tue, Sep 18, 2012 at 11:53 AM, OrackBahama <jd...@metadok.de>wrote:
>
>> Hi,
>> meanwhile I suspect Windows (or the Java VM, not Camel) to be the culprit
>> for not unlocking files immediately after renaming. There's an extended
>> version of the Windows MoveFile-API function (MoveFileEx) that takes extra
>> flags to force Windows to flush copy and delete operations to disk *before*
>> returning from the MoveFile function.
>>
>> It might be possible that this API call is *not used* in the current Java 6
>> VM implementation.
>> I think it's possible to reproduce the (bad) behavior even with correctly
>> closed InputStreams for example with Java copy and delete operations in
>> quick succession.
>>
>> To be more precise, the questions are:
>> 1.) Is there a workaround (e.g. timeout or retry parameters) for my current
>> runtime config ?
>>
> I don't know. You didn't shared your runtime config until now...
>
> 2.) Are all Camel components Java 7 ready ?
> Starting with Camel 2.10.0, we fully support Java 7:
> http://camel.apache.org/camel-2100-release.html
>
> 3.) Is this bug fixed in Java 7 ?
> I don't know. I think you should ask this question on an Oracle mailing
> list.
>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719517.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
> --
>
Re: File renaming problems under Windows
Posted by Christian Müller <ch...@gmail.com>.
Please find my comments inline.
Best,
Christian
On Tue, Sep 18, 2012 at 11:53 AM, OrackBahama <jd...@metadok.de>wrote:
> Hi,
> meanwhile I suspect Windows (or the Java VM, not Camel) to be the culprit
> for not unlocking files immediately after renaming. There's an extended
> version of the Windows MoveFile-API function (MoveFileEx) that takes extra
> flags to force Windows to flush copy and delete operations to disk *before*
> returning from the MoveFile function.
>
> It might be possible that this API call is *not used* in the current Java 6
> VM implementation.
> I think it's possible to reproduce the (bad) behavior even with correctly
> closed InputStreams for example with Java copy and delete operations in
> quick succession.
>
> To be more precise, the questions are:
> 1.) Is there a workaround (e.g. timeout or retry parameters) for my current
> runtime config ?
>
I don't know. You didn't shared your runtime config until now...
2.) Are all Camel components Java 7 ready ?
>
Starting with Camel 2.10.0, we fully support Java 7:
http://camel.apache.org/camel-2100-release.html
3.) Is this bug fixed in Java 7 ?
>
I don't know. I think you should ask this question on an Oracle mailing
list.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719517.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Re: File renaming problems under Windows
Posted by OrackBahama <jd...@metadok.de>.
Hi,
meanwhile I suspect Windows (or the Java VM, not Camel) to be the culprit
for not unlocking files immediately after renaming. There's an extended
version of the Windows MoveFile-API function (MoveFileEx) that takes extra
flags to force Windows to flush copy and delete operations to disk *before*
returning from the MoveFile function.
It might be possible that this API call is *not used* in the current Java 6
VM implementation.
I think it's possible to reproduce the (bad) behavior even with correctly
closed InputStreams for example with Java copy and delete operations in
quick succession.
To be more precise, the questions are:
1.) Is there a workaround (e.g. timeout or retry parameters) for my current
runtime config ?
2.) Are all Camel components Java 7 ready ?
3.) Is this bug fixed in Java 7 ?
--
View this message in context: http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484p5719517.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File renaming problems under Windows
Posted by Christian Müller <ch...@gmail.com>.
failed due cannot delete from file: C:\in\00800530.txt after copy succeeded
This happens when the InputStream was not properly closed. This could be a
Camel issue or an issue on your side, if you use a custom
bean/processor/... and didn't close the InputStream (in this case).
I think you have to share your code (route) with us, to get more help.
And if you can, upgrade to an newer version (Camel 2.10.1 is the latest).
Camel 2.8.x isn't supported anymore.
It would be helpful to know whether this issue exist in 2.10.1 or not.
Best,
Christian
On Mon, Sep 17, 2012 at 6:18 PM, OrackBahama <jd...@metadok.de> wrote:
> Hi,
>
> I'm getting exceptions when polling and processing files (in the same
> filesystem). The last camel step is moving the file into a backup or into
> an
> error directory using the parameters 'move' and 'moveFailed' of the file
> component.
>
> I'm aware of renaming problems if there's a different filesystem or
> concurrent consumers trying to read the file, but I'm worried that this
> happens under relatively simple circumstances (Java 6, Camel 2.8.1). Do I
> have to use Java 7 for better native filesystem locking behavior ? Or can I
> configure several retries ?
>
> This is the exception that occurs - thanks for any recommendation in
> advance
> !
>
> 2012-09-15 10:26:13 ERROR : CamelLogger file.GenericFileOnCompletion :
> Caused by:
> [org.apache.camel.component.file.GenericFileOperationFailedException -
> Error
> renaming file from C:\in\00800530.txt to C:\error\2012-09-15\00800530.txt]
> org.apache.camel.component.file.GenericFileOperationFailedException: Error
> renaming file from C:\in\00800530.txt to C:\error\2012-09-15\00800530.txt
> at
>
> org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:68)
> at
>
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:96)
> at
>
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.rollback(GenericFileRenameProcessStrategy.java:68)
> at
>
> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyRollback(GenericFileOnCompletion.java:154)
> at
>
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:89)
> at
>
> org.apache.camel.component.file.GenericFileOnCompletion.onFailure(GenericFileOnCompletion.java:58)
> at
>
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:53)
> at
> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:208)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:188)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:37)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:146)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:105)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:139)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:106)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
> at
>
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69)
> at
>
> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:353)
> at
>
> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:176)
> at
>
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:137)
> at
>
> org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:138)
> at
>
> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:90)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Renaming file from: C:\in\00800530.txt to:
> C:\error\2012-09-15\00800530.txt failed due cannot delete from file:
> C:\in\00800530.txt after copy succeeded
> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:379)
> at
>
> org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:66)
> ... 31 more
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/File-renaming-problems-under-Windows-tp5719484.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--