You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Sven Nold <Sv...@isb-ag.de> on 2014/09/02 08:44:38 UTC
camel 2.13.2 zipdataformat
Hi Guys,
I am trying to read Zipentries and process them separately. But using the file component it seems the file is not correctly closed. I can see the file for a few seconds in the .done and source directory. But then it disappears in the .done folder and is back in the source directory :
<bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
<property name="usingIterator" value="true" />
</bean>
<route id="readZip">
<from uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000" />
<unmarshal ref="zipFileDataFormat" />
<!-- <split streaming="true"> -->
<!-- <simple>$simple{body}</simple> -->
<!-- <to uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
<!-- </split> -->
<to uri="log://OUT?level=INFO&showAll=true&multiline=true" />
</route>
If I remove the unmarshal the file component works fine (it moves processed file to .done).
Any Ideas? I guess I could do a workaround by using noop=true and moving the file myself, but it would be nice if it works as expected.
Best regards,
Sven
Re: camel 2.13.2 zipdataformat
Posted by "Walzer, Thomas" <th...@integratix.net>.
Thanks Sven, I owe you a beer. zipSplitter works great.
Cheers, Thomas.
Am 03.09.2014 um 07:36 schrieb Sven Nold <Sv...@isb-ag.de>:
> Hi,
>
> sorry didn't fix it, still the same error.
>
>
>
> -----Original Message-----
> From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Sent: Tuesday, September 02, 2014 4:07 PM
> To: users@camel.apache.org
> Subject: Re: camel 2.13.2 zipdataformat
>
> Hi
>
> Thanks. I suspect its because this code creates a new input stream in the message body
> https://github.com/apache/camel/blob/master/components/camel-zipfile/src/main/java/org/apache/camel/dataformat/zipfile/ZipIterator.java#L106
>
> That is not explicit closed.
>
> Can you try changing
>
> answer.setBody(new ZipInputStreamWrapper(zipInputStream));
>
> To
>
> answer.setBody(zipInputStream);
>
> And give it a test on your Windows system?
>
> On Tue, Sep 2, 2014 at 3:13 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>> Hi,
>>
>> same result, but workaround is to use zipSplitter like this:
>>
>> <bean id="zipSplitter"
>> class="org.apache.camel.dataformat.zipfile.ZipSplitter" />
>>
>> <route id="readZip" streamCache="true">
>> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
>> <!-- <unmarshal ref="zipFileDataFormat"/> -->
>> <camel:split streaming="true">
>> <!-- <camel:simple>${body}</camel:simple> -->
>> <camel:ref>zipSplitter</camel:ref>
>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>> <camel:to uri="file://target/unzipped"/>
>> </camel:split>
>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>> </route>
>>
>> ZipSplitter seems not to have this issue. Also works for 2.13.2. which could be helpful for you @Thomas.
>>
>> So from my point of view CAMEL-7577 is not RESOLVED.
>>
>> Best regards,
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
>> Gesendet: Dienstag, 2. September 2014 14:29
>> An: users@camel.apache.org
>> Betreff: Re: camel 2.13.2 zipdataformat
>>
>> You need to build the code to make sure the snapshot is using recent code.
>>
>> On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>>> Hi Claus,
>>>
>>> I changed to 2.13.3-SNAPSHOT but still the same issue.
>>>
>>> Best regards,
>>>
>>> Sven
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
>>> Gesendet: Dienstag, 2. September 2014 13:52
>>> An: users@camel.apache.org
>>> Betreff: Re: camel 2.13.2 zipdataformat
>>>
>>> Hi
>>>
>>> See CAMEL-7577
>>>
>>> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>>>> Hi,
>>>>
>>>> interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
>>>>
>>>> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>>> <property name="usingIterator" value="false"/>
>>>> </bean>
>>>>
>>>> <bean id="zipAggregationStrategy"
>>>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationSt
>>>> r
>>>> a
>>>> tegy"/>
>>>>
>>>> <camelContext xmlns="http://camel.apache.org/schema/spring">
>>>> <!-- write the zip so we can process something repeatedly -->
>>>> <route id="writeZip">
>>>> <from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
>>>> <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
>>>> <correlationExpression>
>>>> <constant>true</constant>
>>>> </correlationExpression>
>>>> <to uri="file://target?fileName=any.xxx"/>
>>>> </aggregate>
>>>> </route>
>>>>
>>>> <!-- extract the zip -->
>>>> <route id="readZip" streamCache="true">
>>>> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
>>>> <unmarshal ref="zipFileDataFormat"/>
>>>> <!--<camel:split streaming="true">
>>>> <camel:simple>${body}</camel:simple>
>>>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>>>> <camel:to uri="file://target/unzipped"/>
>>>> </camel:split> -->
>>>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>>>> </route>
>>>> </camelContext>
>>>>
>>>> Works fine.
>>>>
>>>> Best regards,
>>>>
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
>>>> Gesendet: Dienstag, 2. September 2014 13:38
>>>> An: users@camel.apache.org
>>>> Betreff: AW: camel 2.13.2 zipdataformat
>>>>
>>>> Hi,
>>>>
>>>> thanks for this test now I think here is the problem:
>>>>
>>>> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
>>>> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
>>>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
>>>> at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
>>>> at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>>>> at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
>>>> at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
>>>> at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
>>>> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
>>>> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
>>>> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
>>>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
>>>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
>>>> at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
>>>> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>>>> at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
>>>> at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
>>>> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
>>>> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
>>>> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
>>>> at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
>>>> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
>>>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
>>>> ... 25 more
>>>>
>>>> This is why it grabs the file all the time.
>>>>
>>>> Regarding your questions:
>>>>
>>>> 1) well thanks to your test you can see error above
>>>> 2) yes it is processing it over and over again
>>>> 3) no doesn't move to .error
>>>> 4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
>>>>
>>>> Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
>>>> And again if I remove the unmarshal and split it works fine.
>>>>
>>>> Thanks for any help.
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
>>>> Gesendet: Dienstag, 2. September 2014 10:37
>>>> An: users@camel.apache.org
>>>> Betreff: Re: camel 2.13.2 zipdataformat
>>>>
>>>> Hi,
>>>>
>>>> I just added an example at
>>>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
>>>> This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
>>>>
>>>> Few questions if the example doesn't help you:
>>>>
>>>> - Are there any hints in the logs (try increasing logging level if not)?
>>>> - If the file is back in the source dir does it try to process again?
>>>> - Is there anything in the .error (moveFailed dir)?
>>>> - Could the source file be being updated while you're processing it - if
>>>> so, can you try the preMove option?
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On 2 September 2014 08:46, Walzer, Thomas
>>>> <th...@integratix.net>
>>>> wrote:
>>>>
>>>>> Hi Sven,
>>>>>
>>>>> I seem to have a similar problem. I also use 2.13.2.
>>>>> My code is almost identical. The only difference is my in-url, that
>>>>> looks
>>>>> like:
>>>>>
>>>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${
>>>>> f i l e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>>>>>
>>>>> I get a random number of retrievals (about 4-20) and therefore the
>>>>> same number of files in the „archive“ directory. The timestamps are
>>>>> 6 seconds apart.
>>>>>
>>>>> Cheers, Thomas.
>>>>>
>>>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>> I am trying to read Zipentries and process them separately. But
>>>>>> using
>>>>> the file component it seems the file is not correctly closed. I can
>>>>> see the file for a few seconds in the .done and source directory.
>>>>> But then it disappears in the .done folder and is back in the source directory :
>>>>>>
>>>>>>
>>>>>> <bean id="zipFileDataFormat"
>>>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>>>>> <property name="usingIterator" value="true" /> </bean>
>>>>>>
>>>>>> <route id="readZip">
>>>>>> <from
>>>>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
>>>>> />
>>>>>> <unmarshal ref="zipFileDataFormat" />
>>>>>> <!-- <split streaming="true"> -->
>>>>>> <!-- <simple>$simple{body}</simple> -->
>>>>>> <!-- <to
>>>>> uri="log://OUT?level=INFO&showAll=true&multiline=true" />
>>>>> -->
>>>>>> <!-- </split> -->
>>>>>> <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
>>>>> />
>>>>>> </route>
>>>>>>
>>>>>> If I remove the unmarshal the file component works fine (it moves
>>>>> processed file to .done).
>>>>>>
>>>>>> Any Ideas? I guess I could do a workaround by using noop=true and
>>>>>> moving
>>>>> the file myself, but it would be nice if it works as expected.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Sven
>>>>>
>>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: cibsen@redhat.com
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>> hawtio: http://hawt.io/
>>> fabric8: http://fabric8.io/
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: cibsen@redhat.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> hawtio: http://hawt.io/
>> fabric8: http://fabric8.io/
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
RE: camel 2.13.2 zipdataformat
Posted by Sven Nold <Sv...@isb-ag.de>.
Hi,
sorry didn't fix it, still the same error.
-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Sent: Tuesday, September 02, 2014 4:07 PM
To: users@camel.apache.org
Subject: Re: camel 2.13.2 zipdataformat
Hi
Thanks. I suspect its because this code creates a new input stream in the message body
https://github.com/apache/camel/blob/master/components/camel-zipfile/src/main/java/org/apache/camel/dataformat/zipfile/ZipIterator.java#L106
That is not explicit closed.
Can you try changing
answer.setBody(new ZipInputStreamWrapper(zipInputStream));
To
answer.setBody(zipInputStream);
And give it a test on your Windows system?
On Tue, Sep 2, 2014 at 3:13 PM, Sven Nold <Sv...@isb-ag.de> wrote:
> Hi,
>
> same result, but workaround is to use zipSplitter like this:
>
> <bean id="zipSplitter"
> class="org.apache.camel.dataformat.zipfile.ZipSplitter" />
>
> <route id="readZip" streamCache="true">
> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
> <!-- <unmarshal ref="zipFileDataFormat"/> -->
> <camel:split streaming="true">
> <!-- <camel:simple>${body}</camel:simple> -->
> <camel:ref>zipSplitter</camel:ref>
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> <camel:to uri="file://target/unzipped"/>
> </camel:split>
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> </route>
>
> ZipSplitter seems not to have this issue. Also works for 2.13.2. which could be helpful for you @Thomas.
>
> So from my point of view CAMEL-7577 is not RESOLVED.
>
> Best regards,
>
> -----Ursprüngliche Nachricht-----
> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Gesendet: Dienstag, 2. September 2014 14:29
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> You need to build the code to make sure the snapshot is using recent code.
>
> On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>> Hi Claus,
>>
>> I changed to 2.13.3-SNAPSHOT but still the same issue.
>>
>> Best regards,
>>
>> Sven
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
>> Gesendet: Dienstag, 2. September 2014 13:52
>> An: users@camel.apache.org
>> Betreff: Re: camel 2.13.2 zipdataformat
>>
>> Hi
>>
>> See CAMEL-7577
>>
>> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>>> Hi,
>>>
>>> interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
>>>
>>> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>> <property name="usingIterator" value="false"/>
>>> </bean>
>>>
>>> <bean id="zipAggregationStrategy"
>>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationSt
>>> r
>>> a
>>> tegy"/>
>>>
>>> <camelContext xmlns="http://camel.apache.org/schema/spring">
>>> <!-- write the zip so we can process something repeatedly -->
>>> <route id="writeZip">
>>> <from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
>>> <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
>>> <correlationExpression>
>>> <constant>true</constant>
>>> </correlationExpression>
>>> <to uri="file://target?fileName=any.xxx"/>
>>> </aggregate>
>>> </route>
>>>
>>> <!-- extract the zip -->
>>> <route id="readZip" streamCache="true">
>>> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
>>> <unmarshal ref="zipFileDataFormat"/>
>>> <!--<camel:split streaming="true">
>>> <camel:simple>${body}</camel:simple>
>>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>>> <camel:to uri="file://target/unzipped"/>
>>> </camel:split> -->
>>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>>> </route>
>>> </camelContext>
>>>
>>> Works fine.
>>>
>>> Best regards,
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
>>> Gesendet: Dienstag, 2. September 2014 13:38
>>> An: users@camel.apache.org
>>> Betreff: AW: camel 2.13.2 zipdataformat
>>>
>>> Hi,
>>>
>>> thanks for this test now I think here is the problem:
>>>
>>> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
>>> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
>>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
>>> at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
>>> at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>>> at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
>>> at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
>>> at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
>>> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
>>> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
>>> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
>>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
>>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
>>> at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
>>> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>>> at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
>>> at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
>>> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
>>> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
>>> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
>>> at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
>>> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
>>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
>>> ... 25 more
>>>
>>> This is why it grabs the file all the time.
>>>
>>> Regarding your questions:
>>>
>>> 1) well thanks to your test you can see error above
>>> 2) yes it is processing it over and over again
>>> 3) no doesn't move to .error
>>> 4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
>>>
>>> Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
>>> And again if I remove the unmarshal and split it works fine.
>>>
>>> Thanks for any help.
>>>
>>> Best regards,
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
>>> Gesendet: Dienstag, 2. September 2014 10:37
>>> An: users@camel.apache.org
>>> Betreff: Re: camel 2.13.2 zipdataformat
>>>
>>> Hi,
>>>
>>> I just added an example at
>>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
>>> This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
>>>
>>> Few questions if the example doesn't help you:
>>>
>>> - Are there any hints in the logs (try increasing logging level if not)?
>>> - If the file is back in the source dir does it try to process again?
>>> - Is there anything in the .error (moveFailed dir)?
>>> - Could the source file be being updated while you're processing it - if
>>> so, can you try the preMove option?
>>>
>>> Thanks,
>>>
>>>
>>> On 2 September 2014 08:46, Walzer, Thomas
>>> <th...@integratix.net>
>>> wrote:
>>>
>>>> Hi Sven,
>>>>
>>>> I seem to have a similar problem. I also use 2.13.2.
>>>> My code is almost identical. The only difference is my in-url, that
>>>> looks
>>>> like:
>>>>
>>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${
>>>> f i l e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>>>>
>>>> I get a random number of retrievals (about 4-20) and therefore the
>>>> same number of files in the „archive“ directory. The timestamps are
>>>> 6 seconds apart.
>>>>
>>>> Cheers, Thomas.
>>>>
>>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I am trying to read Zipentries and process them separately. But
>>>> > using
>>>> the file component it seems the file is not correctly closed. I can
>>>> see the file for a few seconds in the .done and source directory.
>>>> But then it disappears in the .done folder and is back in the source directory :
>>>> >
>>>> >
>>>> > <bean id="zipFileDataFormat"
>>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>>> > <property name="usingIterator" value="true" /> </bean>
>>>> >
>>>> > <route id="readZip">
>>>> > <from
>>>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
>>>> />
>>>> > <unmarshal ref="zipFileDataFormat" />
>>>> > <!-- <split streaming="true"> -->
>>>> > <!-- <simple>$simple{body}</simple> -->
>>>> > <!-- <to
>>>> uri="log://OUT?level=INFO&showAll=true&multiline=true" />
>>>> -->
>>>> > <!-- </split> -->
>>>> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
>>>> />
>>>> > </route>
>>>> >
>>>> > If I remove the unmarshal the file component works fine (it moves
>>>> processed file to .done).
>>>> >
>>>> > Any Ideas? I guess I could do a workaround by using noop=true and
>>>> > moving
>>>> the file myself, but it would be nice if it works as expected.
>>>> >
>>>> > Best regards,
>>>> >
>>>> > Sven
>>>>
>>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: cibsen@redhat.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> hawtio: http://hawt.io/
>> fabric8: http://fabric8.io/
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: camel 2.13.2 zipdataformat
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Thanks. I suspect its because this code creates a new input stream in
the message body
https://github.com/apache/camel/blob/master/components/camel-zipfile/src/main/java/org/apache/camel/dataformat/zipfile/ZipIterator.java#L106
That is not explicit closed.
Can you try changing
answer.setBody(new ZipInputStreamWrapper(zipInputStream));
To
answer.setBody(zipInputStream);
And give it a test on your Windows system?
On Tue, Sep 2, 2014 at 3:13 PM, Sven Nold <Sv...@isb-ag.de> wrote:
> Hi,
>
> same result, but workaround is to use zipSplitter like this:
>
> <bean id="zipSplitter" class="org.apache.camel.dataformat.zipfile.ZipSplitter" />
>
> <route id="readZip" streamCache="true">
> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
> <!-- <unmarshal ref="zipFileDataFormat"/> -->
> <camel:split streaming="true">
> <!-- <camel:simple>${body}</camel:simple> -->
> <camel:ref>zipSplitter</camel:ref>
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> <camel:to uri="file://target/unzipped"/>
> </camel:split>
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> </route>
>
> ZipSplitter seems not to have this issue. Also works for 2.13.2. which could be helpful for you @Thomas.
>
> So from my point of view CAMEL-7577 is not RESOLVED.
>
> Best regards,
>
> -----Ursprüngliche Nachricht-----
> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Gesendet: Dienstag, 2. September 2014 14:29
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> You need to build the code to make sure the snapshot is using recent code.
>
> On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>> Hi Claus,
>>
>> I changed to 2.13.3-SNAPSHOT but still the same issue.
>>
>> Best regards,
>>
>> Sven
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
>> Gesendet: Dienstag, 2. September 2014 13:52
>> An: users@camel.apache.org
>> Betreff: Re: camel 2.13.2 zipdataformat
>>
>> Hi
>>
>> See CAMEL-7577
>>
>> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>>> Hi,
>>>
>>> interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
>>>
>>> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>> <property name="usingIterator" value="false"/>
>>> </bean>
>>>
>>> <bean id="zipAggregationStrategy"
>>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStr
>>> a
>>> tegy"/>
>>>
>>> <camelContext xmlns="http://camel.apache.org/schema/spring">
>>> <!-- write the zip so we can process something repeatedly -->
>>> <route id="writeZip">
>>> <from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
>>> <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
>>> <correlationExpression>
>>> <constant>true</constant>
>>> </correlationExpression>
>>> <to uri="file://target?fileName=any.xxx"/>
>>> </aggregate>
>>> </route>
>>>
>>> <!-- extract the zip -->
>>> <route id="readZip" streamCache="true">
>>> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
>>> <unmarshal ref="zipFileDataFormat"/>
>>> <!--<camel:split streaming="true">
>>> <camel:simple>${body}</camel:simple>
>>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>>> <camel:to uri="file://target/unzipped"/>
>>> </camel:split> -->
>>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>>> </route>
>>> </camelContext>
>>>
>>> Works fine.
>>>
>>> Best regards,
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
>>> Gesendet: Dienstag, 2. September 2014 13:38
>>> An: users@camel.apache.org
>>> Betreff: AW: camel 2.13.2 zipdataformat
>>>
>>> Hi,
>>>
>>> thanks for this test now I think here is the problem:
>>>
>>> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
>>> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
>>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
>>> at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
>>> at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>>> at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
>>> at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
>>> at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
>>> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
>>> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
>>> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
>>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
>>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
>>> at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
>>> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>>> at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
>>> at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
>>> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
>>> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
>>> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
>>> at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
>>> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
>>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
>>> ... 25 more
>>>
>>> This is why it grabs the file all the time.
>>>
>>> Regarding your questions:
>>>
>>> 1) well thanks to your test you can see error above
>>> 2) yes it is processing it over and over again
>>> 3) no doesn't move to .error
>>> 4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
>>>
>>> Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
>>> And again if I remove the unmarshal and split it works fine.
>>>
>>> Thanks for any help.
>>>
>>> Best regards,
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
>>> Gesendet: Dienstag, 2. September 2014 10:37
>>> An: users@camel.apache.org
>>> Betreff: Re: camel 2.13.2 zipdataformat
>>>
>>> Hi,
>>>
>>> I just added an example at
>>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
>>> This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
>>>
>>> Few questions if the example doesn't help you:
>>>
>>> - Are there any hints in the logs (try increasing logging level if not)?
>>> - If the file is back in the source dir does it try to process again?
>>> - Is there anything in the .error (moveFailed dir)?
>>> - Could the source file be being updated while you're processing it - if
>>> so, can you try the preMove option?
>>>
>>> Thanks,
>>>
>>>
>>> On 2 September 2014 08:46, Walzer, Thomas
>>> <th...@integratix.net>
>>> wrote:
>>>
>>>> Hi Sven,
>>>>
>>>> I seem to have a similar problem. I also use 2.13.2.
>>>> My code is almost identical. The only difference is my in-url, that
>>>> looks
>>>> like:
>>>>
>>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${f
>>>> i l e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>>>>
>>>> I get a random number of retrievals (about 4-20) and therefore the
>>>> same number of files in the „archive“ directory. The timestamps are
>>>> 6 seconds apart.
>>>>
>>>> Cheers, Thomas.
>>>>
>>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I am trying to read Zipentries and process them separately. But
>>>> > using
>>>> the file component it seems the file is not correctly closed. I can
>>>> see the file for a few seconds in the .done and source directory.
>>>> But then it disappears in the .done folder and is back in the source directory :
>>>> >
>>>> >
>>>> > <bean id="zipFileDataFormat"
>>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>>> > <property name="usingIterator" value="true" /> </bean>
>>>> >
>>>> > <route id="readZip">
>>>> > <from
>>>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
>>>> />
>>>> > <unmarshal ref="zipFileDataFormat" />
>>>> > <!-- <split streaming="true"> -->
>>>> > <!-- <simple>$simple{body}</simple> -->
>>>> > <!-- <to
>>>> uri="log://OUT?level=INFO&showAll=true&multiline=true" />
>>>> -->
>>>> > <!-- </split> -->
>>>> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
>>>> />
>>>> > </route>
>>>> >
>>>> > If I remove the unmarshal the file component works fine (it moves
>>>> processed file to .done).
>>>> >
>>>> > Any Ideas? I guess I could do a workaround by using noop=true and
>>>> > moving
>>>> the file myself, but it would be nice if it works as expected.
>>>> >
>>>> > Best regards,
>>>> >
>>>> > Sven
>>>>
>>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: cibsen@redhat.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> hawtio: http://hawt.io/
>> fabric8: http://fabric8.io/
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
AW: camel 2.13.2 zipdataformat
Posted by Sven Nold <Sv...@isb-ag.de>.
Hi,
same result, but workaround is to use zipSplitter like this:
<bean id="zipSplitter" class="org.apache.camel.dataformat.zipfile.ZipSplitter" />
<route id="readZip" streamCache="true">
<from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
<!-- <unmarshal ref="zipFileDataFormat"/> -->
<camel:split streaming="true">
<!-- <camel:simple>${body}</camel:simple> -->
<camel:ref>zipSplitter</camel:ref>
<camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
<camel:to uri="file://target/unzipped"/>
</camel:split>
<camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
</route>
ZipSplitter seems not to have this issue. Also works for 2.13.2. which could be helpful for you @Thomas.
So from my point of view CAMEL-7577 is not RESOLVED.
Best regards,
-----Ursprüngliche Nachricht-----
Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Gesendet: Dienstag, 2. September 2014 14:29
An: users@camel.apache.org
Betreff: Re: camel 2.13.2 zipdataformat
You need to build the code to make sure the snapshot is using recent code.
On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <Sv...@isb-ag.de> wrote:
> Hi Claus,
>
> I changed to 2.13.3-SNAPSHOT but still the same issue.
>
> Best regards,
>
> Sven
>
> -----Ursprüngliche Nachricht-----
> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Gesendet: Dienstag, 2. September 2014 13:52
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> Hi
>
> See CAMEL-7577
>
> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>> Hi,
>>
>> interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
>>
>> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>> <property name="usingIterator" value="false"/>
>> </bean>
>>
>> <bean id="zipAggregationStrategy"
>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStr
>> a
>> tegy"/>
>>
>> <camelContext xmlns="http://camel.apache.org/schema/spring">
>> <!-- write the zip so we can process something repeatedly -->
>> <route id="writeZip">
>> <from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
>> <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
>> <correlationExpression>
>> <constant>true</constant>
>> </correlationExpression>
>> <to uri="file://target?fileName=any.xxx"/>
>> </aggregate>
>> </route>
>>
>> <!-- extract the zip -->
>> <route id="readZip" streamCache="true">
>> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
>> <unmarshal ref="zipFileDataFormat"/>
>> <!--<camel:split streaming="true">
>> <camel:simple>${body}</camel:simple>
>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>> <camel:to uri="file://target/unzipped"/>
>> </camel:split> -->
>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>> </route>
>> </camelContext>
>>
>> Works fine.
>>
>> Best regards,
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
>> Gesendet: Dienstag, 2. September 2014 13:38
>> An: users@camel.apache.org
>> Betreff: AW: camel 2.13.2 zipdataformat
>>
>> Hi,
>>
>> thanks for this test now I think here is the problem:
>>
>> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
>> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
>> at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
>> at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>> at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
>> at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
>> at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
>> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
>> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
>> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
>> at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
>> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>> at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
>> at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
>> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
>> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
>> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
>> at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
>> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
>> ... 25 more
>>
>> This is why it grabs the file all the time.
>>
>> Regarding your questions:
>>
>> 1) well thanks to your test you can see error above
>> 2) yes it is processing it over and over again
>> 3) no doesn't move to .error
>> 4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
>>
>> Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
>> And again if I remove the unmarshal and split it works fine.
>>
>> Thanks for any help.
>>
>> Best regards,
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
>> Gesendet: Dienstag, 2. September 2014 10:37
>> An: users@camel.apache.org
>> Betreff: Re: camel 2.13.2 zipdataformat
>>
>> Hi,
>>
>> I just added an example at
>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
>> This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
>>
>> Few questions if the example doesn't help you:
>>
>> - Are there any hints in the logs (try increasing logging level if not)?
>> - If the file is back in the source dir does it try to process again?
>> - Is there anything in the .error (moveFailed dir)?
>> - Could the source file be being updated while you're processing it - if
>> so, can you try the preMove option?
>>
>> Thanks,
>>
>>
>> On 2 September 2014 08:46, Walzer, Thomas
>> <th...@integratix.net>
>> wrote:
>>
>>> Hi Sven,
>>>
>>> I seem to have a similar problem. I also use 2.13.2.
>>> My code is almost identical. The only difference is my in-url, that
>>> looks
>>> like:
>>>
>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${f
>>> i l e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>>>
>>> I get a random number of retrievals (about 4-20) and therefore the
>>> same number of files in the „archive“ directory. The timestamps are
>>> 6 seconds apart.
>>>
>>> Cheers, Thomas.
>>>
>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>>>
>>> > Hi Guys,
>>> >
>>> > I am trying to read Zipentries and process them separately. But
>>> > using
>>> the file component it seems the file is not correctly closed. I can
>>> see the file for a few seconds in the .done and source directory.
>>> But then it disappears in the .done folder and is back in the source directory :
>>> >
>>> >
>>> > <bean id="zipFileDataFormat"
>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>> > <property name="usingIterator" value="true" /> </bean>
>>> >
>>> > <route id="readZip">
>>> > <from
>>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
>>> />
>>> > <unmarshal ref="zipFileDataFormat" />
>>> > <!-- <split streaming="true"> -->
>>> > <!-- <simple>$simple{body}</simple> -->
>>> > <!-- <to
>>> uri="log://OUT?level=INFO&showAll=true&multiline=true" />
>>> -->
>>> > <!-- </split> -->
>>> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
>>> />
>>> > </route>
>>> >
>>> > If I remove the unmarshal the file component works fine (it moves
>>> processed file to .done).
>>> >
>>> > Any Ideas? I guess I could do a workaround by using noop=true and
>>> > moving
>>> the file myself, but it would be nice if it works as expected.
>>> >
>>> > Best regards,
>>> >
>>> > Sven
>>>
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: camel 2.13.2 zipdataformat
Posted by Claus Ibsen <cl...@gmail.com>.
You need to build the code to make sure the snapshot is using recent code.
On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <Sv...@isb-ag.de> wrote:
> Hi Claus,
>
> I changed to 2.13.3-SNAPSHOT but still the same issue.
>
> Best regards,
>
> Sven
>
> -----Ursprüngliche Nachricht-----
> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Gesendet: Dienstag, 2. September 2014 13:52
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> Hi
>
> See CAMEL-7577
>
> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sv...@isb-ag.de> wrote:
>> Hi,
>>
>> interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
>>
>> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>> <property name="usingIterator" value="false"/>
>> </bean>
>>
>> <bean id="zipAggregationStrategy"
>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStra
>> tegy"/>
>>
>> <camelContext xmlns="http://camel.apache.org/schema/spring">
>> <!-- write the zip so we can process something repeatedly -->
>> <route id="writeZip">
>> <from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
>> <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
>> <correlationExpression>
>> <constant>true</constant>
>> </correlationExpression>
>> <to uri="file://target?fileName=any.xxx"/>
>> </aggregate>
>> </route>
>>
>> <!-- extract the zip -->
>> <route id="readZip" streamCache="true">
>> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
>> <unmarshal ref="zipFileDataFormat"/>
>> <!--<camel:split streaming="true">
>> <camel:simple>${body}</camel:simple>
>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>> <camel:to uri="file://target/unzipped"/>
>> </camel:split> -->
>> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
>> </route>
>> </camelContext>
>>
>> Works fine.
>>
>> Best regards,
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
>> Gesendet: Dienstag, 2. September 2014 13:38
>> An: users@camel.apache.org
>> Betreff: AW: camel 2.13.2 zipdataformat
>>
>> Hi,
>>
>> thanks for this test now I think here is the problem:
>>
>> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
>> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
>> at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
>> at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>> at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
>> at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
>> at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
>> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
>> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
>> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
>> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
>> at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
>> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>> at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
>> at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
>> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
>> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
>> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
>> at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
>> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
>> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
>> ... 25 more
>>
>> This is why it grabs the file all the time.
>>
>> Regarding your questions:
>>
>> 1) well thanks to your test you can see error above
>> 2) yes it is processing it over and over again
>> 3) no doesn't move to .error
>> 4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
>>
>> Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
>> And again if I remove the unmarshal and split it works fine.
>>
>> Thanks for any help.
>>
>> Best regards,
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
>> Gesendet: Dienstag, 2. September 2014 10:37
>> An: users@camel.apache.org
>> Betreff: Re: camel 2.13.2 zipdataformat
>>
>> Hi,
>>
>> I just added an example at
>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
>> This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
>>
>> Few questions if the example doesn't help you:
>>
>> - Are there any hints in the logs (try increasing logging level if not)?
>> - If the file is back in the source dir does it try to process again?
>> - Is there anything in the .error (moveFailed dir)?
>> - Could the source file be being updated while you're processing it - if
>> so, can you try the preMove option?
>>
>> Thanks,
>>
>>
>> On 2 September 2014 08:46, Walzer, Thomas
>> <th...@integratix.net>
>> wrote:
>>
>>> Hi Sven,
>>>
>>> I seem to have a similar problem. I also use 2.13.2.
>>> My code is almost identical. The only difference is my in-url, that
>>> looks
>>> like:
>>>
>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${fi
>>> l e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>>>
>>> I get a random number of retrievals (about 4-20) and therefore the
>>> same number of files in the „archive“ directory. The timestamps are 6
>>> seconds apart.
>>>
>>> Cheers, Thomas.
>>>
>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>>>
>>> > Hi Guys,
>>> >
>>> > I am trying to read Zipentries and process them separately. But
>>> > using
>>> the file component it seems the file is not correctly closed. I can
>>> see the file for a few seconds in the .done and source directory. But
>>> then it disappears in the .done folder and is back in the source directory :
>>> >
>>> >
>>> > <bean id="zipFileDataFormat"
>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>> > <property name="usingIterator" value="true" /> </bean>
>>> >
>>> > <route id="readZip">
>>> > <from
>>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
>>> />
>>> > <unmarshal ref="zipFileDataFormat" />
>>> > <!-- <split streaming="true"> -->
>>> > <!-- <simple>$simple{body}</simple> -->
>>> > <!-- <to
>>> uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
>>> > <!-- </split> -->
>>> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
>>> />
>>> > </route>
>>> >
>>> > If I remove the unmarshal the file component works fine (it moves
>>> processed file to .done).
>>> >
>>> > Any Ideas? I guess I could do a workaround by using noop=true and
>>> > moving
>>> the file myself, but it would be nice if it works as expected.
>>> >
>>> > Best regards,
>>> >
>>> > Sven
>>>
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
AW: camel 2.13.2 zipdataformat
Posted by Sven Nold <Sv...@isb-ag.de>.
Hi Claus,
I changed to 2.13.3-SNAPSHOT but still the same issue.
Best regards,
Sven
-----Ursprüngliche Nachricht-----
Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Gesendet: Dienstag, 2. September 2014 13:52
An: users@camel.apache.org
Betreff: Re: camel 2.13.2 zipdataformat
Hi
See CAMEL-7577
On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sv...@isb-ag.de> wrote:
> Hi,
>
> interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
>
> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
> <property name="usingIterator" value="false"/>
> </bean>
>
> <bean id="zipAggregationStrategy"
> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStra
> tegy"/>
>
> <camelContext xmlns="http://camel.apache.org/schema/spring">
> <!-- write the zip so we can process something repeatedly -->
> <route id="writeZip">
> <from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
> <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
> <correlationExpression>
> <constant>true</constant>
> </correlationExpression>
> <to uri="file://target?fileName=any.xxx"/>
> </aggregate>
> </route>
>
> <!-- extract the zip -->
> <route id="readZip" streamCache="true">
> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
> <unmarshal ref="zipFileDataFormat"/>
> <!--<camel:split streaming="true">
> <camel:simple>${body}</camel:simple>
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> <camel:to uri="file://target/unzipped"/>
> </camel:split> -->
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> </route>
> </camelContext>
>
> Works fine.
>
> Best regards,
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
> Gesendet: Dienstag, 2. September 2014 13:38
> An: users@camel.apache.org
> Betreff: AW: camel 2.13.2 zipdataformat
>
> Hi,
>
> thanks for this test now I think here is the problem:
>
> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
> at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
> at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
> at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
> at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
> at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
> at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
> at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
> ... 25 more
>
> This is why it grabs the file all the time.
>
> Regarding your questions:
>
> 1) well thanks to your test you can see error above
> 2) yes it is processing it over and over again
> 3) no doesn't move to .error
> 4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
>
> Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
> And again if I remove the unmarshal and split it works fine.
>
> Thanks for any help.
>
> Best regards,
>
>
> -----Ursprüngliche Nachricht-----
> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
> Gesendet: Dienstag, 2. September 2014 10:37
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> Hi,
>
> I just added an example at
> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
> This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
>
> Few questions if the example doesn't help you:
>
> - Are there any hints in the logs (try increasing logging level if not)?
> - If the file is back in the source dir does it try to process again?
> - Is there anything in the .error (moveFailed dir)?
> - Could the source file be being updated while you're processing it - if
> so, can you try the preMove option?
>
> Thanks,
>
>
> On 2 September 2014 08:46, Walzer, Thomas
> <th...@integratix.net>
> wrote:
>
>> Hi Sven,
>>
>> I seem to have a similar problem. I also use 2.13.2.
>> My code is almost identical. The only difference is my in-url, that
>> looks
>> like:
>>
>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${fi
>> l e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>>
>> I get a random number of retrievals (about 4-20) and therefore the
>> same number of files in the „archive“ directory. The timestamps are 6
>> seconds apart.
>>
>> Cheers, Thomas.
>>
>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>>
>> > Hi Guys,
>> >
>> > I am trying to read Zipentries and process them separately. But
>> > using
>> the file component it seems the file is not correctly closed. I can
>> see the file for a few seconds in the .done and source directory. But
>> then it disappears in the .done folder and is back in the source directory :
>> >
>> >
>> > <bean id="zipFileDataFormat"
>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>> > <property name="usingIterator" value="true" /> </bean>
>> >
>> > <route id="readZip">
>> > <from
>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
>> />
>> > <unmarshal ref="zipFileDataFormat" />
>> > <!-- <split streaming="true"> -->
>> > <!-- <simple>$simple{body}</simple> -->
>> > <!-- <to
>> uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
>> > <!-- </split> -->
>> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
>> />
>> > </route>
>> >
>> > If I remove the unmarshal the file component works fine (it moves
>> processed file to .done).
>> >
>> > Any Ideas? I guess I could do a workaround by using noop=true and
>> > moving
>> the file myself, but it would be nice if it works as expected.
>> >
>> > Best regards,
>> >
>> > Sven
>>
>>
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: camel 2.13.2 zipdataformat
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
See CAMEL-7577
On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sv...@isb-ag.de> wrote:
> Hi,
>
> interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
>
> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
> <property name="usingIterator" value="false"/>
> </bean>
>
> <bean id="zipAggregationStrategy" class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStrategy"/>
>
> <camelContext xmlns="http://camel.apache.org/schema/spring">
> <!-- write the zip so we can process something repeatedly -->
> <route id="writeZip">
> <from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
> <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
> <correlationExpression>
> <constant>true</constant>
> </correlationExpression>
> <to uri="file://target?fileName=any.xxx"/>
> </aggregate>
> </route>
>
> <!-- extract the zip -->
> <route id="readZip" streamCache="true">
> <from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
> <unmarshal ref="zipFileDataFormat"/>
> <!--<camel:split streaming="true">
> <camel:simple>${body}</camel:simple>
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> <camel:to uri="file://target/unzipped"/>
> </camel:split> -->
> <camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
> </route>
> </camelContext>
>
> Works fine.
>
> Best regards,
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
> Gesendet: Dienstag, 2. September 2014 13:38
> An: users@camel.apache.org
> Betreff: AW: camel 2.13.2 zipdataformat
>
> Hi,
>
> thanks for this test now I think here is the problem:
>
> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
> at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
> at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
> at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
> at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
> at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
> at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
> at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
> at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
> at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
> at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
> at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
> at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
> at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
> ... 25 more
>
> This is why it grabs the file all the time.
>
> Regarding your questions:
>
> 1) well thanks to your test you can see error above
> 2) yes it is processing it over and over again
> 3) no doesn't move to .error
> 4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
>
> Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
> And again if I remove the unmarshal and split it works fine.
>
> Thanks for any help.
>
> Best regards,
>
>
> -----Ursprüngliche Nachricht-----
> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
> Gesendet: Dienstag, 2. September 2014 10:37
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> Hi,
>
> I just added an example at
> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
> This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
>
> Few questions if the example doesn't help you:
>
> - Are there any hints in the logs (try increasing logging level if not)?
> - If the file is back in the source dir does it try to process again?
> - Is there anything in the .error (moveFailed dir)?
> - Could the source file be being updated while you're processing it - if
> so, can you try the preMove option?
>
> Thanks,
>
>
> On 2 September 2014 08:46, Walzer, Thomas <th...@integratix.net>
> wrote:
>
>> Hi Sven,
>>
>> I seem to have a similar problem. I also use 2.13.2.
>> My code is almost identical. The only difference is my in-url, that
>> looks
>> like:
>>
>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${fil
>> e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>>
>> I get a random number of retrievals (about 4-20) and therefore the
>> same number of files in the „archive“ directory. The timestamps are 6
>> seconds apart.
>>
>> Cheers, Thomas.
>>
>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>>
>> > Hi Guys,
>> >
>> > I am trying to read Zipentries and process them separately. But
>> > using
>> the file component it seems the file is not correctly closed. I can
>> see the file for a few seconds in the .done and source directory. But
>> then it disappears in the .done folder and is back in the source directory :
>> >
>> >
>> > <bean id="zipFileDataFormat"
>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>> > <property name="usingIterator" value="true" /> </bean>
>> >
>> > <route id="readZip">
>> > <from
>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
>> />
>> > <unmarshal ref="zipFileDataFormat" />
>> > <!-- <split streaming="true"> -->
>> > <!-- <simple>$simple{body}</simple> -->
>> > <!-- <to
>> uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
>> > <!-- </split> -->
>> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
>> />
>> > </route>
>> >
>> > If I remove the unmarshal the file component works fine (it moves
>> processed file to .done).
>> >
>> > Any Ideas? I guess I could do a workaround by using noop=true and
>> > moving
>> the file myself, but it would be nice if it works as expected.
>> >
>> > Best regards,
>> >
>> > Sven
>>
>>
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
AW: camel 2.13.2 zipdataformat
Posted by Sven Nold <Sv...@isb-ag.de>.
Hi,
interesting is also that if I only have on zipEntry and define the zipDataFormat as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on the file:
<bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
<property name="usingIterator" value="false"/>
</bean>
<bean id="zipAggregationStrategy" class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStrategy"/>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<!-- write the zip so we can process something repeatedly -->
<route id="writeZip">
<from uri="file://src/main/data?antInclude=message1.xml&noop=true"/>
<aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true" eagerCheckCompletion="true" completionSize="2">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<to uri="file://target?fileName=any.xxx"/>
</aggregate>
</route>
<!-- extract the zip -->
<route id="readZip" streamCache="true">
<from uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/>
<unmarshal ref="zipFileDataFormat"/>
<!--<camel:split streaming="true">
<camel:simple>${body}</camel:simple>
<camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
<camel:to uri="file://target/unzipped"/>
</camel:split> -->
<camel:to uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/>
</route>
</camelContext>
Works fine.
Best regards,
-----Ursprüngliche Nachricht-----
Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
Gesendet: Dienstag, 2. September 2014 13:38
An: users@camel.apache.org
Betreff: AW: camel 2.13.2 zipdataformat
Hi,
thanks for this test now I think here is the problem:
[l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
... 25 more
This is why it grabs the file all the time.
Regarding your questions:
1) well thanks to your test you can see error above
2) yes it is processing it over and over again
3) no doesn't move to .error
4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
And again if I remove the unmarshal and split it works fine.
Thanks for any help.
Best regards,
-----Ursprüngliche Nachricht-----
Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
Gesendet: Dienstag, 2. September 2014 10:37
An: users@camel.apache.org
Betreff: Re: camel 2.13.2 zipdataformat
Hi,
I just added an example at
https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
Few questions if the example doesn't help you:
- Are there any hints in the logs (try increasing logging level if not)?
- If the file is back in the source dir does it try to process again?
- Is there anything in the .error (moveFailed dir)?
- Could the source file be being updated while you're processing it - if
so, can you try the preMove option?
Thanks,
On 2 September 2014 08:46, Walzer, Thomas <th...@integratix.net>
wrote:
> Hi Sven,
>
> I seem to have a similar problem. I also use 2.13.2.
> My code is almost identical. The only difference is my in-url, that
> looks
> like:
>
> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${fil
> e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>
> I get a random number of retrievals (about 4-20) and therefore the
> same number of files in the „archive“ directory. The timestamps are 6
> seconds apart.
>
> Cheers, Thomas.
>
> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>
> > Hi Guys,
> >
> > I am trying to read Zipentries and process them separately. But
> > using
> the file component it seems the file is not correctly closed. I can
> see the file for a few seconds in the .done and source directory. But
> then it disappears in the .done folder and is back in the source directory :
> >
> >
> > <bean id="zipFileDataFormat"
> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
> > <property name="usingIterator" value="true" /> </bean>
> >
> > <route id="readZip">
> > <from
> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
> />
> > <unmarshal ref="zipFileDataFormat" />
> > <!-- <split streaming="true"> -->
> > <!-- <simple>$simple{body}</simple> -->
> > <!-- <to
> uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
> > <!-- </split> -->
> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
> />
> > </route>
> >
> > If I remove the unmarshal the file component works fine (it moves
> processed file to .done).
> >
> > Any Ideas? I guess I could do a workaround by using noop=true and
> > moving
> the file myself, but it would be nice if it works as expected.
> >
> > Best regards,
> >
> > Sven
>
>
AW: camel 2.13.2 zipdataformat
Posted by Sven Nold <Sv...@isb-ag.de>.
Hi,
thanks for this test now I think here is the problem:
[l-1) thread #1 - file://target] GenericFileOnCompletion WARN Error during commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip]
org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after copy succeeded
at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
... 25 more
This is why it grabs the file all the time.
Regarding your questions:
1) well thanks to your test you can see error above
2) yes it is processing it over and over again
3) no doesn't move to .error
4) no file is not updated, in your test case I am not quite sure, will increase delay or as you suggested preMove option.
Due to known problems off windows and zip files I even renamed it to .xxx to keep away explorer from analyzing, but still it seems there is a handle on that file.
And again if I remove the unmarshal and split it works fine.
Thanks for any help.
Best regards,
-----Ursprüngliche Nachricht-----
Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
Gesendet: Dienstag, 2. September 2014 10:37
An: users@camel.apache.org
Betreff: Re: camel 2.13.2 zipdataformat
Hi,
I just added an example at
https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
This example creates the zip file first (so that it's repeatable) & then does pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful with large files) & finally writes each file to a file. Seems to be doing what it should...
Few questions if the example doesn't help you:
- Are there any hints in the logs (try increasing logging level if not)?
- If the file is back in the source dir does it try to process again?
- Is there anything in the .error (moveFailed dir)?
- Could the source file be being updated while you're processing it - if
so, can you try the preMove option?
Thanks,
On 2 September 2014 08:46, Walzer, Thomas <th...@integratix.net>
wrote:
> Hi Sven,
>
> I seem to have a similar problem. I also use 2.13.2.
> My code is almost identical. The only difference is my in-url, that
> looks
> like:
>
> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${fil
> e:name}.${date:now:HHmmss}&antInclude=*.ZIP
>
> I get a random number of retrievals (about 4-20) and therefore the
> same number of files in the „archive“ directory. The timestamps are 6
> seconds apart.
>
> Cheers, Thomas.
>
> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>
> > Hi Guys,
> >
> > I am trying to read Zipentries and process them separately. But
> > using
> the file component it seems the file is not correctly closed. I can
> see the file for a few seconds in the .done and source directory. But
> then it disappears in the .done folder and is back in the source directory :
> >
> >
> > <bean id="zipFileDataFormat"
> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
> > <property name="usingIterator" value="true" /> </bean>
> >
> > <route id="readZip">
> > <from
> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
> />
> > <unmarshal ref="zipFileDataFormat" />
> > <!-- <split streaming="true"> -->
> > <!-- <simple>$simple{body}</simple> -->
> > <!-- <to
> uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
> > <!-- </split> -->
> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
> />
> > </route>
> >
> > If I remove the unmarshal the file component works fine (it moves
> processed file to .done).
> >
> > Any Ideas? I guess I could do a workaround by using noop=true and
> > moving
> the file myself, but it would be nice if it works as expected.
> >
> > Best regards,
> >
> > Sven
>
>
Re: camel 2.13.2 zipdataformat
Posted by Jimmi Dyson <ji...@gmail.com>.
Hi,
I just added an example at
https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
This example creates the zip file first (so that it's repeatable) & then
does pretty much what you're doing: read from file uri, unmarshal, log each
file contents (careful with large files) & finally writes each file to a
file. Seems to be doing what it should...
Few questions if the example doesn't help you:
- Are there any hints in the logs (try increasing logging level if not)?
- If the file is back in the source dir does it try to process again?
- Is there anything in the .error (moveFailed dir)?
- Could the source file be being updated while you're processing it - if
so, can you try the preMove option?
Thanks,
On 2 September 2014 08:46, Walzer, Thomas <th...@integratix.net>
wrote:
> Hi Sven,
>
> I seem to have a similar problem. I also use 2.13.2.
> My code is almost identical. The only difference is my in-url, that looks
> like:
>
> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${file:name}.${date:now:HHmmss}&antInclude=*.ZIP
>
> I get a random number of retrievals (about 4-20) and therefore the same
> number of files in the „archive“ directory. The timestamps are 6 seconds
> apart.
>
> Cheers, Thomas.
>
> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
>
> > Hi Guys,
> >
> > I am trying to read Zipentries and process them separately. But using
> the file component it seems the file is not correctly closed. I can see the
> file for a few seconds in the .done and source directory. But then it
> disappears in the .done folder and is back in the source directory :
> >
> >
> > <bean id="zipFileDataFormat"
> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
> > <property name="usingIterator" value="true" />
> > </bean>
> >
> > <route id="readZip">
> > <from
> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000"
> />
> > <unmarshal ref="zipFileDataFormat" />
> > <!-- <split streaming="true"> -->
> > <!-- <simple>$simple{body}</simple> -->
> > <!-- <to
> uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
> > <!-- </split> -->
> > <to uri="log://OUT?level=INFO&showAll=true&multiline=true"
> />
> > </route>
> >
> > If I remove the unmarshal the file component works fine (it moves
> processed file to .done).
> >
> > Any Ideas? I guess I could do a workaround by using noop=true and moving
> the file myself, but it would be nice if it works as expected.
> >
> > Best regards,
> >
> > Sven
>
>
Re: camel 2.13.2 zipdataformat
Posted by "Walzer, Thomas" <th...@integratix.net>.
Hi Sven,
I seem to have a similar problem. I also use 2.13.2.
My code is almost identical. The only difference is my in-url, that looks like:
file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${file:name}.${date:now:HHmmss}&antInclude=*.ZIP
I get a random number of retrievals (about 4-20) and therefore the same number of files in the „archive“ directory. The timestamps are 6 seconds apart.
Cheers, Thomas.
Am 02.09.2014 um 08:44 schrieb Sven Nold <Sv...@isb-ag.de>:
> Hi Guys,
>
> I am trying to read Zipentries and process them separately. But using the file component it seems the file is not correctly closed. I can see the file for a few seconds in the .done and source directory. But then it disappears in the .done folder and is back in the source directory :
>
>
> <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
> <property name="usingIterator" value="true" />
> </bean>
>
> <route id="readZip">
> <from uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000" />
> <unmarshal ref="zipFileDataFormat" />
> <!-- <split streaming="true"> -->
> <!-- <simple>$simple{body}</simple> -->
> <!-- <to uri="log://OUT?level=INFO&showAll=true&multiline=true" /> -->
> <!-- </split> -->
> <to uri="log://OUT?level=INFO&showAll=true&multiline=true" />
> </route>
>
> If I remove the unmarshal the file component works fine (it moves processed file to .done).
>
> Any Ideas? I guess I could do a workaround by using noop=true and moving the file myself, but it would be nice if it works as expected.
>
> Best regards,
>
> Sven