Done file not deleted

[Ashwin Pejavar <as...@gmail.com>]

This is specific to camel 2.11.0. Done file deletion worked as expected in
2.10.0

Scenario:
I have a fileConsumer configured for a directory with done files. It is
expected that every file dropped in the directory has a <filename>.done
file.

In both 2.10.0 and 2.11.0 files are consumed successfully when copied over
with the appropriate done file. However in 2.11.0, only the done file
associated with the last file is deleted. All the other done files remain.

Looking at the code in GenericFileOnCompletion.processStrategyCommit(...)
for 2.11.0, only the done file associated with the last file in batch is
deleted.

It sounds like the intent was only one done file for a batch. But this is an
incorrect assumption that does not line up with documentation. It might work
when I have a non-specific done file name (e.g. start) when I want to
process all files in a directory, but definitely won't work when I have a
done file per input file.





--
View this message in context: http://camel.465427.n5.nabble.com/Done-file-not-deleted-tp5735497.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Done file not deleted

[Claus Ibsen <cl...@gmail.com>]

Hi

Its wrong assumption - the done file is also deleted per file when the
done file is dynamic, eg it has {file:name in the name, such as
doneFile=${file:name}.done

But if the done file is a static name such as doneFile=end.txt
Then its only deleted at the end of batch.


On Mon, Apr 27, 2015 at 4:47 PM, dornseif <do...@gmail.com> wrote:
> I see the original fix was not re-opened?  Please at the very least can you
> add a boolean to deleteLastDoneFileOnly or deleteAllDoneFiles?
>
> In our case we need to delete all the done files after processing.
> We have a situation where many thousands of accounts exist on an SFTP site.
> Each account represented by a directory structure with multiple sub folders.
> each sub folder could have a new set of batch files.  The number of files
> uploaded is not known and the done file is used to indicate that all files
> have been uploaded for the current batch set in the current sub folder, and
> are now ready to process.  We iterate through all the accounts and all the
> sub folders of each account and need to remove the done file in each sub
> folder once the files have been processed (in this case removed).  This
> serves several purposes on of which is to let the up loader know they can
> start uploading the next batch set.
>
> Tom
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Done-file-not-deleted-tp5735497p5766387.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/

Re: Done file not deleted

[dornseif <do...@gmail.com>]

I see the original fix was not re-opened?  Please at the very least can you
add a boolean to deleteLastDoneFileOnly or deleteAllDoneFiles?  

In our case we need to delete all the done files after processing.
We have a situation where many thousands of accounts exist on an SFTP site. 
Each account represented by a directory structure with multiple sub folders. 
each sub folder could have a new set of batch files.  The number of files
uploaded is not known and the done file is used to indicate that all files
have been uploaded for the current batch set in the current sub folder, and
are now ready to process.  We iterate through all the accounts and all the
sub folders of each account and need to remove the done file in each sub
folder once the files have been processed (in this case removed).  This
serves several purposes on of which is to let the up loader know they can
start uploading the next batch set.

Tom



--
View this message in context: http://camel.465427.n5.nabble.com/Done-file-not-deleted-tp5735497p5766387.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Done file not deleted

[Willem jiang <wi...@gmail.com>]

I just checked the change log, it looks like the patch of CAMEL-5883 didn't address the batch issue well.
So I just reopen the JIRA[1] for it.

[1]https://issues.apache.org/jira/browse/CAMEL-5883  

--  
Willem Jiang

Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
          http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem





On Thursday, July 11, 2013 at 8:33 AM, Ashwin Pejavar wrote:

> This is specific to camel 2.11.0. Done file deletion worked as expected in
> 2.10.0
>  
> Scenario:
> I have a fileConsumer configured for a directory with done files. It is
> expected that every file dropped in the directory has a <filename>.done
> file.
>  
> In both 2.10.0 and 2.11.0 files are consumed successfully when copied over
> with the appropriate done file. However in 2.11.0, only the done file
> associated with the last file is deleted. All the other done files remain.
>  
> Looking at the code in GenericFileOnCompletion.processStrategyCommit(...)
> for 2.11.0, only the done file associated with the last file in batch is
> deleted.
>  
> It sounds like the intent was only one done file for a batch. But this is an
> incorrect assumption that does not line up with documentation. It might work
> when I have a non-specific done file name (e.g. start) when I want to
> process all files in a directory, but definitely won't work when I have a
> done file per input file.
>  
>  
>  
>  
>  
> --
> View this message in context: http://camel.465427.n5.nabble.com/Done-file-not-deleted-tp5735497.html
> Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).