GenericFileOperationFailedException - 'Cannot rename file' on Linux

["Shiraz.Iskenderian" <Sh...@reece.com.au>]

Hi,

We have an integration system based on Camel v2.16.1 that runs on a Jboss v6
Linux platform.  There are multiple interfaces running simultaneously each
with a different polling rate.

We are intermittently experiencing 'Cannot rename file' issue with Camel
failing to backup to the 'done' folder successfully processed and
transmitted files from the ftp source.  Restarting the camel application
fixes the issue.

Basically, at regular intervals triggered by a quartz scheduler, the route: 
1) picks up files from a source via ftp,
2) processes them, smooks + xsl transformations
3) delivers the generated flat file to an endpoint via ftp.

If multiple files are read from the source directory, then all the files are
appended together in a temporary file before being processed.

The Camel ftp configuration uses the following url:

ftp://xxxx/export?antInclude=dsciord_*.dat&inProgressRepository=#warehouseIntegrationIdempotentRepository&preMove=in_progress_bpo/$simple{date:now:yyyyMMddHHmm}/$simple{file:name}&move=done&consumer.bridgeErrorHandler=true

a) read files dsciord_*.dat from /export directory
b) use custom inprogressRepository to store the read filename into a local
db (this was done to prevent contention issue with a second cluster node,
however, currently only a single node is live.  This option is unnecessary
and can be removed speeding up the process).
c) move files to an in_progress_bpo/201609061522 directory, where the
subdirectory is created based on the date_timestamp.  
d) move them to the in_progress_bpo/201609061522/done subdirectory once
successfully processed.

In vast majority of cases the route works with no issues, however, sometimes
the file(s) cannot be moved to the done folder (see error below).  Even in
this case, the route can sometimes continue successfully at the next polling
cycle, however, in other cases the route enters a state when even if the
quartz scheduler triggers the poll, the route fails to detect any files in
the source /export directory even when there ARE files there.

org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
rename file: RemoteFile[in_progress_bpo/201609060502/dsciord_3605752.dat]
to: RemoteFile[in_progress_bpo/201609060502/done/dsciord_3605752.dat]

Notes: We are using 
1) a single instance of a ConsumerTemplate to handle our interfaces.
2) a custom inprogressRepository to store the file names read.

Obviously, there must be a system locking the source files and this is
causing the Camel route to stop processing further files.

Any ideas/suggestions on debugging/resolving this issue would be greatly
appreciated.  The issues that I read through the camel-users forum seem to
deal with Windows-related deployments, sometimes Smooks failing to close the
input stream.  I've check and we don't use the 
org.milyn.templating.xslt.XslTemplateProcessor#bypass method where Smooks
fails to close the underlying input stream.

Thanks in advance,
Shiraz Iskenderian
Melbourne, Australia



--
View this message in context: http://camel.465427.n5.nabble.com/GenericFileOperationFailedException-Cannot-rename-file-on-Linux-tp5787225.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: GenericFileOperationFailedException - 'Cannot rename file' on Linux

["Shiraz.Iskenderian" <Sh...@reece.com.au>]

I ended up NOT using Camel consumerTemplate.  Instead used a custom approach
of reading files from a remote ftp server.  Quicker, reliable and allows the
option of specifying absolute paths!



--
View this message in context: http://camel.465427.n5.nabble.com/GenericFileOperationFailedException-Cannot-rename-file-on-Linux-tp5787225p5787714.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: GenericFileOperationFailedException - 'Cannot rename file' on Linux

["Shiraz.Iskenderian" <Sh...@reece.com.au>]

Finally I have been able to reproduce/identify the issue.  

Given that we are using a relative path to move the processed files into
once successfully ftp-ed to the destination servers:

../../../u/4gl_upload/warehouse_integration_2/trs-server/export/in_progress_bpo/201609081030/done

However, for some reason instead of traversing the via correct path to move
the processed files the camel consumer creates a new subdirectory tree
starting from where it is and this could be quite long as follows.  Hence
the problem.  It doesn’t know where it is and it doesn’t reset itself.

/u/4gl_upload/warehouse_integration_2/trs-server/u/4gl_upload/warehouse_integration_2/trs-server/export/in_progress_bpo/201609081030

This was reproduced with the option stepwise=false, which means it traverses
the subdirectories in a single step instead of step wise.

Still don’t know what best solution is.





--
View this message in context: http://camel.465427.n5.nabble.com/GenericFileOperationFailedException-Cannot-rename-file-on-Linux-tp5787225p5787338.html
Sent from the Camel - Users mailing list archive at Nabble.com.