Skipping as file is already in progress

[Ludovic Boué <lb...@afone.com>]

Hello,

I use camel to consume files from a server via SFTP with a application running on Apache Tomcat. Files remains on the server for archiving reasons. So I set noop=true and Camel set idempotent=true as well, to avoid consuming the same files over and over again.

Here are the versions that I use :
	Server version: Apache Tomcat/7.0.53
	OS Version:     2.6.18-371.8.1.el5
	Architecture:   amd64
	JVM Version:    1.7.0_51-b13
	JVM Vendor:     Oracle Corporation
	Camel 2.13.0

My idempotentRepository is defined here:
	<!-- this is our file based idempotent store configured to use the .filestore.dat as file -->
	<bean id="fileIdempotentRepositoryTickets" class="org.apache.camel.processor.idempotent.FileIdempotentRepository">
	    <!-- the filename for the store -->
	    <property name="fileStore" value="#{tickets['tickets.local.idempotentRepository.fileStore']}"/>
	    <!-- the max filesize in bytes for the file. Camel will trunk and flush the cache
	         if the file gets bigger -->
	    <property name="maxFileStoreSize" value="#{tickets['tickets.local.idempotentRepository.maxFileStoreSize']}"/>
	    <!-- the number of elements in our store -->
	    <property name="cacheSize" value="#{tickets['tickets.local.idempotentRepository.cacheSize']}"/>
	</bean>

My SftpConsumer is defined here:
ticket.Cirpack.CBV.address=10.0.34.**
ticket.Cirpack.CBV.username=user_mediation
ticket.Cirpack.CBV.password=***********
ticket.Cirpack.CBV.dir=/home/omni/tickets/CBV2
ticket.Cirpack.CBV.fileName=grnti_.*
ticket.Cirpack.CBV.dir.local=CirpackCBV


I don't understand why the SftpConsumer does not process files. So I set org.apache.camel.component.file to trace level to find what goes wrong.
	2014-06-18 17:48:32,518 DEBUG [SftpConsumer:doStart] Starting consumer: Consumer[sftp://user_mediation@10.0.34.**//home/omni/tickets/CBV2?consumer.delay=60000&disconnect=true&exclude=grnti_201406180000&include=grnti_.*&initialDelay=1000&noop=true&password=xxxxxx&stepwise=false]
	2014-06-18 17:48:32,523 DEBUG [FileConsumer:doStart] Starting consumer: Consumer[file:///home/rdn/tickets/IN/CirpackCBV?delay=10000&move=..%2F..%2FARC%2FCirpackCBV&moveFailed=..%2F..%2FERR%2FCirpackCBV&preMove=..%2F..%2FTRT%2FCirpackCBV]

We can see that grnti_201402150000 is a valid file but we skip it because it's as file is already in progress. But I can't find this file on disk or in my fileIdempotentRepositoryTickets (.ticketsFilestore.dat). How is this possible?

	2014-06-18 17:50:54,119 TRACE [SftpConsumer:doPollDirectory] SftpFile[fileName=.., longName=drwxr-xr-x    4 omni     omni         4096 Dec 19  2011 .., dir=true]
	2014-06-18 17:50:54,119 TRACE [SftpConsumer:doPollDirectory] SftpFile[fileName=grnti_201402150000, longName=-rw-r--r--    1 root     root     120882059 Feb 15 23:59 grnti_201402150000, dir=false]
	2014-06-18 17:50:54,119 TRACE [SftpConsumer:isValidFile] Skipping as file is already in progress: grnti_201402150000
	2014-06-18 17:50:54,119 TRACE [SftpConsumer:doPollDirectory] SftpFile[fileName=grnti_201406090000, longName=-rw-r--r--    1 root     root     416718575 Jun  9 23:59 grnti_201406090000, dir=false]
	2014-06-18 17:50:54,119 TRACE [SftpConsumer:isValidFile] Skipping as file is already in progress: grnti_201406090000
	2014-06-18 17:50:54,119 TRACE [SftpConsumer:doPollDirectory] SftpFile[fileName=grnti_201404080000, longName=-rw-r--r--    1 root     root     498040703 Apr  8 23:59 grnti_201404080000, dir=false]

How do I figure out what's wrong? How can I clean this progress list?

Regards,
Ludovic