You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by borgel <bo...@gmail.com> on 2008/09/15 11:14:17 UTC
Ftp consumer
I have a route that polls a directory on a ftp server. The files are not
always downloaded. It seems like if I create a file locally and then waits a
while before uploading to the ftp server, the file is not consumed by Camel.
Could it be that the file don't get a new time stamp when uploaded and Camel
ignores it?
In Camel 1.5 there is a new property; consumer.deleteFile. If this is set to
true I think Camel should download and delete the file without looking at
the timestamp.
Btw: Great functionality in Camel 1.5 ftp and file handling!
--
View this message in context: http://www.nabble.com/Ftp-consumer-tp19489526s22882p19489526.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Ftp consumer
Posted by Claus Ibsen <ci...@silverbullet.dk>.
Hi
I have committed a fix in trunk.
Can you test it with the latest camel 1.5-SNAPSHOT. And report back here. That would be awesome.
The timestamp stuff is now disabled by default. So you should always consume files on the FTP server regardless of their local modified timestamp. To not consume the same file twice or more you should use the consume.deleteFile=true option.
There is an option to enabled the old behavior. However its @depreacted and will be removed in Camel 2.0
See the JIRA ticket for more details.
Med venlig hilsen
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Claus Ibsen [mailto:ci@silverbullet.dk]
Sent: 15. september 2008 13:03
To: camel-user@activemq.apache.org
Subject: RE: Ftp consumer
Hi
Yes the bug is still there. I will fix it in CAMEL-903.
Changing the delay to > 60 sec should NOT fix it, even if the clocks on both servers is 100% match
Step 1: poll at 09:45:13 => 0 files
Step 2: FTP server creates a new file: myfile 09:45:22
Step 3: poll at 09:46:13 => 1 files
FTP server LIST returns listing: myfile 09:45
Camel interprets this time as: 09:45:00 but the timestamp from step 1 is 09:45:13 and thus Camel do NOT see this as a NEW file
The only solution is to set the clock on the Camel server to be 60+ seconds behind from the FTP Server.
AFAIR read the code.
Med venlig hilsen
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: borgel [mailto:borge.lotre@gmail.com]
Sent: 15. september 2008 12:48
To: camel-user@activemq.apache.org
Subject: RE: Ftp consumer
Hi
I am using 1.5-SNAPSHOT and have set consumer.deleteFile=true. Files are
still missed.
I had consumer.delay set to only 30 sec and this is probably the reason for
the misses then. I will try to set consumer.delay to more than one minute.
Claus Ibsen wrote:
>
> Hi
>
> Yes the ftp consumer uses by default timestamp for discovering new files.
> However timestamps on some (maybe most or all) FTP servers sent using the
> LIST command is not always precise to the second - only HH:MM (no
> seconds).
>
> (...)
>
> A different strategy is encouraged, such as renaming or deleting the files
> after download. These features are now provided in Camel 1.5 that however
> hasn't been released yet.
>
>
>
--
View this message in context: http://www.nabble.com/Ftp-consumer-tp19489526s22882p19490717.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Ftp consumer
Posted by Claus Ibsen <ci...@silverbullet.dk>.
Hi
Yes the bug is still there. I will fix it in CAMEL-903.
Changing the delay to > 60 sec should NOT fix it, even if the clocks on both servers is 100% match
Step 1: poll at 09:45:13 => 0 files
Step 2: FTP server creates a new file: myfile 09:45:22
Step 3: poll at 09:46:13 => 1 files
FTP server LIST returns listing: myfile 09:45
Camel interprets this time as: 09:45:00 but the timestamp from step 1 is 09:45:13 and thus Camel do NOT see this as a NEW file
The only solution is to set the clock on the Camel server to be 60+ seconds behind from the FTP Server.
AFAIR read the code.
Med venlig hilsen
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: borgel [mailto:borge.lotre@gmail.com]
Sent: 15. september 2008 12:48
To: camel-user@activemq.apache.org
Subject: RE: Ftp consumer
Hi
I am using 1.5-SNAPSHOT and have set consumer.deleteFile=true. Files are
still missed.
I had consumer.delay set to only 30 sec and this is probably the reason for
the misses then. I will try to set consumer.delay to more than one minute.
Claus Ibsen wrote:
>
> Hi
>
> Yes the ftp consumer uses by default timestamp for discovering new files.
> However timestamps on some (maybe most or all) FTP servers sent using the
> LIST command is not always precise to the second - only HH:MM (no
> seconds).
>
> (...)
>
> A different strategy is encouraged, such as renaming or deleting the files
> after download. These features are now provided in Camel 1.5 that however
> hasn't been released yet.
>
>
>
--
View this message in context: http://www.nabble.com/Ftp-consumer-tp19489526s22882p19490717.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Ftp consumer
Posted by borgel <bo...@gmail.com>.
Hi
I am using 1.5-SNAPSHOT and have set consumer.deleteFile=true. Files are
still missed.
I had consumer.delay set to only 30 sec and this is probably the reason for
the misses then. I will try to set consumer.delay to more than one minute.
Claus Ibsen wrote:
>
> Hi
>
> Yes the ftp consumer uses by default timestamp for discovering new files.
> However timestamps on some (maybe most or all) FTP servers sent using the
> LIST command is not always precise to the second - only HH:MM (no
> seconds).
>
> (...)
>
> A different strategy is encouraged, such as renaming or deleting the files
> after download. These features are now provided in Camel 1.5 that however
> hasn't been released yet.
>
>
>
--
View this message in context: http://www.nabble.com/Ftp-consumer-tp19489526s22882p19490717.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Ftp consumer
Posted by Claus Ibsen <ci...@silverbullet.dk>.
BTW:
I have created a ticket: CAMEL-903 to fix the timestamp issue.
Bottom line:
Don't use timestamp comparison for matching new files. It's not reliable over the FTP protocol.
Med venlig hilsen
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Claus Ibsen [mailto:ci@silverbullet.dk]
Sent: 15. september 2008 11:24
To: camel-user@activemq.apache.org
Subject: RE: Ftp consumer
Hi
Yes the ftp consumer uses by default timestamp for discovering new files. However timestamps on some (maybe most or all) FTP servers sent using the LIST command is not always precise to the second - only HH:MM (no seconds).
In that regard Camel is not that smart as it uses System.currentTimeMillis for "last run", and thus with bad luck could miss a file.
I don't encourage using this "timestamp" feature. For instance, if you restart Camel then it will set the timestamp to 0 and thus download all files again. Maybe I should write something about this on the wiki. Also timezone is not considered as well.
A different strategy is encouraged, such as renaming or deleting the files after download. These features are now provided in Camel 1.5 that however hasn't been released yet.
Med venlig hilsen
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: borgel [mailto:borge.lotre@gmail.com]
Sent: 15. september 2008 11:14
To: camel-user@activemq.apache.org
Subject: Ftp consumer
I have a route that polls a directory on a ftp server. The files are not
always downloaded. It seems like if I create a file locally and then waits a
while before uploading to the ftp server, the file is not consumed by Camel.
Could it be that the file don't get a new time stamp when uploaded and Camel
ignores it?
In Camel 1.5 there is a new property; consumer.deleteFile. If this is set to
true I think Camel should download and delete the file without looking at
the timestamp.
Btw: Great functionality in Camel 1.5 ftp and file handling!
--
View this message in context: http://www.nabble.com/Ftp-consumer-tp19489526s22882p19489526.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Ftp consumer
Posted by Claus Ibsen <ci...@silverbullet.dk>.
Hi
Yes the ftp consumer uses by default timestamp for discovering new files. However timestamps on some (maybe most or all) FTP servers sent using the LIST command is not always precise to the second - only HH:MM (no seconds).
In that regard Camel is not that smart as it uses System.currentTimeMillis for "last run", and thus with bad luck could miss a file.
I don't encourage using this "timestamp" feature. For instance, if you restart Camel then it will set the timestamp to 0 and thus download all files again. Maybe I should write something about this on the wiki. Also timezone is not considered as well.
A different strategy is encouraged, such as renaming or deleting the files after download. These features are now provided in Camel 1.5 that however hasn't been released yet.
Med venlig hilsen
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: borgel [mailto:borge.lotre@gmail.com]
Sent: 15. september 2008 11:14
To: camel-user@activemq.apache.org
Subject: Ftp consumer
I have a route that polls a directory on a ftp server. The files are not
always downloaded. It seems like if I create a file locally and then waits a
while before uploading to the ftp server, the file is not consumed by Camel.
Could it be that the file don't get a new time stamp when uploaded and Camel
ignores it?
In Camel 1.5 there is a new property; consumer.deleteFile. If this is set to
true I think Camel should download and delete the file without looking at
the timestamp.
Btw: Great functionality in Camel 1.5 ftp and file handling!
--
View this message in context: http://www.nabble.com/Ftp-consumer-tp19489526s22882p19489526.html
Sent from the Camel - Users mailing list archive at Nabble.com.