[jira] [Created] (FLUME-2866) Add fileTimeMinOffsetSeconds property to Spooling Directory Source

["Philippe Lantin (JIRA)" <ji...@apache.org>]

Philippe Lantin created FLUME-2866:
--------------------------------------

             Summary: Add fileTimeMinOffsetSeconds property to Spooling Directory Source
                 Key: FLUME-2866
                 URL: https://issues.apache.org/jira/browse/FLUME-2866
             Project: Flume
          Issue Type: New Feature
          Components: Sinks+Sources
            Reporter: Philippe Lantin
            Priority: Minor


When using a spooling directory source, it would be useful to have the ability to specify that files must have a last modified timestamp that is off by a configurable amount of seconds, either in the future or the past.

For example, if I copy a large file to the spooling directory and it takes several minutes to copy, I do not want my file to started being processed before it is completed. A practical way to do this is by looking at the last modified timestamp: files that are being transferred will update this timestamp.

In many filesystems, it is possible for clients to set the time in the future, though this is usually done after a file has been completed transferred. For example "cp -p" on linux.

I propose a new property for the Spooling Directory Source: fileTimeMinOffsetSeconds. The default would be "0", preserving current behavior.

If fileTimeMinOffsetSeconds=60, files will only be picked up if the last modified time is +/- 60 seconds.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)