[jira] [Created] (NIFI-10142) ListSFTP assumes that time on SFTP server is the same as the time on local server

["Mark Payne (Jira)" <ji...@apache.org>]

Mark Payne created NIFI-10142:
---------------------------------

             Summary: ListSFTP assumes that time on SFTP server is the same as the time on local server
                 Key: NIFI-10142
                 URL: https://issues.apache.org/jira/browse/NIFI-10142
             Project: Apache NiFi
          Issue Type: Bug
          Components: Extensions
            Reporter: Mark Payne


The ListSFTP processor was updated in NIFI-5977 to add "Minimum File Age" and "Maximum File Age" properties. This was done in order to prevent ListSFTP from constantly listing files as they are being updated, and done in a way that is similar to ListFile.

However, with ListFile (at least when the file is local) we can be assured that the timestamp was generated on the system whose time is the same as the localhost's time.

For an SFTP server, we cannot assume that. So we should not be comparing local timestamp to the Last Modified time reported by the SFTP server.

We should deprecate the Max File Age property, and we should update the behavior such that if Min File Age is set to say 10 mins, we don't ingest a file until we see its Last Modified Time set to time t, and then at least 10 mins later see the Last Modified Time still set to t. This is the only way we can guarantee that the file has not been modified for at least 10 minutes.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)