[jira] [Commented] (NIFI-4069) ListXXX processors can miss files those created while the processor is listing and filesystem does not provide timestamp milliseconds precision

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143899#comment-16143899 ] 

ASF subversion and git services commented on NIFI-4069:
-------------------------------------------------------

Commit 28ee70222b892fb799f5f74a31a9de678d9fb629 in nifi's branch refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=28ee702 ]

NIFI-4069: Make ListXXX work with timestamp precision in seconds or minutes

- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test because Java File API reports timestamp in milliseconds at the best granularity. Also, System.nanoTime should not be used in mix with epoch milliseconds because it uses arbitrary origin and measured differently.
- Changed TestListFile to use more longer interval between file timestamps those are used by testFilterAge to provide more consistent test result because sleep time can be longer with filesystems whose timestamp in seconds precision.
- Added logging at TestListFile.
- Added TestWatcher to dump state in case assertion fails for further investigation.
- Added Timestamp Precision property so that user can set if auto-detect is not enough
- Adjust timestamps for ages test

This closes #1915.

Signed-off-by: Bryan Bende <bb...@apache.org>


> ListXXX processors can miss files those created while the processor is listing and filesystem does not provide timestamp milliseconds precision
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-4069
>                 URL: https://issues.apache.org/jira/browse/NIFI-4069
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>    Affects Versions: 1.0.0
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
>         Attachments: ListFilesWithoutMilliseconds.png
>
>
> For some filesystems such as Mac OS X HFS (Hierarchical File System) or EXT3 are known that only support timestamp in seconds precision. Also some FTP server is reported that it can only provides timestamp precision in minutes.
> This can cause files to NOT be listed as ListXXX processors logic expects timestamps in milliseconds.
> Specifically, if generate several files in one second, not all files will be listened.
> Steps to reproduce:
> 1. start processor ListFile
> 2. generate 10000 zero size files with following command:
> {code}
> for i in {1..10000}; do touch ./test_$i; done
> {code}
> 3. see processor stats: out 3952 (0 bytes)
> Current AbstractListProcessor logic adopts LISTING_LAG_NANOS (100ms) and postponing the files those have the latest timestamp within a listing iteration to next iteration, however with those filesystem without milliseconds precision, these logics do not work as expected.
> This issue is originally reported at nifi-dev ML. http://apache-nifi-developer-list.39713.n7.nabble.com/processors-ListFile-ListSFTP-do-not-store-milliseconds-in-timestamp-td16037.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)