[jira] [Created] (NIFI-6137) The Test for List File ListFile.filterAge fails

["Ryan Withers (JIRA)" <ji...@apache.org>]

Ryan Withers created NIFI-6137:
----------------------------------

             Summary: The Test for List File ListFile.filterAge fails 
                 Key: NIFI-6137
                 URL: https://issues.apache.org/jira/browse/NIFI-6137
             Project: Apache NiFi
          Issue Type: Bug
          Components: Extensions
    Affects Versions: 1.9.0
         Environment: My OS Version is:
 
18.2.0 Darwin Kernel Version 18.2.0: Fri Oct  5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64

I noticed that my root file system is apfs via mount -v  /dev/disk1s1 on / (apfs, local, journaled)

            Reporter: Ryan Withers
             Fix For: 1.10.0


Couple notes this is a unit test that fails consistently for me running on my local machine.  I know there is so much variability in build machine and environment that it might not make sense to log an issue like this.  I've asked the mailing list and had one person come back and verify that they have also encountered this similar issue.  I'm logging this so that I can work on it after I'm finished with one that I have in progress.  If I've made an error in logging the issue or it isn't consistent with community norms I'm happy to fix it.  Please see below for additional information.  

I'm encountering an error in the TestListFile.java class and in the testFilterAge() specifically.  This unit test and associated class can be found in the Nifi-Standard-Bundle.  I've noticed a couple issues have been reported on ListFile before. My issue is most closely related to this one: https://issues.apache.org/jira/browse/NIFI-3831.  After reading that one and considering the problem I'm thinking the problem I'm having is separate, but would be happy getting guidance on that. 

I have never seen this particular test pass before and I've attempted building 3 different versions of nifi now.  

In looking at the ListFiles.testFilterAge() and related methods I found a comment above the getTestModifiedTime talking about hfs+ only having granularity to one second.  For APFS this no longer seems to be the case based on the the link tagged below.  I've modified this method removing the minus 1 and it doesn't seem to have any affect on the failures I get.  I have included a validation that I'm on APFS in the environment window above.  I need to get closer to what is actually happening here. 

/*
 * HFS+, default for OS X, only has granularity to one second, accordingly, we go back in time to establish consistent test cases
 *
 * Provides "now" minus 1 second in millis
*/
private static long getTestModifiedTime() {
    final long nowMillis = System.currentTimeMillis();
    // Subtract a second to avoid possible rounding issues
    final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis, TimeUnit.MILLISECONDS) - 1;
    return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS);
}

https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/
 "Namely, APFS addresses file checksum, nanosecond time stamps, volume snapshots, and other features which were lacking under HFS+." 

All the file ranges seem shifted, but I'm surprised that nobody else has encountered this same issue.  I haven't raised this issue as a story yet because no one else has noticed the same problem.  I would think it would be a lot more common. 

In trying to gain additional understanding I've been modifying the unit test for TestListFile.testFilterAge() and I found incrementing the boundaries by one time range unit has allowed me to build a passing test case.   

         // exclude oldest
         runner.setProperty(ListFile.MIN_AGE, age0);
-        runner.setProperty(ListFile.MAX_AGE, age3);
+        runner.setProperty(ListFile.MAX_AGE, age4);
         runNext.apply(true);
         runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
         final List<MockFlowFile> successFiles2 = runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
@@ -362,7 +362,7 @@ public class TestListFile {
         assertEquals(file1.getName(), successFiles2.get(1).getAttribute("filename"));

         // exclude newest
-        runner.setProperty(ListFile.MIN_AGE, age1);
+        runner.setProperty(ListFile.MIN_AGE, age2);
         runner.setProperty(ListFile.MAX_AGE, age5);
         runNext.apply(true);
         runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
@@ -372,8 +372,8 @@ public class TestListFile {
         assertEquals(file2.getName(), successFiles3.get(1).getAttribute("filename"));

         // exclude oldest and newest
-        runner.setProperty(ListFile.MIN_AGE, age1);
-        runner.setProperty(ListFile.MAX_AGE, age3);
+        runner.setProperty(ListFile.MIN_AGE, age2);
+        runner.setProperty(ListFile.MAX_AGE, age4);
         runNext.apply(true);
         runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
         final List<MockFlowFile> successFiles4 = runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);

It makes me wonder if the generated timing is conflicting with the following line in ListFile.java

final long fileAge = System.currentTimeMillis() - attributes.lastModifiedTime().toMillis();

That line can be found on or around line 622 in the ListFile class in the createFileFilter method.  I'm thinking that utilizing System.currentTimeMillis() instead of providing some kind of shim where the current clock time can be reliably mocked is causing some inconsistencies here.  I'm not 100% confident in this because I would think that others would be encountering the same or similar issues, as I discussed above. 





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)