You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Ryan Withers <wi...@gmail.com> on 2019/03/14 22:41:15 UTC
Question about ListFile Test
Hello all,
I'm encountering an error in the TestListFile.java class and in the
testFilterAge() specifically. This unit test and associated class can be
found in the Nifi-Standard-Bundle. I've noticed a couple issues have been
reported on ListFile before. My issue is most closely related to this one:
https://issues.apache.org/jira/browse/NIFI-3831, although it may be an
issue altogether separate. I have never seen this particular test pass
before and I've attempted building several versions of nifi now.
I'm doing the nifi build on a new model mac. My OS Version is:
18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018;
root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64
I noticed that my root file system is apfs via mount -v */dev/disk1s1 on /
(apfs, local, journaled)*
In looking at the ListFiles.testFilterAge() and related methods I found a
comment above the getTestModifiedTime talking about hfs+ only having
granularity to one second. For APFS this no longer seems to be the case
based on the the link tagged below. I've modified this method removing the
minus 1 and it doesn't seem to have any affect on the failures I get.
/*
* HFS+, default for OS X, only has granularity to one second,
accordingly, we go back in time to establish consistent test cases
*
* Provides "now" minus 1 second in millis
*/
private static long getTestModifiedTime() {
final long nowMillis = System.currentTimeMillis();
// Subtract a second to avoid possible rounding issues
final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis,
TimeUnit.MILLISECONDS) - 1;
return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS);
}
https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/
"Namely, APFS addresses file checksum, nanosecond time stamps, volume
snapshots, and other features which were lacking under HFS+."
All the file ranges seem shifted, but I'm surprised that nobody else has
encountered this same issue. I haven't raised this issue as a story yet
because no one else has noticed the same problem.
In trying to gain additional understanding I've been modifying the unit
test for testFilterAge and I found incrementing the boundaries by one time
range unit has allowed me to build a passing test case.
// exclude oldest
runner.setProperty(ListFile.MIN_AGE, age0);
- runner.setProperty(ListFile.MAX_AGE, age3);
+ runner.setProperty(ListFile.MAX_AGE, age4);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
final List<MockFlowFile> successFiles2 =
runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
@@ -362,7 +362,7 @@ public class TestListFile {
assertEquals(file1.getName(),
successFiles2.get(1).getAttribute("filename"));
// exclude newest
- runner.setProperty(ListFile.MIN_AGE, age1);
+ runner.setProperty(ListFile.MIN_AGE, age2);
runner.setProperty(ListFile.MAX_AGE, age5);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
@@ -372,8 +372,8 @@ public class TestListFile {
assertEquals(file2.getName(),
successFiles3.get(1).getAttribute("filename"));
// exclude oldest and newest
- runner.setProperty(ListFile.MIN_AGE, age1);
- runner.setProperty(ListFile.MAX_AGE, age3);
+ runner.setProperty(ListFile.MIN_AGE, age2);
+ runner.setProperty(ListFile.MAX_AGE, age4);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
final List<MockFlowFile> successFiles4 =
runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
Has anyone else encountered this? Are there special platform specific
flags I need to pass when running on a mac? It makes me wonder if the
generated timing is conflicting with the following line in ListFile.java
final long fileAge = System.currentTimeMillis() -
attributes.lastModifiedTime().toMillis();
That line can be found on line 622 in the ListFile class in the
createFileFilter method. I'm thinking that utilizing
System.currentTImeMillis() instead of providing some kind of shim where the
current clock time can be reliably mocked is causing some inconsistencies
here. I'm not 100% confident in this because I would think that others
would be encountering the same or similar issues.
Any help would be greatly appreciated. I'm also happy to clarify if
anything seems unclear.
Thanks in advance,
--
Ryan Withers
Senior Software Developer / Analyst
http://www.linkedin.com/in/ryanwithers
Re: Question about ListFile Test
Posted by Ryan Withers <wi...@gmail.com>.
Bryan,
That's what I was looking for. Someone else that had encountered the
error in the ListFile.filterAge() unit test. That way I know I'm not
crazy. I think I have a pretty good idea of what is causing the
unpredictable behavior I'm going to continue verifying today.
Thanks for your help,
Ryan
On Thu, Mar 14, 2019 at 7:15 PM Bryan Rosander <br...@gmail.com>
wrote:
> I've seen similar issues building locally on OSx. I tried going back and
> building 1.8.0 and saw similar behavior as master.
>
> Haven't been able to sort it out yet though.
>
> Thanks,
> Bryan
>
> On Thu, Mar 14, 2019 at 6:41 PM Ryan Withers <wi...@gmail.com>
> wrote:
>
> > Hello all,
> >
> > I'm encountering an error in the TestListFile.java class and in the
> > testFilterAge() specifically. This unit test and associated class can be
> > found in the Nifi-Standard-Bundle. I've noticed a couple issues have
> been
> > reported on ListFile before. My issue is most closely related to this
> one:
> > https://issues.apache.org/jira/browse/NIFI-3831, although it may be an
> > issue altogether separate. I have never seen this particular test pass
> > before and I've attempted building several versions of nifi now.
> >
> > I'm doing the nifi build on a new model mac. My OS Version is:
> >
> > 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018;
> > root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64
> >
> > I noticed that my root file system is apfs via mount -v */dev/disk1s1
> on /
> > (apfs, local, journaled)*
> >
> > In looking at the ListFiles.testFilterAge() and related methods I found a
> > comment above the getTestModifiedTime talking about hfs+ only having
> > granularity to one second. For APFS this no longer seems to be the case
> > based on the the link tagged below. I've modified this method removing
> the
> > minus 1 and it doesn't seem to have any affect on the failures I get.
> >
> > /*
> > * HFS+, default for OS X, only has granularity to one second,
> > accordingly, we go back in time to establish consistent test cases
> > *
> > * Provides "now" minus 1 second in millis
> > */
> > private static long getTestModifiedTime() {
> > final long nowMillis = System.currentTimeMillis();
> > // Subtract a second to avoid possible rounding issues
> > final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis,
> > TimeUnit.MILLISECONDS) - 1;
> > return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS);
> > }
> >
> > https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/
> > "Namely, APFS addresses file checksum, nanosecond time stamps, volume
> > snapshots, and other features which were lacking under HFS+."
> >
> > All the file ranges seem shifted, but I'm surprised that nobody else has
> > encountered this same issue. I haven't raised this issue as a story yet
> > because no one else has noticed the same problem.
> >
> > In trying to gain additional understanding I've been modifying the unit
> > test for testFilterAge and I found incrementing the boundaries by one
> time
> > range unit has allowed me to build a passing test case.
> >
> > // exclude oldest
> > runner.setProperty(ListFile.MIN_AGE, age0);
> > - runner.setProperty(ListFile.MAX_AGE, age3);
> > + runner.setProperty(ListFile.MAX_AGE, age4);
> > runNext.apply(true);
> > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> > final List<MockFlowFile> successFiles2 =
> > runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
> > @@ -362,7 +362,7 @@ public class TestListFile {
> > assertEquals(file1.getName(),
> > successFiles2.get(1).getAttribute("filename"));
> >
> > // exclude newest
> > - runner.setProperty(ListFile.MIN_AGE, age1);
> > + runner.setProperty(ListFile.MIN_AGE, age2);
> > runner.setProperty(ListFile.MAX_AGE, age5);
> > runNext.apply(true);
> > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> > @@ -372,8 +372,8 @@ public class TestListFile {
> > assertEquals(file2.getName(),
> > successFiles3.get(1).getAttribute("filename"));
> >
> > // exclude oldest and newest
> > - runner.setProperty(ListFile.MIN_AGE, age1);
> > - runner.setProperty(ListFile.MAX_AGE, age3);
> > + runner.setProperty(ListFile.MIN_AGE, age2);
> > + runner.setProperty(ListFile.MAX_AGE, age4);
> > runNext.apply(true);
> > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> > final List<MockFlowFile> successFiles4 =
> > runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
> >
> > Has anyone else encountered this? Are there special platform specific
> > flags I need to pass when running on a mac? It makes me wonder if the
> > generated timing is conflicting with the following line in ListFile.java
> >
> > final long fileAge = System.currentTimeMillis() -
> > attributes.lastModifiedTime().toMillis();
> >
> >
> > That line can be found on line 622 in the ListFile class in the
> > createFileFilter method. I'm thinking that utilizing
> > System.currentTImeMillis() instead of providing some kind of shim where
> the
> > current clock time can be reliably mocked is causing some inconsistencies
> > here. I'm not 100% confident in this because I would think that others
> > would be encountering the same or similar issues.
> >
> > Any help would be greatly appreciated. I'm also happy to clarify if
> > anything seems unclear.
> >
> > Thanks in advance,
> >
> > --
> > Ryan Withers
> > Senior Software Developer / Analyst
> >
> > http://www.linkedin.com/in/ryanwithers
> >
>
--
Ryan Withers
Senior Software Developer / Analyst
http://www.linkedin.com/in/ryanwithers
Re: Question about ListFile Test
Posted by Bryan Rosander <br...@gmail.com>.
I've seen similar issues building locally on OSx. I tried going back and
building 1.8.0 and saw similar behavior as master.
Haven't been able to sort it out yet though.
Thanks,
Bryan
On Thu, Mar 14, 2019 at 6:41 PM Ryan Withers <wi...@gmail.com> wrote:
> Hello all,
>
> I'm encountering an error in the TestListFile.java class and in the
> testFilterAge() specifically. This unit test and associated class can be
> found in the Nifi-Standard-Bundle. I've noticed a couple issues have been
> reported on ListFile before. My issue is most closely related to this one:
> https://issues.apache.org/jira/browse/NIFI-3831, although it may be an
> issue altogether separate. I have never seen this particular test pass
> before and I've attempted building several versions of nifi now.
>
> I'm doing the nifi build on a new model mac. My OS Version is:
>
> 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018;
> root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64
>
> I noticed that my root file system is apfs via mount -v */dev/disk1s1 on /
> (apfs, local, journaled)*
>
> In looking at the ListFiles.testFilterAge() and related methods I found a
> comment above the getTestModifiedTime talking about hfs+ only having
> granularity to one second. For APFS this no longer seems to be the case
> based on the the link tagged below. I've modified this method removing the
> minus 1 and it doesn't seem to have any affect on the failures I get.
>
> /*
> * HFS+, default for OS X, only has granularity to one second,
> accordingly, we go back in time to establish consistent test cases
> *
> * Provides "now" minus 1 second in millis
> */
> private static long getTestModifiedTime() {
> final long nowMillis = System.currentTimeMillis();
> // Subtract a second to avoid possible rounding issues
> final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis,
> TimeUnit.MILLISECONDS) - 1;
> return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS);
> }
>
> https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/
> "Namely, APFS addresses file checksum, nanosecond time stamps, volume
> snapshots, and other features which were lacking under HFS+."
>
> All the file ranges seem shifted, but I'm surprised that nobody else has
> encountered this same issue. I haven't raised this issue as a story yet
> because no one else has noticed the same problem.
>
> In trying to gain additional understanding I've been modifying the unit
> test for testFilterAge and I found incrementing the boundaries by one time
> range unit has allowed me to build a passing test case.
>
> // exclude oldest
> runner.setProperty(ListFile.MIN_AGE, age0);
> - runner.setProperty(ListFile.MAX_AGE, age3);
> + runner.setProperty(ListFile.MAX_AGE, age4);
> runNext.apply(true);
> runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> final List<MockFlowFile> successFiles2 =
> runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
> @@ -362,7 +362,7 @@ public class TestListFile {
> assertEquals(file1.getName(),
> successFiles2.get(1).getAttribute("filename"));
>
> // exclude newest
> - runner.setProperty(ListFile.MIN_AGE, age1);
> + runner.setProperty(ListFile.MIN_AGE, age2);
> runner.setProperty(ListFile.MAX_AGE, age5);
> runNext.apply(true);
> runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> @@ -372,8 +372,8 @@ public class TestListFile {
> assertEquals(file2.getName(),
> successFiles3.get(1).getAttribute("filename"));
>
> // exclude oldest and newest
> - runner.setProperty(ListFile.MIN_AGE, age1);
> - runner.setProperty(ListFile.MAX_AGE, age3);
> + runner.setProperty(ListFile.MIN_AGE, age2);
> + runner.setProperty(ListFile.MAX_AGE, age4);
> runNext.apply(true);
> runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> final List<MockFlowFile> successFiles4 =
> runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
>
> Has anyone else encountered this? Are there special platform specific
> flags I need to pass when running on a mac? It makes me wonder if the
> generated timing is conflicting with the following line in ListFile.java
>
> final long fileAge = System.currentTimeMillis() -
> attributes.lastModifiedTime().toMillis();
>
>
> That line can be found on line 622 in the ListFile class in the
> createFileFilter method. I'm thinking that utilizing
> System.currentTImeMillis() instead of providing some kind of shim where the
> current clock time can be reliably mocked is causing some inconsistencies
> here. I'm not 100% confident in this because I would think that others
> would be encountering the same or similar issues.
>
> Any help would be greatly appreciated. I'm also happy to clarify if
> anything seems unclear.
>
> Thanks in advance,
>
> --
> Ryan Withers
> Senior Software Developer / Analyst
>
> http://www.linkedin.com/in/ryanwithers
>