You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/10/15 06:47:47 UTC
[GitHub] [hadoop] vinayakumarb opened a new pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
vinayakumarb opened a new pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#discussion_r507473450
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java
##########
@@ -203,4 +203,27 @@ public void testPermission() throws Exception {
}
}
+ @Test
+ public void testMTimeAtime() throws IOException {
+ RawLocalFileSystem.setUseDeprecatedFileStatus(true);
+ try {
+ Path testDir = getTestBaseDir();
+ String testFilename = "testmtime";
+ Path path = new Path(testDir, testFilename);
+ Path file = new Path(path, "file");
+ fs.create(file);
+ long now = System.currentTimeMillis();
+ long mtime = (now % 1000 == 0) ? now + 1 : now;
+ long atime = (now % 1000 == 0) ? now + 2 : now;
+ fs.setTimes(file, mtime, atime);
+ FileStatus fileStatus = fs.getFileStatus(file);
+ if (!Shell.MAC) {
+ //HADOOP-17306 ; JDK-8177809 not fixed in MacOs
+ assertEquals(mtime, fileStatus.getModificationTime());
Review comment:
HFS+ does not support milliseconds for mtime, so I agreed to skip this in MacOS. In contrast, APFS supports nanoseconds and JDK-8177809 is fixed if the filesystem is APFS.
```
➜ hadoop git:(trunk) uname -a
Darwin <snip> 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
➜ hadoop git:(trunk) jshell
| Welcome to JShell -- Version 11.0.8
| For an introduction type: /help intro
jshell> Files.getLastModifiedTime(Paths.get("./pom.xml"))
$1 ==> 2020-10-19T04:51:58.69531Z
jshell> new File("./pom.xml").lastModified()
$1 ==> 1603083118695
```
Would you update the comment like "Skip MacOS because HFS+ does not support milliseconds for mtime." ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb merged pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
vinayakumarb merged pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#issuecomment-709125763
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 55s | | trunk passed |
| +1 :green_heart: | compile | 19m 53s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 59s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 49s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 18s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 16s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 19m 38s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 19m 38s | | the patch passed |
| +1 :green_heart: | compile | 17m 37s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 17m 37s | | the patch passed |
| -0 :warning: | checkstyle | 0m 47s | [/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) |
| +1 :green_heart: | mvnsite | 1m 39s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 56s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 2m 51s | | the patch passed |
|||| _ Other Tests _ |
| -1 :x: | unit | 10m 38s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 167m 33s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.metrics2.source.TestJvmMetrics |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2387 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 591e8a374231 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e45407128d4 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/2/testReport/ |
| Max. process+thread count | 1498 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on a change in pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on a change in pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#discussion_r507475468
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java
##########
@@ -203,4 +203,27 @@ public void testPermission() throws Exception {
}
}
+ @Test
+ public void testMTimeAtime() throws IOException {
+ RawLocalFileSystem.setUseDeprecatedFileStatus(true);
+ try {
+ Path testDir = getTestBaseDir();
+ String testFilename = "testmtime";
+ Path path = new Path(testDir, testFilename);
+ Path file = new Path(path, "file");
+ fs.create(file);
+ long now = System.currentTimeMillis();
+ long mtime = (now % 1000 == 0) ? now + 1 : now;
+ long atime = (now % 1000 == 0) ? now + 2 : now;
+ fs.setTimes(file, mtime, atime);
+ FileStatus fileStatus = fs.getFileStatus(file);
+ if (!Shell.MAC) {
+ //HADOOP-17306 ; JDK-8177809 not fixed in MacOs
+ assertEquals(mtime, fileStatus.getModificationTime());
Review comment:
Sure. Thanks for details @aajisaka
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#issuecomment-714933027
Merged to trunk. Thanks @ayushtkn and @aajisaka for reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#issuecomment-709274078
Thanx @vinayakumarb for the fix, the lgtm, but the test fails for me on `macOs`. Can you exclude the Os type?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#issuecomment-709399060
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 26s | | trunk passed |
| +1 :green_heart: | compile | 21m 11s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 27s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 19s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 16s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 21m 13s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 21m 13s | | the patch passed |
| +1 :green_heart: | compile | 18m 42s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 18m 42s | | the patch passed |
| -0 :warning: | checkstyle | 0m 47s | [/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/3/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 49s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 2m 26s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 50s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 174m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2387 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c9bb7e373194 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e45407128d4 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/3/testReport/ |
| Max. process+thread count | 1354 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2387/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#discussion_r507473450
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java
##########
@@ -203,4 +203,27 @@ public void testPermission() throws Exception {
}
}
+ @Test
+ public void testMTimeAtime() throws IOException {
+ RawLocalFileSystem.setUseDeprecatedFileStatus(true);
+ try {
+ Path testDir = getTestBaseDir();
+ String testFilename = "testmtime";
+ Path path = new Path(testDir, testFilename);
+ Path file = new Path(path, "file");
+ fs.create(file);
+ long now = System.currentTimeMillis();
+ long mtime = (now % 1000 == 0) ? now + 1 : now;
+ long atime = (now % 1000 == 0) ? now + 2 : now;
+ fs.setTimes(file, mtime, atime);
+ FileStatus fileStatus = fs.getFileStatus(file);
+ if (!Shell.MAC) {
+ //HADOOP-17306 ; JDK-8177809 not fixed in MacOs
+ assertEquals(mtime, fileStatus.getModificationTime());
Review comment:
HFS+ does not support milliseconds for mtime, so we have to skip this. In contrast, APFS supports nanoseconds and JDK-8177809 is fixed if the filesystem is APFS.
```
➜ hadoop git:(trunk) uname -a
Darwin <snip> 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
➜ hadoop git:(trunk) jshell
| Welcome to JShell -- Version 11.0.8
| For an introduction type: /help intro
jshell> Files.getLastModifiedTime(Paths.get("./pom.xml"))
$1 ==> 2020-10-19T04:51:58.69531Z
jshell> new File("./pom.xml").lastModified()
$1 ==> 1603083118695
```
Would you update the comment like "Skip MacOS because HFS+ does not support milliseconds for mtime." ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2387: HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2387:
URL: https://github.com/apache/hadoop/pull/2387#issuecomment-715027944
Thank you @vinayakumarb and @ayushtkn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org