You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/01/12 15:14:44 UTC

[GitHub] [hadoop] amahussein opened a new pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

amahussein opened a new pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615


   Replace both `System.currentTimeMillis()` and `Time.now()` with `Time.monotonicNow()` to measure elapsed time.
   Code changes are minimal to reduce the size of the patch.
   I intentionally left the check style errors in `hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java` because those would require reformatting the entire file, increasing the patch and confusing the reviewer.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] ayushtkn edited a comment on pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
ayushtkn edited a comment on pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#issuecomment-767045170


   There was a discussion about this earlier too, changing Time.now to Time.monotonicNow and it has a -1
   https://issues.apache.org/jira/browse/HADOOP-15901


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein commented on pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#issuecomment-855020066


   I haven't seen the discussion when I searched for old jiras.
   Thanks @jbrennan333  and @ayushtkn for the feedback.
   
   Closing the PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] amahussein closed pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
amahussein closed pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] jbrennan333 commented on a change in pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
jbrennan333 commented on a change in pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#discussion_r645071212



##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -620,7 +621,7 @@ private static String getName(IdCache domain, int id) throws IOException {
         ? USER_ID_NAME_CACHE : GROUP_ID_NAME_CACHE;
       String name;
       CachedName cachedName = idNameCache.get(id);
-      long now = System.currentTimeMillis();
+      long now = Time.monotonicNow();
       if (cachedName != null && (cachedName.timestamp + cacheTimeout) > now) {

Review comment:
       When switching from System.currentTimeMillis, or Time.now() to Time.monotonicNow(), keep in mind that the latter can return a negative number.   In this case, cachedName.timestamp could be negative due to overflow, while now might still be positive, in which case this code will not take the right path.  The correct comparison would be `((now - cachedName.timestamp) < cacheTimeout)`
   But I would also avoid using monitonicNow for any field called `timestamp`.
   
   

##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestReconfiguration.java
##########
@@ -317,8 +317,8 @@ public void testThread() throws ReconfigurationException {
     }
     dummy.reconfigureProperty(PROP1, VAL2);
 
-    long endWait = Time.now() + 2000;
-    while (dummyThread.isAlive() && Time.now() < endWait) {
+    long endWait = Time.monotonicNow() + 2000;
+    while (dummyThread.isAlive() && Time.monotonicNow() < endWait) {

Review comment:
       Another dangerous comparison.

##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
##########
@@ -419,7 +419,7 @@ public RpcErrorCodeProto getRpcErrorCodeProto() {
         ioe = nrthe;
       }
       // check if timed out
-      if (Time.now()-timeout >= startTime) {
+      if (Time.monotonicNow() - timeout >= startTime) {

Review comment:
       Similarly, I think `Time.monotonicNow() - startTime >= timeout` would be better here.

##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/CacheableIPList.java
##########
@@ -48,7 +48,7 @@ private void updateCacheExpiryTime() {
     if (cacheTimeout < 0) {
       cacheExpiryTimeStamp = -1; // no automatic cache expiry.
     }else {
-      cacheExpiryTimeStamp = System.currentTimeMillis() + cacheTimeout;
+      cacheExpiryTimeStamp = Time.monotonicNow() + cacheTimeout;

Review comment:
       Timestamps should use currentTimeMillis or similar.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] ayushtkn commented on pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#issuecomment-767045170


   There was a discussion about this earlier too, changing Time.now to Time.monotonicNow
   https://issues.apache.org/jira/browse/HADOOP-15901


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] ayushtkn commented on pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#issuecomment-767045170


   There was a discussion about this earlier too, changing Time.now to Time.monotonicNow
   https://issues.apache.org/jira/browse/HADOOP-15901


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#issuecomment-758819835


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 32s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch appears to include 23 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |   4m 23s |  |  Maven dependency ordering for branch  |
   | -1 :x: |  mvninstall  |   6m 25s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/artifact/out/branch-mvninstall-root.txt) |  root in trunk failed.  |
   | -1 :x: |  compile  |   6m 55s | [/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/artifact/out/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) |  root in trunk failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.  |
   | -1 :x: |  compile  |   5m 16s | [/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) |  root in trunk failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.  |
   | +1 :green_heart: |  checkstyle  |   1m  9s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 38s |  |  trunk passed  |
   | -1 :x: |  shadedclient  |  18m 33s |  |  branch has errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 15s |  |  trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   1m 38s |  |  trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   0m 38s |  |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 39s |  |  trunk passed  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   6m  7s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   1m 16s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 17s |  |  the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | -1 :x: |  javac  |  20m 17s | [/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) |  root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 820 new + 1215 unchanged - 0 fixed = 2035 total (was 1215)  |
   | +1 :green_heart: |  compile  |  17m 23s |  |  the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | -1 :x: |  javac  |  17m 23s | [/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) |  root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 760 new + 1170 unchanged - 0 fixed = 1930 total (was 1170)  |
   | -0 :warning: |  checkstyle  |   1m 25s | [/diff-checkstyle-hadoop-common-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/artifact/out/diff-checkstyle-hadoop-common-project.txt) |  hadoop-common-project: The patch generated 8 new + 1155 unchanged - 14 fixed = 1163 total (was 1169)  |
   | +1 :green_heart: |  mvnsite  |   2m 14s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  15m 20s |  |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 41s |  |  the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   2m 14s |  |  the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   3m 24s |  |  the patch passed  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 38s |  |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  unit  |   3m 41s |  |  hadoop-kms in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 138m 21s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2615 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux 325f3fb52227 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 85b1c017eed |
   | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/testReport/ |
   | Max. process+thread count | 3416 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2615/1/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] ayushtkn edited a comment on pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
ayushtkn edited a comment on pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#issuecomment-767045170


   There was a discussion about this earlier too, changing Time.now to Time.monotonicNow and it has a -1
   https://issues.apache.org/jira/browse/HADOOP-15901


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] jbrennan333 commented on a change in pull request #2615: HADOOP-17463. use monotonicNow for elapsed time.

Posted by GitBox <gi...@apache.org>.
jbrennan333 commented on a change in pull request #2615:
URL: https://github.com/apache/hadoop/pull/2615#discussion_r645071212



##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -620,7 +621,7 @@ private static String getName(IdCache domain, int id) throws IOException {
         ? USER_ID_NAME_CACHE : GROUP_ID_NAME_CACHE;
       String name;
       CachedName cachedName = idNameCache.get(id);
-      long now = System.currentTimeMillis();
+      long now = Time.monotonicNow();
       if (cachedName != null && (cachedName.timestamp + cacheTimeout) > now) {

Review comment:
       When switching from System.currentTimeMillis, or Time.now() to Time.monotonicNow(), keep in mind that the latter can return a negative number.   In this case, cachedName.timestamp could be negative due to overflow, while now might still be positive, in which case this code will not take the right path.  The correct comparison would be `((now - cachedName.timestamp) < cacheTimeout)`
   But I would also avoid using monitonicNow for any field called `timestamp`.
   
   

##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestReconfiguration.java
##########
@@ -317,8 +317,8 @@ public void testThread() throws ReconfigurationException {
     }
     dummy.reconfigureProperty(PROP1, VAL2);
 
-    long endWait = Time.now() + 2000;
-    while (dummyThread.isAlive() && Time.now() < endWait) {
+    long endWait = Time.monotonicNow() + 2000;
+    while (dummyThread.isAlive() && Time.monotonicNow() < endWait) {

Review comment:
       Another dangerous comparison.

##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
##########
@@ -419,7 +419,7 @@ public RpcErrorCodeProto getRpcErrorCodeProto() {
         ioe = nrthe;
       }
       // check if timed out
-      if (Time.now()-timeout >= startTime) {
+      if (Time.monotonicNow() - timeout >= startTime) {

Review comment:
       Similarly, I think `Time.monotonicNow() - startTime >= timeout` would be better here.

##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/CacheableIPList.java
##########
@@ -48,7 +48,7 @@ private void updateCacheExpiryTime() {
     if (cacheTimeout < 0) {
       cacheExpiryTimeStamp = -1; // no automatic cache expiry.
     }else {
-      cacheExpiryTimeStamp = System.currentTimeMillis() + cacheTimeout;
+      cacheExpiryTimeStamp = Time.monotonicNow() + cacheTimeout;

Review comment:
       Timestamps should use currentTimeMillis or similar.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org