You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/09/21 18:27:07 UTC

[GitHub] [hadoop] steveloughran opened a new pull request #2323: HADOOP-16830. Add public IOStatistics API.

steveloughran opened a new pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323


   
   Hadoop-common side of the patch
   
   * API definition
   * interface
   * tests
   * implementation
   * wiring up of the IO core classes


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran edited a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-700065247


   > For DurationTrackers in IOStatisticsStore() if we add a tracker in a try block, what happens to it in case of failure should be looked at to avoid inaccurate values for the trackers.
   
   I was thinking about failure reporting myself
   
   - we may want to count failures
   - any failure which with a longer or shorter duration than successful operations Will skew the results. Example: network failures -> long durations; auth failures -> short ones.
   
   At the same time, try-with-resources is nice. What to do?
   
   For each set of duration stats, we add counter/mean/min/max of failures
   on a failure, those statistics are updated instead.
   
   Issue: how best to record a failure, given we can't get at the try-with-resources classes in catch or finally? I'd initially thought we could set it in the catch(), but it'd be out of scope.
   
   1. Pessimistic: assume that all attempts are failure, make last operation in every try clause set the success flag. Ugly.
   1. Move construction out of try-with-resources and instead explicit catch and finally. Differently ugly
   
   Fancy lambda-expression wrapper thing? Doable.
   
   ```
   object = DurationTrackerFactory.track("statistic", () ->
     s3.listObjects());
   ```
     
   Then in that code we'd put the code of option #2 in
   
   Fancy curried-function-Haskell-elitism option
   
   Duration tracker takes a function and returns a new one
   
   ```
   FunctionRaisingIOE<A, B> track(String, FunctionRaisingIOE<A, B> inner)
   ```
   
   you'd get a function back which you could then apply at leisure.
   
   ```
   DurationTrackerFactory.track("statistic", () ->
     s3.listObjects()).apply();
   ```
     
   Maybe worth doing both. I could also look at adding into the S3A Invoke code, as every iteration of a retried operation we'd want the statistic updated. 
   
   ```
   invoker.invoke(()-> durationTracker.track("listings",  () -> s3a.list()
   ```
   
   That is where curried functions come out to play, with something like
   
   invoker.invoke(durationTracker.track("listings",  () -> s3a.list()). apply
   
   At the same time: this gets complex fast. Could we make the design of this a followup?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-700777072


   Update: you will get the ability to wrap a Callable, CallableRaisingIOE or functionRaisingIOE with a duration tracking, which then returns a new instance of the same
   
   ```java
   submit(unboundedThreadPool,
             trackDurationOfCallable(
                 listingContext.getDurationTrackerFactory(),
                 OBJECT_LIST_REQUEST,
                 () -> listObjects(request)));
   ```
   
   When there's a failure we update the failure count, move timings into the failures section. Attempts are counted for both, so success == attempts - failures.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on a change in pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#discussion_r496880269



##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsStoreImpl.java
##########
@@ -0,0 +1,424 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import javax.annotation.Nullable;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.statistics.DurationTracker;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.MeanStatistic;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MAX;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MEAN;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MIN;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMaximums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMinimums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.dynamicIOStatistics;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMaximum;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMinimum;
+
+/**
+ * Implementation of {@link IOStatisticsStore}.
+ */
+final class IOStatisticsStoreImpl extends WrappedIOStatistics
+    implements IOStatisticsStore {
+
+  /**
+   * Log changes at debug.
+   * Noisy, but occasionally useful.
+   */
+  private static final Logger LOG =
+      LoggerFactory.getLogger(IOStatisticsStoreImpl.class);
+
+  private final Map<String, AtomicLong> counterMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> gaugeMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> minimumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> maximumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, MeanStatistic> meanStatisticMap
+      = new ConcurrentHashMap<>();
+
+  /**
+   * Constructor invoked via the builder.
+   * @param counters keys to use for the counter statistics.
+   * @param gauges names of gauges
+   * @param minimums names of minimums
+   * @param maximums names of maximums
+   * @param meanStatistics names of mean statistics.
+   */
+  IOStatisticsStoreImpl(
+      final List<String> counters,
+      final List<String> gauges,
+      final List<String> minimums,
+      final List<String> maximums,
+      final List<String> meanStatistics) {
+    // initially create the superclass with no wrapped mapping;
+    super(null);
+
+    // now construct a dynamic statistics source mapping to
+    // the various counters, gauges etc dynamically created
+    // into maps
+    DynamicIOStatisticsBuilder builder = dynamicIOStatistics();
+    if (counters != null) {
+      for (String key : counters) {
+        AtomicLong counter = new AtomicLong();
+        counterMap.put(key, counter);
+        builder.withAtomicLongCounter(key, counter);
+      }
+    }
+    if (gauges != null) {
+      for (String key : gauges) {
+        AtomicLong gauge = new AtomicLong();
+        gaugeMap.put(key, gauge);
+        builder.withAtomicLongGauge(key, gauge);
+      }
+    }
+    if (maximums != null) {
+      for (String key : maximums) {
+        AtomicLong maximum = new AtomicLong(MAX_UNSET_VALUE);
+        maximumMap.put(key, maximum);
+        builder.withAtomicLongMaximum(key, maximum);
+      }
+    }
+    if (minimums != null) {
+      for (String key : minimums) {
+        AtomicLong minimum = new AtomicLong(MIN_UNSET_VALUE);
+        minimumMap.put(key, minimum);
+        builder.withAtomicLongMinimum(key, minimum);
+      }
+    }
+    if (meanStatistics != null) {
+      for (String key : meanStatistics) {
+        meanStatisticMap.put(key, new MeanStatistic());
+        builder.withMeanStatisticFunction(key, k -> meanStatisticMap.get(k));
+      }
+    }
+    setWrapped(builder.build());
+  }
+
+  /**
+   * Set an atomic long to a value.
+   * @param aLong atomic long; may be null
+   * @param value value to set to
+   */
+  private void setAtomicLong(final AtomicLong aLong, final long value) {
+    if (aLong != null) {
+      aLong.set(value);
+    }
+  }
+
+  /**
+   * increment an atomic long and return its value;
+   * null long is no-op returning 0.
+   * @param aLong atomic long; may be null
+   * @param increment amount to increment; -ve for a decrement
+   * @return final value or 0
+   */
+  private long incAtomicLong(final AtomicLong aLong,
+      final long increment) {
+    if (aLong != null) {
+      return aLong.addAndGet(increment);
+    } else {
+      return 0;
+    }
+  }
+
+  @Override
+  public void setCounter(final String key, final long value) {
+    setAtomicLong(counterMap.get(key), value);
+    LOG.debug("Setting counter {} to {}", key, value);
+  }
+
+  @Override
+  public long incrementCounter(final String key, final long value) {

Review comment:
       fixed now




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-704549467






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-717435068


   Also, should use java.io.UncheckedIOException for the wrapper class for an IOE. That's new in Java8, but as its public API there, what we should adopt. This is good, even if its a bit more work


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-701491715


   > error. This is solved when I add ".mean" at the end of the DurationTracker stat name. The same with Max(".max" has to be added) and Min(".min" has to be added) to fetch their values.
   > Was thinking if we could add it to the getter rather than having to append with stat name.
   
   I'm a bit reluctant to, as then people would ask about failures next. What could be handy would be something in the support class to get all stats for a duration (or null), with some struct to contain them all. e.g
   
   `fetchDurationStatistics(IOStatistics, key) -> {count, min, max ,mean, failed, failed.min, failed.max, failed.mean)`, and you'd then work off that. Seem good? 
   Troublespot: what if only some of the values were found? they'd be null in the result. Maybe we'd let you ask for the success/failure stats separately, deal with it that way, and have some `boolean isComplete()` probe to check all are set


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696627308


   I see the problem. This PR only includes the common changes, but as that moved some fs.impl methods to being public in a new package, the uses in the s3 code won't build. 
   ```
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  19:15 min
   [INFO] Finished at: 2020-09-21T20:50:29Z
   [INFO] ------------------------------------------------------------------------
   [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-aws: Compilation failure: Compilation failure: 
   [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32] error: cannot find symbol
   [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25] error: package FunctionsRaisingIOE does not exist
   [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1638,10] error: lambda expression not expected here
   [ERROR] -> [Help 1]
   [ERROR] 
   ```
   Briefly I will reinstate the delete class from fs.impl so everything builds again


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran closed pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696627308


   I see the problem. This PR only includes the common changes, but as that moved some fs.impl methods to being public in a new package, the uses in the s3 code won't build. 
   ```
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  19:15 min
   [INFO] Finished at: 2020-09-21T20:50:29Z
   [INFO] ------------------------------------------------------------------------
   [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-aws: Compilation failure: Compilation failure: 
   [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32] error: cannot find symbol
   [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25] error: package FunctionsRaisingIOE does not exist
   [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1638,10] error: lambda expression not expected here
   [ERROR] -> [Help 1]
   [ERROR] 
   ```
   Briefly I will reinstate the delete class from fs.impl so everything builds again


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-702777116


   Yetus: javac is all about the deprecation changes fixed in #2324 
   
   checkstyles from new tests & code
   ```
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:41:   * @deprecated use {@link org.apache.hadoop.util.functional.FunctionRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:54:   * @deprecated use {@link org.apache.hadoop.util.functional.BiFunctionRaisingIOE}: Line is longer than 80 characters (found 83). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:65:   * @deprecated use {@link org.apache.hadoop.util.functional.CallableRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/BufferedIOStatisticsInputStream.java:51:  /**: First sentence should end with a period. [JavadocStyle]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/package-info.java:28: * {@link org.apache.hadoop.fs.statistics.IOStatisticsSource#getIOStatistics()} .: Line is longer than 80 characters (found 81). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/RemoteIterators.java:412:    /**: First sentence should end with a period. [JavadocStyle]
   ./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/statistics/TestDurationTracking.java:112:        (Integer x) -> x);: 'lambda arguments' has incorrect indentation level 8, expected level should be one of the following: 10, 12. [Indentation]
   ./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/statistics/TestDurationTracking.java:200:  /**: First sentence should end with a period. [JavadocStyle]
   ```
   will fix those


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-702364693


   @mehakmeet 
   * duration tracking for classic function. Issue: are the names of the `trackDuration` calls correct now?
   * class DurationStatisticSummary to store and extract duration stats from a statistic.
   * which is used in the tests to verify that the new functions all work 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-705108526


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 31s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch appears to include 11 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  29m 20s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 42s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  16m 58s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 57s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 29s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 56s |  |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 34s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 16s |  |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 13s |  |  trunk passed  |
   | -0 :warning: |  patch  |   2m 36s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 52s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 59s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javac  |  18m 59s | [/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/9/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt) |  root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 20 new + 2050 unchanged - 0 fixed = 2070 total (was 2050)  |
   | +1 :green_heart: |  compile  |  16m 58s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  javac  |  16m 58s | [/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/9/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt) |  root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 20 new + 1946 unchanged - 0 fixed = 1966 total (was 1946)  |
   | -0 :warning: |  checkstyle  |   0m 55s | [/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/9/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt) |  hadoop-common-project/hadoop-common: The patch generated 4 new + 141 unchanged - 4 fixed = 145 total (was 145)  |
   | +1 :green_heart: |  mvnsite  |   1m 28s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  14m 11s |  |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 36s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 36s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   2m 22s |  |  the patch passed  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 45s |  |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 56s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 161m 40s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/9/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux 32ccda09dff3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 82522d60fb5 |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/9/testReport/ |
   | Max. process+thread count | 1410 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/9/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696387784


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 37s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 10 new or modified test files.  |
   ||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  30m 45s |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m 32s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  19m 19s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 59s |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 45s |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  20m 24s |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 38s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 45s |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 42s |  trunk passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m  2s |  the patch passed  |
   | -1 :x: |  compile  |  22m  3s |  root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.  |
   | -1 :x: |  javac  |  22m  3s |  root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.  |
   | -1 :x: |  compile  |  19m 42s |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -1 :x: |  javac  |  19m 42s |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -0 :warning: |  checkstyle  |   0m 53s |  hadoop-common-project/hadoop-common: The patch generated 17 new + 142 unchanged - 4 fixed = 159 total (was 146)  |
   | +1 :green_heart: |  mvnsite  |   1m 31s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  16m 50s |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javadoc  |   1m 40s |  hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)  |
   | -1 :x: |  findbugs  |   2m 50s |  hadoop-common-project/hadoop-common generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0)  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m  5s |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  The patch does not generate ASF License warnings.  |
   |  |   | 180m 35s |   |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hadoop-common-project/hadoop-common |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.counters; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 183] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.gauges; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 188] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.maximums; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 198] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.meanStatistics; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 203] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.minimums; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 117] |
   |  |  org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized  At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.MeanStatistic.samples; locked 93% of time  Unsynchronized access at MeanStatistic.java:93% of time  Unsynchronized access at MeanStatistic.java:[line 261] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.MeanStatistic.sum; locked 91% of time  Unsynchronized access at MeanStatistic.java:91% of time  Unsynchronized access at MeanStatistic.java:[line 261] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux c43559ab4697 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / c8c1cc43d35 |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
   | javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/testReport/ |
   | Max. process+thread count | 2630 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696387784


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 37s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 10 new or modified test files.  |
   ||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  30m 45s |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m 32s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  19m 19s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 59s |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 45s |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  20m 24s |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 38s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 45s |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 42s |  trunk passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m  2s |  the patch passed  |
   | -1 :x: |  compile  |  22m  3s |  root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.  |
   | -1 :x: |  javac  |  22m  3s |  root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.  |
   | -1 :x: |  compile  |  19m 42s |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -1 :x: |  javac  |  19m 42s |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -0 :warning: |  checkstyle  |   0m 53s |  hadoop-common-project/hadoop-common: The patch generated 17 new + 142 unchanged - 4 fixed = 159 total (was 146)  |
   | +1 :green_heart: |  mvnsite  |   1m 31s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  16m 50s |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javadoc  |   1m 40s |  hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)  |
   | -1 :x: |  findbugs  |   2m 50s |  hadoop-common-project/hadoop-common generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0)  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m  5s |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  The patch does not generate ASF License warnings.  |
   |  |   | 180m 35s |   |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hadoop-common-project/hadoop-common |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.counters; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 183] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.gauges; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 188] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.maximums; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 198] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.meanStatistics; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 203] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.minimums; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 117] |
   |  |  org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized  At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.MeanStatistic.samples; locked 93% of time  Unsynchronized access at MeanStatistic.java:93% of time  Unsynchronized access at MeanStatistic.java:[line 261] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.MeanStatistic.sum; locked 91% of time  Unsynchronized access at MeanStatistic.java:91% of time  Unsynchronized access at MeanStatistic.java:[line 261] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux c43559ab4697 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / c8c1cc43d35 |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
   | javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/testReport/ |
   | Max. process+thread count | 2630 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-698303509


   findbugs
   ```
   module:hadoop-common-project/hadoop-common
   --
     | org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254]
   ```
   
   The synchronized was lifted from TreeMap writeObject. When I turn it off I'll be reasonably confident that findbugs will start complaining about unsynchronized accesses to field. Let's see. If it blows up then I'll just disable the check here
   
   checkstyles are on the uses of the now deprecated methods; will move them on
   
   
   ```
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[34,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[83,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[24,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[68,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:[436,51] [deprecation] FAST_UPLOAD in Constants has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[829,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,6] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[103,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[121,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[33,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[48,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[49,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[52,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[212,22] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,21] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,40] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[324,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[535,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-697395565


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  4s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch appears to include 10 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  29m  3s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 28s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  16m 49s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 57s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 28s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  5s |  |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 36s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 17s |  |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 15s |  |  trunk passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 52s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 46s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javac  |  18m 46s | [/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt) |  root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 20 new + 2055 unchanged - 0 fixed = 2075 total (was 2055)  |
   | +1 :green_heart: |  compile  |  16m 52s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  javac  |  16m 52s | [/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt) |  root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 20 new + 1948 unchanged - 0 fixed = 1968 total (was 1948)  |
   | -0 :warning: |  checkstyle  |   0m 55s | [/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt) |  hadoop-common-project/hadoop-common: The patch generated 4 new + 141 unchanged - 4 fixed = 145 total (was 145)  |
   | +1 :green_heart: |  mvnsite  |   1m 26s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  14m 11s |  |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 35s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  findbugs  |   2m 24s | [/new-findbugs-hadoop-common-project_hadoop-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html) |  hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m  5s |  |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 54s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 161m 46s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hadoop-common-project/hadoop-common |
   |  |  org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized  At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux c96eef23ca40 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 18f9d77c46f |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/testReport/ |
   | Max. process+thread count | 2467 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on a change in pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#discussion_r496866549



##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsStoreImpl.java
##########
@@ -0,0 +1,424 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import javax.annotation.Nullable;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.statistics.DurationTracker;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.MeanStatistic;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MAX;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MEAN;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MIN;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMaximums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMinimums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.dynamicIOStatistics;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMaximum;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMinimum;
+
+/**
+ * Implementation of {@link IOStatisticsStore}.
+ */
+final class IOStatisticsStoreImpl extends WrappedIOStatistics
+    implements IOStatisticsStore {
+
+  /**
+   * Log changes at debug.
+   * Noisy, but occasionally useful.
+   */
+  private static final Logger LOG =
+      LoggerFactory.getLogger(IOStatisticsStoreImpl.class);
+
+  private final Map<String, AtomicLong> counterMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> gaugeMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> minimumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> maximumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, MeanStatistic> meanStatisticMap
+      = new ConcurrentHashMap<>();
+
+  /**
+   * Constructor invoked via the builder.
+   * @param counters keys to use for the counter statistics.
+   * @param gauges names of gauges
+   * @param minimums names of minimums
+   * @param maximums names of maximums
+   * @param meanStatistics names of mean statistics.
+   */
+  IOStatisticsStoreImpl(
+      final List<String> counters,
+      final List<String> gauges,
+      final List<String> minimums,
+      final List<String> maximums,
+      final List<String> meanStatistics) {
+    // initially create the superclass with no wrapped mapping;
+    super(null);
+
+    // now construct a dynamic statistics source mapping to
+    // the various counters, gauges etc dynamically created
+    // into maps
+    DynamicIOStatisticsBuilder builder = dynamicIOStatistics();
+    if (counters != null) {
+      for (String key : counters) {
+        AtomicLong counter = new AtomicLong();
+        counterMap.put(key, counter);
+        builder.withAtomicLongCounter(key, counter);
+      }
+    }
+    if (gauges != null) {
+      for (String key : gauges) {
+        AtomicLong gauge = new AtomicLong();
+        gaugeMap.put(key, gauge);
+        builder.withAtomicLongGauge(key, gauge);
+      }
+    }
+    if (maximums != null) {
+      for (String key : maximums) {
+        AtomicLong maximum = new AtomicLong(MAX_UNSET_VALUE);
+        maximumMap.put(key, maximum);
+        builder.withAtomicLongMaximum(key, maximum);
+      }
+    }
+    if (minimums != null) {
+      for (String key : minimums) {
+        AtomicLong minimum = new AtomicLong(MIN_UNSET_VALUE);
+        minimumMap.put(key, minimum);
+        builder.withAtomicLongMinimum(key, minimum);
+      }
+    }
+    if (meanStatistics != null) {
+      for (String key : meanStatistics) {
+        meanStatisticMap.put(key, new MeanStatistic());
+        builder.withMeanStatisticFunction(key, k -> meanStatisticMap.get(k));
+      }
+    }
+    setWrapped(builder.build());
+  }
+
+  /**
+   * Set an atomic long to a value.
+   * @param aLong atomic long; may be null
+   * @param value value to set to
+   */
+  private void setAtomicLong(final AtomicLong aLong, final long value) {
+    if (aLong != null) {
+      aLong.set(value);
+    }
+  }
+
+  /**
+   * increment an atomic long and return its value;
+   * null long is no-op returning 0.
+   * @param aLong atomic long; may be null
+   * @param increment amount to increment; -ve for a decrement
+   * @return final value or 0
+   */
+  private long incAtomicLong(final AtomicLong aLong,
+      final long increment) {
+    if (aLong != null) {
+      return aLong.addAndGet(increment);
+    } else {
+      return 0;
+    }
+  }
+
+  @Override
+  public void setCounter(final String key, final long value) {
+    setAtomicLong(counterMap.get(key), value);
+    LOG.debug("Setting counter {} to {}", key, value);
+  }
+
+  @Override
+  public long incrementCounter(final String key, final long value) {

Review comment:
       I will make a no-op. And test this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-697395565


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  4s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch appears to include 10 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  29m  3s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 28s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  16m 49s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 57s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 28s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  5s |  |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 36s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 17s |  |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 15s |  |  trunk passed  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 52s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 46s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javac  |  18m 46s | [/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt) |  root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 20 new + 2055 unchanged - 0 fixed = 2075 total (was 2055)  |
   | +1 :green_heart: |  compile  |  16m 52s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  javac  |  16m 52s | [/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt) |  root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 20 new + 1948 unchanged - 0 fixed = 1968 total (was 1948)  |
   | -0 :warning: |  checkstyle  |   0m 55s | [/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt) |  hadoop-common-project/hadoop-common: The patch generated 4 new + 141 unchanged - 4 fixed = 145 total (was 145)  |
   | +1 :green_heart: |  mvnsite  |   1m 26s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  14m 11s |  |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 35s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  findbugs  |   2m 24s | [/new-findbugs-hadoop-common-project_hadoop-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html) |  hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m  5s |  |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 54s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 161m 46s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hadoop-common-project/hadoop-common |
   |  |  org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized  At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux c96eef23ca40 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 18f9d77c46f |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/testReport/ |
   | Max. process+thread count | 2467 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/3/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696855219


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  7s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 10 new or modified test files.  |
   ||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  31m 35s |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m  3s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  17m 53s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 48s |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 23s |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  18m 32s |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 27s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 17s |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 14s |  trunk passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 51s |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 12s |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javac  |  20m 12s |  root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 20 new + 2054 unchanged - 0 fixed = 2074 total (was 2054)  |
   | +1 :green_heart: |  compile  |  17m 44s |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  javac  |  17m 44s |  root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 20 new + 1950 unchanged - 0 fixed = 1970 total (was 1950)  |
   | -0 :warning: |  checkstyle  |   0m 48s |  hadoop-common-project/hadoop-common: The patch generated 20 new + 141 unchanged - 4 fixed = 161 total (was 145)  |
   | +1 :green_heart: |  mvnsite  |   1m 24s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  15m 52s |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 29s |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javadoc  |   1m 27s |  hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)  |
   | -1 :x: |  findbugs  |   2m 24s |  hadoop-common-project/hadoop-common generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m 15s |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 47s |  The patch does not generate ASF License warnings.  |
   |  |   | 171m  3s |   |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hadoop-common-project/hadoop-common |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.counters; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 115] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.gauges; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 116] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.maximums; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 118] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.meanStatistics; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 119] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.minimums; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 117] |
   |  |  org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized  At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux 0de6be7071fb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 6b5d9e2334b |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
   | javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/testReport/ |
   | Max. process+thread count | 3223 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696855219


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  7s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 10 new or modified test files.  |
   ||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  31m 35s |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m  3s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  17m 53s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 48s |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 23s |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  18m 32s |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 27s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 17s |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 14s |  trunk passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 51s |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 12s |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javac  |  20m 12s |  root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 20 new + 2054 unchanged - 0 fixed = 2074 total (was 2054)  |
   | +1 :green_heart: |  compile  |  17m 44s |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  javac  |  17m 44s |  root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 20 new + 1950 unchanged - 0 fixed = 1970 total (was 1950)  |
   | -0 :warning: |  checkstyle  |   0m 48s |  hadoop-common-project/hadoop-common: The patch generated 20 new + 141 unchanged - 4 fixed = 161 total (was 145)  |
   | +1 :green_heart: |  mvnsite  |   1m 24s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  15m 52s |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 29s |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javadoc  |   1m 27s |  hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)  |
   | -1 :x: |  findbugs  |   2m 24s |  hadoop-common-project/hadoop-common generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m 15s |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 47s |  The patch does not generate ASF License warnings.  |
   |  |   | 171m  3s |   |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hadoop-common-project/hadoop-common |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.counters; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 115] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.gauges; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 116] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.maximums; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 118] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.meanStatistics; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 119] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.minimums; locked 80% of time  Unsynchronized access at IOStatisticsSnapshot.java:80% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 117] |
   |  |  org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized  At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux 0de6be7071fb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 6b5d9e2334b |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
   | javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/testReport/ |
   | Max. process+thread count | 3223 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/2/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] mehakmeet commented on a change in pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
mehakmeet commented on a change in pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#discussion_r497473660



##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsStoreImpl.java
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import javax.annotation.Nullable;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.statistics.DurationTracker;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.MeanStatistic;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MAX;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MEAN;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MIN;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMaximums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMinimums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.dynamicIOStatistics;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMaximum;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMinimum;
+
+/**
+ * Implementation of {@link IOStatisticsStore}.
+ */
+final class IOStatisticsStoreImpl extends WrappedIOStatistics
+    implements IOStatisticsStore {
+
+  /**
+   * Log changes at debug.
+   * Noisy, but occasionally useful.
+   */
+  private static final Logger LOG =
+      LoggerFactory.getLogger(IOStatisticsStoreImpl.class);
+
+  private final Map<String, AtomicLong> counterMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> gaugeMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> minimumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> maximumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, MeanStatistic> meanStatisticMap
+      = new ConcurrentHashMap<>();
+
+  /**
+   * Constructor invoked via the builder.
+   * @param counters keys to use for the counter statistics.
+   * @param gauges names of gauges
+   * @param minimums names of minimums
+   * @param maximums names of maximums
+   * @param meanStatistics names of mean statistics.
+   */
+  IOStatisticsStoreImpl(
+      final List<String> counters,
+      final List<String> gauges,
+      final List<String> minimums,
+      final List<String> maximums,
+      final List<String> meanStatistics) {
+    // initially create the superclass with no wrapped mapping;
+    super(null);
+
+    // now construct a dynamic statistics source mapping to
+    // the various counters, gauges etc dynamically created
+    // into maps
+    DynamicIOStatisticsBuilder builder = dynamicIOStatistics();
+    if (counters != null) {
+      for (String key : counters) {
+        AtomicLong counter = new AtomicLong();
+        counterMap.put(key, counter);
+        builder.withAtomicLongCounter(key, counter);
+      }
+    }
+    if (gauges != null) {
+      for (String key : gauges) {
+        AtomicLong gauge = new AtomicLong();
+        gaugeMap.put(key, gauge);
+        builder.withAtomicLongGauge(key, gauge);
+      }
+    }
+    if (maximums != null) {
+      for (String key : maximums) {
+        AtomicLong maximum = new AtomicLong(MAX_UNSET_VALUE);
+        maximumMap.put(key, maximum);
+        builder.withAtomicLongMaximum(key, maximum);
+      }
+    }
+    if (minimums != null) {
+      for (String key : minimums) {
+        AtomicLong minimum = new AtomicLong(MIN_UNSET_VALUE);
+        minimumMap.put(key, minimum);
+        builder.withAtomicLongMinimum(key, minimum);
+      }
+    }
+    if (meanStatistics != null) {
+      for (String key : meanStatistics) {
+        meanStatisticMap.put(key, new MeanStatistic());
+        builder.withMeanStatisticFunction(key, k -> meanStatisticMap.get(k));
+      }
+    }
+    setWrapped(builder.build());
+  }
+
+  /**
+   * Set an atomic long to a value.
+   * @param aLong atomic long; may be null
+   * @param value value to set to
+   */
+  private void setAtomicLong(final AtomicLong aLong, final long value) {
+    if (aLong != null) {
+      aLong.set(value);
+    }
+  }
+
+  /**
+   * increment an atomic long and return its value;
+   * null long is no-op returning 0.
+   * @param aLong atomic long; may be null
+   * @param increment amount to increment; -ve for a decrement
+   * @return final value or 0
+   */
+  private long incAtomicLong(final AtomicLong aLong,
+      final long increment) {
+    if (aLong != null) {
+      return aLong.addAndGet(increment);
+    } else {
+      return 0;
+    }
+  }
+
+  @Override
+  public void setCounter(final String key, final long value) {
+    setAtomicLong(counterMap.get(key), value);
+    LOG.debug("Setting counter {} to {}", key, value);
+  }
+
+  @Override
+  public long incrementCounter(final String key, final long value) {
+    AtomicLong counter = counterMap.get(key);
+    if (counter == null) {
+      LOG.debug("Ignoring counter increment for unknown counter {}",
+          key);
+      return 0;
+    }
+    if (value < 0) {
+      LOG.debug("Ignoring negative incrment value {} for counter {}",

Review comment:
       typo in debug statement (incrment).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-704549467


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 31s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch appears to include 11 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  28m 46s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  19m 34s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  16m 50s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 57s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 29s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  1s |  |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 36s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 18s |  |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 16s |  |  trunk passed  |
   | -0 :warning: |  patch  |   2m 38s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 50s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  18m 53s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javac  |  18m 53s | [/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/8/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt) |  root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 20 new + 2050 unchanged - 0 fixed = 2070 total (was 2050)  |
   | +1 :green_heart: |  compile  |  16m 47s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | -1 :x: |  javac  |  16m 47s | [/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/8/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt) |  root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 20 new + 1946 unchanged - 0 fixed = 1966 total (was 1946)  |
   | -0 :warning: |  checkstyle  |   0m 57s | [/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/8/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt) |  hadoop-common-project/hadoop-common: The patch generated 5 new + 141 unchanged - 4 fixed = 146 total (was 145)  |
   | +1 :green_heart: |  mvnsite  |   1m 26s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  14m  6s |  |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 32s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   2m 21s |  |  the patch passed  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   9m 46s |  |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 160m 36s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/8/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux 28081b443bf7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / b361f29dda0 |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/8/testReport/ |
   | Max. process+thread count | 2599 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/8/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-707756428


   build failure was in native IO. full PR in #2324 didn't show this. I'd rebase this PR except it would complicate everything else too much


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-707314832


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |  29m 38s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  2s |  |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch appears to include 11 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  29m 53s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 24s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  20m 48s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 58s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 38s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  19m 36s |  |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 35s |  |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 43s |  |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 50s |  |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 47s |  |  trunk passed  |
   | -0 :warning: |  patch  |   3m 11s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 57s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  24m 10s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javac  |  24m 10s | [/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt) |  root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 20 new + 2050 unchanged - 0 fixed = 2070 total (was 2050)  |
   | -1 :x: |  compile  |   5m 14s | [/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt) |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -1 :x: |  javac  |   5m 14s | [/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt) |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -0 :warning: |  checkstyle  |   0m 36s | [/buildtool-patch-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/artifact/out/buildtool-patch-checkstyle-hadoop-common-project_hadoop-common.txt) |  The patch fails to run checkstyle in hadoop-common  |
   | -1 :x: |  mvnsite  |   0m 37s | [/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt) |  hadoop-common in the patch failed.  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |   0m 43s |  |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 33s |  |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 45s |  |  the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   2m 57s |  |  the patch passed  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  10m 52s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 54s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 182m 10s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.ipc.TestRPCCallBenchmark |
   |   | hadoop.metrics2.source.TestJvmMetrics |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux 57087f185719 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / b92f72758bf |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/testReport/ |
   | Max. process+thread count | 1421 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/10/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696292801


   hadoop common part of #2069 : API and impl,
   
   While this is isolated for review, know that I'll be doing my changes on #2324 and cherry-picking here. It's just isolated for ease of review/to scare people less.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran edited a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-745459150


   Closing as it's impossible to maintain the interface and impl branches. The combined -common and -aws changes make for a bigger pr, but each patch has gone in separately, It will be possible to merge the changes into Hadoop as two separate patches 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran edited a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-698303509


   findbugs
   ```
   module:hadoop-common-project/hadoop-common
   --
     | org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254]
   ```
   
   The synchronized was lifted from TreeMap writeObject. When I turn it off I'll be reasonably confident that findbugs will start complaining about unsynchronized accesses to field. Let's see. If it blows up then I'll just disable the check here
   
   checkstyles are on the uses of the now deprecated methods;
   
   
   ```
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[34,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[83,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[24,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[68,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:[436,51] [deprecation] FAST_UPLOAD in Constants has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[829,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,6] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[103,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[121,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[33,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[48,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[49,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[52,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[212,22] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,21] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,40] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[324,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[535,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   ```
   
   In the full s3a patch these are all migrated; in this PR I'm restricting all changes to hadoop-common, so *will not fixe them in this patch*


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] mehakmeet commented on a change in pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
mehakmeet commented on a change in pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#discussion_r495853308



##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsStoreImpl.java
##########
@@ -0,0 +1,424 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import javax.annotation.Nullable;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.statistics.DurationTracker;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.MeanStatistic;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MAX;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MEAN;
+import static org.apache.hadoop.fs.statistics.StoreStatisticNames.SUFFIX_MIN;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMaximums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.aggregateMinimums;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.dynamicIOStatistics;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMaximum;
+import static org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.maybeUpdateMinimum;
+
+/**
+ * Implementation of {@link IOStatisticsStore}.
+ */
+final class IOStatisticsStoreImpl extends WrappedIOStatistics
+    implements IOStatisticsStore {
+
+  /**
+   * Log changes at debug.
+   * Noisy, but occasionally useful.
+   */
+  private static final Logger LOG =
+      LoggerFactory.getLogger(IOStatisticsStoreImpl.class);
+
+  private final Map<String, AtomicLong> counterMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> gaugeMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> minimumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, AtomicLong> maximumMap = new ConcurrentHashMap<>();
+
+  private final Map<String, MeanStatistic> meanStatisticMap
+      = new ConcurrentHashMap<>();
+
+  /**
+   * Constructor invoked via the builder.
+   * @param counters keys to use for the counter statistics.
+   * @param gauges names of gauges
+   * @param minimums names of minimums
+   * @param maximums names of maximums
+   * @param meanStatistics names of mean statistics.
+   */
+  IOStatisticsStoreImpl(
+      final List<String> counters,
+      final List<String> gauges,
+      final List<String> minimums,
+      final List<String> maximums,
+      final List<String> meanStatistics) {
+    // initially create the superclass with no wrapped mapping;
+    super(null);
+
+    // now construct a dynamic statistics source mapping to
+    // the various counters, gauges etc dynamically created
+    // into maps
+    DynamicIOStatisticsBuilder builder = dynamicIOStatistics();
+    if (counters != null) {
+      for (String key : counters) {
+        AtomicLong counter = new AtomicLong();
+        counterMap.put(key, counter);
+        builder.withAtomicLongCounter(key, counter);
+      }
+    }
+    if (gauges != null) {
+      for (String key : gauges) {
+        AtomicLong gauge = new AtomicLong();
+        gaugeMap.put(key, gauge);
+        builder.withAtomicLongGauge(key, gauge);
+      }
+    }
+    if (maximums != null) {
+      for (String key : maximums) {
+        AtomicLong maximum = new AtomicLong(MAX_UNSET_VALUE);
+        maximumMap.put(key, maximum);
+        builder.withAtomicLongMaximum(key, maximum);
+      }
+    }
+    if (minimums != null) {
+      for (String key : minimums) {
+        AtomicLong minimum = new AtomicLong(MIN_UNSET_VALUE);
+        minimumMap.put(key, minimum);
+        builder.withAtomicLongMinimum(key, minimum);
+      }
+    }
+    if (meanStatistics != null) {
+      for (String key : meanStatistics) {
+        meanStatisticMap.put(key, new MeanStatistic());
+        builder.withMeanStatisticFunction(key, k -> meanStatisticMap.get(k));
+      }
+    }
+    setWrapped(builder.build());
+  }
+
+  /**
+   * Set an atomic long to a value.
+   * @param aLong atomic long; may be null
+   * @param value value to set to
+   */
+  private void setAtomicLong(final AtomicLong aLong, final long value) {
+    if (aLong != null) {
+      aLong.set(value);
+    }
+  }
+
+  /**
+   * increment an atomic long and return its value;
+   * null long is no-op returning 0.
+   * @param aLong atomic long; may be null
+   * @param increment amount to increment; -ve for a decrement
+   * @return final value or 0
+   */
+  private long incAtomicLong(final AtomicLong aLong,
+      final long increment) {
+    if (aLong != null) {
+      return aLong.addAndGet(increment);
+    } else {
+      return 0;
+    }
+  }
+
+  @Override
+  public void setCounter(final String key, final long value) {
+    setAtomicLong(counterMap.get(key), value);
+    LOG.debug("Setting counter {} to {}", key, value);
+  }
+
+  @Override
+  public long incrementCounter(final String key, final long value) {

Review comment:
       Should check if value>0 before incrementing to avoid negative values for counters.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran edited a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-698303509


   findbugs
   ```
   module:hadoop-common-project/hadoop-common
   --
     | org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254]
   ```
   
   The synchronized was lifted from TreeMap writeObject. When I turn it off I'll be reasonably confident that findbugs will start complaining about unsynchronized accesses to field. Let's see. If it blows up then I'll just disable the check here
   
   checkstyles are on the uses of the now deprecated methods;
   
   
   ```
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[34,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[83,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[24,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[68,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:[436,51] [deprecation] FAST_UPLOAD in Constants has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[829,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,6] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[103,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[121,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[33,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[48,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[49,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[52,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[212,22] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,21] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,40] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[324,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[535,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   ```
   
   In the full s3a patch these are all migrated; in this PR I'm restricting all changes to hadoop-common, so *will not fixe them in this patch*


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696387784


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 37s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files found.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 10 new or modified test files.  |
   ||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  30m 45s |  trunk passed  |
   | +1 :green_heart: |  compile  |  21m 32s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  compile  |  19m 19s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 59s |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 45s |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  20m 24s |  branch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 38s |  trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01  |
   | +0 :ok: |  spotbugs  |   2m 45s |  Used deprecated FindBugs config; considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   2m 42s |  trunk passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m  2s |  the patch passed  |
   | -1 :x: |  compile  |  22m  3s |  root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.  |
   | -1 :x: |  javac  |  22m  3s |  root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.  |
   | -1 :x: |  compile  |  19m 42s |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -1 :x: |  javac  |  19m 42s |  root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.  |
   | -0 :warning: |  checkstyle  |   0m 53s |  hadoop-common-project/hadoop-common: The patch generated 17 new + 142 unchanged - 4 fixed = 159 total (was 146)  |
   | +1 :green_heart: |  mvnsite  |   1m 31s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  16m 50s |  patch has no errors when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1  |
   | -1 :x: |  javadoc  |   1m 40s |  hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)  |
   | -1 :x: |  findbugs  |   2m 50s |  hadoop-common-project/hadoop-common generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0)  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  10m  5s |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 55s |  The patch does not generate ASF License warnings.  |
   |  |   | 180m 35s |   |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hadoop-common-project/hadoop-common |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.counters; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 183] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.gauges; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 188] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.maximums; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 198] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.meanStatistics; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 203] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.minimums; locked 60% of time  Unsynchronized access at IOStatisticsSnapshot.java:60% of time  Unsynchronized access at IOStatisticsSnapshot.java:[line 117] |
   |  |  org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized  At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.MeanStatistic.samples; locked 93% of time  Unsynchronized access at MeanStatistic.java:93% of time  Unsynchronized access at MeanStatistic.java:[line 261] |
   |  |  Inconsistent synchronization of org.apache.hadoop.fs.statistics.MeanStatistic.sum; locked 91% of time  Unsynchronized access at MeanStatistic.java:91% of time  Unsynchronized access at MeanStatistic.java:[line 261] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2323 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
   | uname | Linux c43559ab4697 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / c8c1cc43d35 |
   | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
   | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
   | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
   | javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
   | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/testReport/ |
   | Max. process+thread count | 2630 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2323/1/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-745459150


   Closing as nobody was reviewing this and its impossible to maintain the interface and impl branches. The combined -common and -aws changes make for a bigger pr, but each patch has gone in separately, It will be possible to merge the changes into Hadoop as two separate patches 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-704549467






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-700065247


   > For DurationTrackers in IOStatisticsStore() if we add a tracker in a try block, what happens to it in case of failure should be looked at to avoid inaccurate values for the trackers.
   
   I was thinking about failure reporting myself
   
   - we may want to count failures
   - any failure which with a longer or shorter duration than successful operations Will skew the results. Example: network failures -> long durations; auth failures -> short ones.
   
   At the same time, try-with-resources is nice. What to do?
   
   For each set of duration stats, we add counter/mean/min/max of failures
   on a failure, those statistics are updated instead.
   
   Issue: how best to record a failure, given we can't get at the try-with-resources classes in catch or finally? I'd initially thought we could set it in the catch(), but it'd be out of scope.
   
   1. Pessimistic: assume that all attempts are failure, make last operation in every try clause set the success flag. Ugly.
   1. Move construction out of try-with-resources and instead explicit catch and finally. Differently ugly
   
   Fancy lambda-expression wrapper thing? Doable.
   
   ```
   object = DurationTrackerFactory.track("statistic", () ->
     s3.listObjects());
   ```
     
   Then in that code we'd put the code of option #2 in
   
   Fancy curried-function-Haskell-elitism option
   
   Duration tracker takes a function and returns a new one
   
   ```
   FunctionRaisingIOE<A, B> track(String, FunctionRaisingIOE<A, B> inner)
   ```
   
   you'd get a function back which you could then apply at leisure.
   
   ```
   DurationTrackerFactory.track("statistic", () ->
     s3.listObjects()).apply();
   ```
     
   Maybe worth doing both. I could also look at adding into the S3A Invoke code, as every iteration of a retried operation we'd want the statistic updated. 
   
   ```
   
   At the same time: this gets complex fast. Could we make the design of this a followup?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-705609349


   javac failures on newly deprecated static fs.impl.FutureIOSupport methods I'm retaining to avoiding breaking things; checkstyle on 4 javadoc lines which are too wide because they reference other classes
   
   ```
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:41:   * @deprecated use {@link org.apache.hadoop.util.functional.FunctionRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:54:   * @deprecated use {@link org.apache.hadoop.util.functional.BiFunctionRaisingIOE}: Line is longer than 80 characters (found 83). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:65:   * @deprecated use {@link org.apache.hadoop.util.functional.CallableRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/package-info.java:28: * {@link org.apache.hadoop.fs.statistics.IOStatisticsSource#getIOStatistics()} .: Line is longer than 80 characters (found 81). [LineLength]
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696387784






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-718839259


   This PR won't merge right now due to the move to shaded Preconditons. The PR #2324 has been rebased; I'll be rebuilding this PR from that one as a trunk + one big fat patch


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] mehakmeet commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
mehakmeet commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-702072245


   In IOStatisticsBinding class we have methods for tracking duration but, I am not able to wrap it around a normal function.
   There are 3 methods for tracking durations which are for Callable<B>, CallableRaisingIOE<B>, and FunctionRaisingIOE<A, B>. We should add 1 more for a normal function too.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696387784






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-698303509


   findbugs
   ```
   module:hadoop-common-project/hadoop-common
   --
     | org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method is synchronized At IOStatisticsSnapshot.java: At IOStatisticsSnapshot.java:[lines 241-254]
   ```
   
   The synchronized was lifted from TreeMap writeObject. When I turn it off I'll be reasonably confident that findbugs will start complaining about unsynchronized accesses to field. Let's see. If it blows up then I'll just disable the check here
   
   checkstyles are on the uses of the now deprecated methods; will move them on
   
   
   ```
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[34,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[83,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[24,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[68,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:[436,51] [deprecation] FAST_UPLOAD in Constants has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[829,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,6] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[103,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[121,18] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[33,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[48,32] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[49,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[52,32] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[212,22] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,21] [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[320,40] [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[324,16] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   [WARNING] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[535,13] [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been deprecated
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-705609349


   javac failures on newly deprecated static fs.impl.FutureIOSupport methods I'm retaining to avoiding breaking things; checkstyle on 4 javadoc lines which are too wide because they reference other classes
   
   ```
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:41:   * @deprecated use {@link org.apache.hadoop.util.functional.FunctionRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:54:   * @deprecated use {@link org.apache.hadoop.util.functional.BiFunctionRaisingIOE}: Line is longer than 80 characters (found 83). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:65:   * @deprecated use {@link org.apache.hadoop.util.functional.CallableRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
   ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/package-info.java:28: * {@link org.apache.hadoop.fs.statistics.IOStatisticsSource#getIOStatistics()} .: Line is longer than 80 characters (found 81). [LineLength]
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696292801


   hadoop common part of #2069 : API and impl,
   
   While this is isolated for review, know that I'll be doing my changes on #2324 and cherry-picking here. It's just isolated for ease of review/to scare people less.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org