You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/04/30 12:53:08 UTC
[GitHub] [hadoop] mehakmeet opened a new pull request #1990: HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS
mehakmeet opened a new pull request #1990:
URL: https://github.com/apache/hadoop/pull/1990
In some cases, ABFS-prefetch thread runs in the background which returns some bytes from the buffer and gives an extra readOp. Thus, making readOps values arbitrary and giving intermittent failures in some cases. Hence, readOps values of 2 or 3 are seen in different setups.
test runs:
- mvn -T 1C -Dparallel-tests=abfs clean verify
- mvn -T 1C -Dparallel-tests=abfs clean verify -Dtest=none -Dit.test=ITestAbfsStreamStatistics
Region: East US, West US
```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 165.964 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (integration-test-abfs-parallel-classes) @ hadoop-azure ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 256.797 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (integration-test-abfs-parallel-classesAndMethods) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (integration-test-abfs-parallel-classes) @ hadoop-azure ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:14 min (Wall Clock)
[INFO] Finished at: 2020-04-30T18:01:26+05:30
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #1990: HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #1990:
URL: https://github.com/apache/hadoop/pull/1990#issuecomment-625203452
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 15s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 56s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 14s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 32s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 1m 10s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 29s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | The patch does not generate ASF License warnings. |
| | | 64m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1990 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f961982a26d6 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 99840aaba66 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/2/testReport/ |
| Max. process+thread count | 343 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #1990: HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #1990:
URL: https://github.com/apache/hadoop/pull/1990#issuecomment-621894812
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 53s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 3s | trunk passed |
| +1 :green_heart: | compile | 0m 45s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 20s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | trunk passed |
| +0 :ok: | spotbugs | 1m 19s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 15s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | the patch passed |
| +1 :green_heart: | compile | 0m 29s | the patch passed |
| +1 :green_heart: | javac | 0m 29s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 22s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 19m 26s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | the patch passed |
| +1 :green_heart: | findbugs | 1m 54s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 51s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | The patch does not generate ASF License warnings. |
| | | 85m 54s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.azure.TestClientThrottlingAnalyzer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1990 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 51d371e18d22 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b5b45c53a4e |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/testReport/ |
| Max. process+thread count | 366 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1990: HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1990:
URL: https://github.com/apache/hadoop/pull/1990#discussion_r420678631
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStreamStatistics.java
##########
@@ -84,12 +84,20 @@ public void testAbfsStreamOps() throws Exception {
LOG.info("Result of Read operation : {}", result);
/*
- Testing if 2 read_ops value is coming after reading full content from a
- file (3 if anything to read from Buffer too).
- Reason: read() call gives read_ops=1,
- reading from AbfsClient(http GET) gives read_ops=2.
+ * Testing if 2 read_ops value is coming after reading full content
+ * from a file (3 if anything to read from Buffer too). Reason: read()
+ * call gives read_ops=1, reading from AbfsClient(http GET) gives
+ * read_ops=2.
+ *
+ * In some cases ABFS-prefetch thread runs in the background which
+ * returns some bytes from buffer and gives an extra readOp.
+ * Thus, making readOps values arbitrary and giving intermittent
+ * failures in some cases. Hence, readOps values of 2 or 3 is seen in
+ * different setups.
+ *
*/
- assertReadWriteOps("read", 2, statistics.getReadOps());
+ assertTrue("Mismatch in read operations",
Review comment:
Error text needs to include actual values
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #1990: HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #1990:
URL: https://github.com/apache/hadoop/pull/1990#issuecomment-625190659
+1, LGTM. IF it fails now, we'd know where to begin debugging it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org