You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/03/22 12:04:21 UTC
[GitHub] [hadoop] sumangala-patki opened a new pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
sumangala-patki opened a new pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795
The default value of readahead queue depth is currently set to the number of available processors. However, this can result in one inputstream instance consuming more processor time. To ensure equal thread allocation during read for all inputstreams created in a session, we change the default readahead queue depth to a constant (2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore edited a comment on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
surendralilhore edited a comment on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730
Thanks @anoopsjohn for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-840357499
When there are not many parallel FIS reads, this (possible) reduction in read ahead parallelism will affect the read throughput? Specially Spark kind of? From HBase standpoint this is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
surendralilhore commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-857378838
@steveloughran , I updated the Jira.
> Are there any plans to backport to branch-3.3? A retest of the cherrypick is all which should be needed
Yes, @sumangala-patki is working on backport and testing it for 3.2 and 3.3 branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore merged pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
surendralilhore merged pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842000750
@anoopsjohn ==> wrt your comment on "will this hamper spark jobs".
1. This is a speculative read ahead. So whenever this is wrong, it is waste io and iops.
2. With parquet etc, the reads tend to be random. Hence a smaller read ahead depth is prefereable
3. Based on debugging various customer perf isues, we rarely saw benefit of > 2
4. This config is per inputstream and not global across streams.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore edited a comment on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
surendralilhore edited a comment on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730
Thanks @anoopsjohn for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-857390775
@surendralilhore thanks for updating the JIRA
@steveloughran yes, but planning to ensure a gap (maybe a week or two) between check-in to trunk and backport of PRs. Will keep a track of pending backports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore merged pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
surendralilhore merged pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
surendralilhore commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730
Thanks @anoopsjohn for test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842063424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-804012337
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 17s | | https://github.com/apache/hadoop/pull/2795 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/2795 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on a change in pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#discussion_r631605580
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##########
@@ -569,6 +570,20 @@ public void testDiffReadRequestSizeAndRAHBlockSize() throws Exception {
testReadAheads(inputStream, FORTY_EIGHT_KB, SIXTEEN_KB);
}
+ @Test
+ public void testDefaultReadaheadQueueDepth() throws Exception {
+ Configuration config = getRawConfiguration();
+ config.unset("fs.azure.readaheadqueue.depth");
Review comment:
This in effect change this config (Is set already) for this test class. As we direct change in Raw config.
Should we create a cloned conf from rawConfig and on that unset and use that for FS created for this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842063424
Thanks @vinaysbadami . Ya speculative part I agree. Points #2 and #3 are very important. You can think whether can include this in abfs.md even. Will be good.
>This config is per inputstream and not global across streams.
No I dont think so. This config is cluster level and all input streams opened in this will use this value.
Now we have openFile(Path path) API usage in AzureBlobFileSystem. This will allow passing per input stream options. But this config support is not added yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-857108374
1. Now this has been merged, the JIRA MUST be updated with fix version
2. Are there any plans to backport to branch-3.3? A retest of the cherrypick is all which should be needed
Keeping both branches in sync is essential for cherrypicking future work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
surendralilhore commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-853700730
Thanks @anoopsjohn for test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-808031395
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 13m 16s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 15s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 34s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 8s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 87m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 1a2a6de7b3dd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b8fc3459e10b21826844ba99302aa13e5559d71c |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-840495522
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 3s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 21s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 8s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 75m 0s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 10f225a42410 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 56635ba7beb34ed35ecdf4b88a9065dc12c57af0 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/testReport/ |
| Max. process+thread count | 689 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-804643431
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 33s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 51s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 7s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 75m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 75564054a6e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b165323da580b2b1ed88b9c09b9fb42d7891c6f7 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/testReport/ |
| Max. process+thread count | 665 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-804657103
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
```
HNS OAuth
[INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 497, Failures: 0, Errors: 0, Skipped: 70
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 49
HNS-SharedKey
[INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 495, Failures: 0, Errors: 0, Skipped: 24
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40
Non-HNS SharedKey
[INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 250
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842000750
@anoopsjohn ==> wrt your comment on "will this hamper spark jobs".
1. This is a speculative read ahead. So whenever this is wrong, it is waste io and iops.
2. With parquet etc, the reads tend to be random. Hence a smaller read ahead depth is prefereable
3. Based on debugging various customer perf isues, we rarely saw benefit of > 2
4. This config is per inputstream and not global across streams.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#issuecomment-842063951
> This will allow passing per input stream options. But this config support is not added yet.
We can follow up in a jira. Will be nice to have as a feature support from Driver side.
+1 for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2795: HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2795:
URL: https://github.com/apache/hadoop/pull/2795#discussion_r631702871
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##########
@@ -569,6 +570,20 @@ public void testDiffReadRequestSizeAndRAHBlockSize() throws Exception {
testReadAheads(inputStream, FORTY_EIGHT_KB, SIXTEEN_KB);
}
+ @Test
+ public void testDefaultReadaheadQueueDepth() throws Exception {
+ Configuration config = getRawConfiguration();
+ config.unset("fs.azure.readaheadqueue.depth");
Review comment:
For each test, a new raw Configuration instance is created and used to initialize abfsConfig and fileSystem. Hence, changes made to the config within a test will not affect other tests. This modification will only be reflected in the fs instance created in getFileSystem(config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org