You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/09/08 13:15:53 UTC
[GitHub] [hbase] javierluca opened a new pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
javierluca opened a new pull request #2365:
URL: https://github.com/apache/hbase/pull/2365
https://issues.apache.org/jira/browse/HBASE-24994
Hedged reads use same thread pool for the original read:
https://github.com/apache/hadoop/blob/0b8464d75227fcee2c6e7f2410377b3d53d3d5f8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1349-L1357
When the executor is full and rejects the task, it will be executed in current thread, increasing this metric:
https://github.com/apache/hadoop/blob/0b8464d75227fcee2c6e7f2410377b3d53d3d5f8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L2913-L2918
I want to expose this metric to HBase since it will be useful to estimate the correct thread pool size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2365:
URL: https://github.com/apache/hbase/pull/2365#issuecomment-689035054
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 5m 9s | master passed |
| +1 :green_heart: | compile | 3m 21s | master passed |
| +1 :green_heart: | shadedjars | 8m 2s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 17s | root in master failed. |
| -0 :warning: | javadoc | 0m 20s | hbase-hadoop-compat in master failed. |
| -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 49s | the patch passed |
| +1 :green_heart: | compile | 3m 38s | the patch passed |
| +1 :green_heart: | javac | 3m 38s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 48s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 20s | hbase-hadoop-compat in the patch failed. |
| -0 :warning: | javadoc | 0m 45s | hbase-server in the patch failed. |
| -0 :warning: | javadoc | 0m 15s | root in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 227m 53s | root in the patch failed. |
| | | 267m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2365 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 133819aaed9d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0d95a8f91b |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/testReport/ |
| Max. process+thread count | 3222 (vs. ulimit of 12500) |
| modules | C: hbase-hadoop-compat hbase-server . U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] javierluca commented on pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
javierluca commented on pull request #2365:
URL: https://github.com/apache/hbase/pull/2365#issuecomment-689240094
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] javierluca edited a comment on pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
javierluca edited a comment on pull request #2365:
URL: https://github.com/apache/hbase/pull/2365#issuecomment-689243780
It seems that failed Tests below are not related to this patch. (and not reproduced in my local test)
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/testReport/
| Test name | Duration | Age |
| ------------- | ------------- | ------------- |
| precommit checks / yetus jdk11 hadoop3 checks / org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor5.testScanMetrics[1] | 8.6 sec | 1 |
| precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot. | 0.26 sec | 1 |
| precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot. | 0.25 sec | 1 |
@Apache9 Could you take a look at this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] javierluca commented on pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
javierluca commented on pull request #2365:
URL: https://github.com/apache/hbase/pull/2365#issuecomment-689243780
It seems that failed Tests below are not related to this patch. (and not reproduced in my local test)
| Test name | Duration | Age |
| ------------- | ------------- | ------------- |
| precommit checks / yetus jdk11 hadoop3 checks / org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor5.testScanMetrics[1] | 8.6 sec | 1 |
| precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot. | 0.26 sec | 1 |
| precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot. | 0.25 sec | 1 |
@Apache9 Could you take a look at this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
Apache9 merged pull request #2365:
URL: https://github.com/apache/hbase/pull/2365
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2365:
URL: https://github.com/apache/hbase/pull/2365#issuecomment-689067261
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 19s | master passed |
| +1 :green_heart: | compile | 2m 23s | master passed |
| +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 2m 57s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 23s | the patch passed |
| +1 :green_heart: | compile | 2m 17s | the patch passed |
| +1 :green_heart: | javac | 2m 17s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 31s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 2m 54s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 295m 34s | root in the patch failed. |
| | | 330m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2365 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 73b1b58900ba 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0d95a8f91b |
| Default Java | 1.8.0_232 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/testReport/ |
| Max. process+thread count | 4131 (vs. ulimit of 12500) |
| modules | C: hbase-hadoop-compat hbase-server . U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] javierluca removed a comment on pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
javierluca removed a comment on pull request #2365:
URL: https://github.com/apache/hbase/pull/2365#issuecomment-689240094
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2365: HBASE-24994 Add hedgedReadOpsInCurThread metric
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2365:
URL: https://github.com/apache/hbase/pull/2365#issuecomment-688910211
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 0s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 44s | master passed |
| +1 :green_heart: | checkstyle | 1m 58s | master passed |
| +0 :ok: | refguide | 4m 52s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
| +1 :green_heart: | spotbugs | 11m 13s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 23s | the patch passed |
| -0 :warning: | checkstyle | 1m 53s | root: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +0 :ok: | refguide | 4m 59s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
| +1 :green_heart: | hadoopcheck | 11m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 11m 33s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. |
| | | 65m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2365 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide |
| uname | Linux a2ee2c335c82 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0d95a8f91b |
| refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-general-check/output/branch-site/book.html |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt |
| refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/artifact/yetus-general-check/output/patch-site/book.html |
| Max. process+thread count | 137 (vs. ulimit of 12500) |
| modules | C: hbase-hadoop-compat hbase-server . U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2365/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org