You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/03/18 05:50:03 UTC
[GitHub] [hbase] saintstack opened a new pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
saintstack opened a new pull request #3061:
URL: https://github.com/apache/hbase/pull/3061
…comes a bottleneck when heavy load
Don't have every handler update regionserver metrics on each
scan#nextRaw; instead, do a batch update just before Scan
returns. Otherwise, all running handlers end up contending
on metrics update.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Update of regionserver metrics counters moved out to caller where
can be done as a batch update instead of per-next.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
Class doc to encourage batch updating metrics.
Remove the single update as unused anymore.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
Count calls to nextRaw. Update regionserver count in finally block when
scan is done rather than per nextRaw call. Move all metrics updates to
finally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
ZhaoBQ commented on pull request #3061:
URL: https://github.com/apache/hbase/pull/3061#issuecomment-801701327
LGTM.
Will the readRequestsCount update also affect performance?
`if (!outResults.isEmpty()) {
readRequestsCount.increment();
if (metricsRegion != null) {
metricsRegion.updateReadRequestCount();
}
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3061:
URL: https://github.com/apache/hbase/pull/3061#issuecomment-801733745
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 23s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 16s | branch-2.3 passed |
| +1 :green_heart: | compile | 1m 3s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 5m 53s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 55s | the patch passed |
| +1 :green_heart: | compile | 1m 4s | the patch passed |
| +1 :green_heart: | javac | 1m 4s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 51s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 135m 9s | hbase-server in the patch passed. |
| | | 162m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3061 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 28b07c92150e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / bd250f57f7 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/testReport/ |
| Max. process+thread count | 4126 (vs. ulimit of 12500) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3061:
URL: https://github.com/apache/hbase/pull/3061#issuecomment-801670763
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 41s | branch-2.3 passed |
| +1 :green_heart: | compile | 3m 6s | branch-2.3 passed |
| +1 :green_heart: | checkstyle | 1m 12s | branch-2.3 passed |
| +1 :green_heart: | spotbugs | 1m 58s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 13s | the patch passed |
| +1 :green_heart: | compile | 2m 58s | the patch passed |
| +1 :green_heart: | javac | 2m 58s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 11s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 6s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 2m 11s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| | | 44m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3061 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ab70d29b4eb8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / bd250f57f7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 96 (vs. ulimit of 12500) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #3061:
URL: https://github.com/apache/hbase/pull/3061#issuecomment-802189258
@ZhaoBQ I looked at readRequestsCount over in increment/append and it seemed ok to me. The increment is done on the end after all increments/appends for a row. Increments/appends also take more time -- read, then write -- so I figured the update rate here should be fine.
Thanks for taking a look (Thanks for +1 @Reidddddd )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
huaxiangsun commented on pull request #3061:
URL: https://github.com/apache/hbase/pull/3061#issuecomment-802266244
I left a comment that this can be turned on/off as a feature. This is in the data path, people may not need this on all the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
ZhaoBQ commented on pull request #3061:
URL: https://github.com/apache/hbase/pull/3061#issuecomment-802536065
> @ZhaoBQ I looked at readRequestsCount over in increment/append and it seemed ok to me. The increment is done on the end after all increments/appends for a row. Increments/appends also take more time -- read, then write -- so I figured the update rate here should be fine.
In fact, I’m talking about the readRequestsCount in RegionScannerImpl.nextRaw :
https://github.com/apache/hbase/blob/585aca1f0579179ca234c0324420d27630054085/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionScannerImpl.java#L273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
saintstack merged pull request #3061:
URL: https://github.com/apache/hbase/pull/3061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3061: HBASE-25677 Server+table counters on each scan #nextRaw invocation be…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3061:
URL: https://github.com/apache/hbase/pull/3061#issuecomment-801731457
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 37s | branch-2.3 passed |
| +1 :green_heart: | compile | 0m 53s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 5m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 12s | the patch passed |
| +1 :green_heart: | compile | 0m 55s | the patch passed |
| +1 :green_heart: | javac | 0m 55s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 135m 59s | hbase-server in the patch passed. |
| | | 158m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3061 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 8647b87ac581 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / bd250f57f7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/testReport/ |
| Max. process+thread count | 4100 (vs. ulimit of 12500) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3061/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org