You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/01/19 08:01:26 UTC
[GitHub] [hbase] Yiran-wu opened a new pull request #1069: HBASE-23708
Support metrics for region traffic
Yiran-wu opened a new pull request #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069
I think we need some metrics to measure the amount of traffic being read and written to each region, which can help us more clearly analyze how much network resources are consumed by each region.
putBytes/getBytes/scanBytes/deleteBytes/incrementBytes/appendBytes/bulkLoadBytes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Yiran-wu commented on a change in pull request #1069:
HBASE-23708 Support metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Yiran-wu commented on a change in pull request #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#discussion_r368800553
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -3699,9 +3682,11 @@ public void prepareMiniBatchOperations(MiniBatchOperationInProgress<Mutation> mi
if (mutation instanceof Put) {
HRegion.updateCellTimestamps(familyCellMaps[index].values(), byteTS);
miniBatchOp.incrementNumOfPuts();
+ miniBatchOp.incrementBytesForPuts(mutation.heapSize());
Review comment:
@saintstack Thanks for review
At present, we only have aggregate information and cannot know how much network resources are used in each region.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1069: HBASE-23708 Support
metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#issuecomment-575997610
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 9s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 5m 46s | master passed |
| +1 :green_heart: | compile | 1m 32s | master passed |
| +1 :green_heart: | checkstyle | 1m 41s | master passed |
| +1 :green_heart: | shadedjars | 5m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | master passed |
| +0 :ok: | spotbugs | 5m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 6m 2s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 29s | the patch passed |
| +1 :green_heart: | compile | 1m 37s | the patch passed |
| +1 :green_heart: | javac | 1m 37s | the patch passed |
| -1 :x: | checkstyle | 1m 20s | hbase-server: The patch generated 1 new + 269 unchanged - 2 fixed = 270 total (was 271) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 5m 53s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 19m 6s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
| -1 :x: | findbugs | 5m 10s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 0m 39s | hbase-hadoop2-compat in the patch passed. |
| +1 :green_heart: | unit | 156m 36s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | The patch does not generate ASF License warnings. |
| | | 230m 48s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hbase-server |
| | Nullcheck of storeFileInfo at line 5669 of value previously dereferenced in org.apache.hadoop.hbase.regionserver.HRegion.replayWALBulkLoadEventMarker(WALProtos$BulkLoadDescriptor) At HRegion.java:5669 of value previously dereferenced in org.apache.hadoop.hbase.regionserver.HRegion.replayWALBulkLoadEventMarker(WALProtos$BulkLoadDescriptor) At HRegion.java:[line 5669] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1069 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 187d8b9a8373 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1069/out/precommit/personality/provided.sh |
| git revision | master / 75b85016eb |
| Default Java | 1.8.0_181 |
| checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/artifact/out/diff-checkstyle-hbase-server.txt |
| findbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/artifact/out/new-findbugs-hbase-server.html |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/testReport/ |
| Max. process+thread count | 5469 (vs. ulimit of 10000) |
| modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1069: HBASE-23708 Support
metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#issuecomment-615878182
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/1069 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hbase/pull/1069 |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1069:
HBASE-23708 Support metrics for region traffic
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#discussion_r368655470
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -3699,9 +3682,11 @@ public void prepareMiniBatchOperations(MiniBatchOperationInProgress<Mutation> mi
if (mutation instanceof Put) {
HRegion.updateCellTimestamps(familyCellMaps[index].values(), byteTS);
miniBatchOp.incrementNumOfPuts();
+ miniBatchOp.incrementBytesForPuts(mutation.heapSize());
Review comment:
We spend a lot of cpu counting currently. This adds to the counting load.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1069:
HBASE-23708 Support metrics for region traffic
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#discussion_r368655764
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java
##########
@@ -44,31 +44,48 @@ public void close() {
source.close();
}
- public void updatePut() {
+ public void updatePut(final long size) {
source.updatePut();
+ source.updatePutBytes(size);
+ source.updateReceivedBytes(size);
Review comment:
What is the difference between put and received? They are not the same?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1069: HBASE-23708 Support
metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#issuecomment-576120142
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 41s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 5m 52s | master passed |
| +1 :green_heart: | compile | 1m 29s | master passed |
| +1 :green_heart: | checkstyle | 1m 41s | master passed |
| +1 :green_heart: | shadedjars | 5m 3s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | master passed |
| +0 :ok: | spotbugs | 4m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 51s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 28s | the patch passed |
| +1 :green_heart: | compile | 1m 33s | the patch passed |
| +1 :green_heart: | javac | 1m 33s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-hadoop-compat |
| +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-hadoop2-compat |
| +1 :green_heart: | checkstyle | 1m 17s | hbase-server: The patch generated 0 new + 269 unchanged - 2 fixed = 269 total (was 271) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 5m 3s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 17m 13s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
| -1 :x: | findbugs | 5m 11s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 32s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 0m 39s | hbase-hadoop2-compat in the patch passed. |
| +1 :green_heart: | unit | 159m 27s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 1m 13s | The patch does not generate ASF License warnings. |
| | | 232m 46s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hbase-server |
| | Nullcheck of storeFileInfo at line 5669 of value previously dereferenced in org.apache.hadoop.hbase.regionserver.HRegion.replayWALBulkLoadEventMarker(WALProtos$BulkLoadDescriptor) At HRegion.java:5669 of value previously dereferenced in org.apache.hadoop.hbase.regionserver.HRegion.replayWALBulkLoadEventMarker(WALProtos$BulkLoadDescriptor) At HRegion.java:[line 5669] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1069 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ef5413e83807 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1069/out/precommit/personality/provided.sh |
| git revision | master / 50e264485d |
| Default Java | 1.8.0_181 |
| findbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/2/artifact/out/new-findbugs-hbase-server.html |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/2/testReport/ |
| Max. process+thread count | 5354 (vs. ulimit of 10000) |
| modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/2/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1069:
HBASE-23708 Support metrics for region traffic
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#discussion_r368655213
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -3699,9 +3682,11 @@ public void prepareMiniBatchOperations(MiniBatchOperationInProgress<Mutation> mi
if (mutation instanceof Put) {
HRegion.updateCellTimestamps(familyCellMaps[index].values(), byteTS);
miniBatchOp.incrementNumOfPuts();
+ miniBatchOp.incrementBytesForPuts(mutation.heapSize());
Review comment:
Do we traverse any synchronize blocks here or update any volatiles? In other words, wondering what the cost of these updates are.
We are also bulking up metrics w/ these added numbers... the metrics we send the master on each report (every second or so).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1069: HBASE-23708 Support
metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#issuecomment-615878206
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/1069 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hbase/pull/1069 |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1069: HBASE-23708 Support
metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#issuecomment-615878207
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/1069 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hbase/pull/1069 |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1069/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Yiran-wu commented on issue #1069: HBASE-23708 Support
metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Yiran-wu commented on issue #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#issuecomment-576146784
This error can be ignored. I didn't change it. What about the others?
> Nullcheck of storeFileInfo at line 5669
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Yiran-wu commented on a change in pull request #1069:
HBASE-23708 Support metrics for region traffic
Posted by GitBox <gi...@apache.org>.
Yiran-wu commented on a change in pull request #1069: HBASE-23708 Support metrics for region traffic
URL: https://github.com/apache/hbase/pull/1069#discussion_r368799499
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java
##########
@@ -44,31 +44,48 @@ public void close() {
source.close();
}
- public void updatePut() {
+ public void updatePut(final long size) {
source.updatePut();
+ source.updatePutBytes(size);
+ source.updateReceivedBytes(size);
Review comment:
Received contains all write operations such as increment/append/put/bulkload, putBytes consists of its own
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services