You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/05/12 09:46:13 UTC
[GitHub] [hbase] sunhelly opened a new pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
sunhelly opened a new pull request #3260:
URL: https://github.com/apache/hbase/pull/3260
…eps when balancing by table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-842941351
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 14s | master passed |
| +1 :green_heart: | compile | 1m 2s | master passed |
| +1 :green_heart: | shadedjars | 8m 58s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 12s | the patch passed |
| +1 :green_heart: | compile | 1m 5s | the patch passed |
| +1 :green_heart: | javac | 1m 5s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 55s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 212m 33s | hbase-server in the patch passed. |
| | | 245m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 58d1a6b774df 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 741b4b4674 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/testReport/ |
| Max. process+thread count | 3303 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly closed pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
sunhelly closed pull request #3260:
URL: https://github.com/apache/hbase/pull/3260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-839805036
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 47s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 51s | master passed |
| +1 :green_heart: | compile | 1m 35s | master passed |
| +1 :green_heart: | shadedjars | 10m 31s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 53s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 44s | the patch passed |
| +1 :green_heart: | compile | 1m 46s | the patch passed |
| +1 :green_heart: | javac | 1m 46s | the patch passed |
| +1 :green_heart: | shadedjars | 11m 0s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 221m 58s | hbase-server in the patch passed. |
| | | 264m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 4956a7047efd 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 630c73fda4 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/testReport/ |
| Max. process+thread count | 3197 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-900291831
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 57s | master passed |
| +1 :green_heart: | compile | 0m 36s | master passed |
| +1 :green_heart: | checkstyle | 0m 16s | master passed |
| +1 :green_heart: | spotbugs | 0m 36s | master passed |
| -0 :warning: | patch | 0m 45s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 39s | the patch passed |
| +1 :green_heart: | compile | 0m 32s | the patch passed |
| +1 :green_heart: | javac | 0m 32s | the patch passed |
| -0 :warning: | checkstyle | 0m 14s | hbase-balancer: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 18m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 0m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 38m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4ec2437b6b9d 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 721cb96f8c |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt |
| Max. process+thread count | 95 (vs. ulimit of 30000) |
| modules | C: hbase-balancer U: hbase-balancer |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
sunhelly commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840323482
> I think even with only 1 table, we could still compute the table skew count?
>
> For example, we have a table with 100 regions, and there are 3 region servers, but all the regions are placed on a single server?
Hi, @Apache9 , when there is only 1 table, just like balance by table (the cluster state only contains distribution info of exactly one table and all the online RSes), the RegionCountSkewCostFunction can replace the TableSkewCostFunction. For the example of [100,0,0], the cost of RegionCountSkewCostFunction is 1, which will generate balance actions in the next steps.
But when the table regions has already been evenly distributed, for example [1,0,0,0], the cost of TableSkewCostFunction will not be 0, it is 1(the cost of RegionCountSkewCostFunction is 0), but the table does not need to be balanced, it brings many redundant computation steps of actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840338342
> > I think even with only 1 table, we could still compute the table skew count?
> > For example, we have a table with 100 regions, and there are 3 region servers, but all the regions are placed on a single server?
>
> Hi, @Apache9 , when there is only 1 table, just like balance by table (the cluster state only contains distribution info of exactly one table and all the online RSes), the RegionCountSkewCostFunction can replace the TableSkewCostFunction. For the example of [100,0,0], the cost of RegionCountSkewCostFunction is 1, which will generate balance actions in the next steps.
> But when the table regions has already been evenly distributed, for example [1,0,0,0], the cost of TableSkewCostFunction will not be 0, it is 1(the cost of RegionCountSkewCostFunction is 0), but the table does not need to be balanced, it brings many redundant computation steps of actions.
OK, got your point. So you mean if there is only one table, RegionCountSkewCostFunction is enough, and TableSkewCostFunction will generate unnecesary actions.
@clarax WDYT? Is this a valid point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-842940300
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 58s | master passed |
| +1 :green_heart: | compile | 1m 19s | master passed |
| +1 :green_heart: | shadedjars | 8m 59s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 43s | the patch passed |
| +1 :green_heart: | compile | 1m 19s | the patch passed |
| +1 :green_heart: | javac | 1m 19s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 5s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 208m 53s | hbase-server in the patch passed. |
| | | 243m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d1d88bba5754 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 741b4b4674 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/testReport/ |
| Max. process+thread count | 3429 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
sunhelly commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840337094
> I am not sure if this Jira is needed. First of all, cost function should be implemented independently for tuning consideration. if this function is not needed, operator can turn it off by setting the weight to 0.
> Secondly, as I understand, the example would cause a problem no matter how many tables we have. I have another Jira to improve this cost function. https://issues.apache.org/jira/browse/HBASE-25739 which completely rewrite the implementation and fix the problem completely.
Hi, @clarax , thanks for attention.
I think the TableSkewCostFunction can only set the regions be roughly balanced instead of fully balanced from the perspective of one table. For example, if there are two tables on the cluster, the whole cluster distribution is [10,10,10,10], one table distribution is [10,0,10,0], while another table is [0,10,0,10], then the cost of TableSkewCostFunction is 10/30. But when one table distribution is [2,8,2,8], another is [8,2,8,2], then the cost of TableSkewCostFunction is 6/30, smaller than the previous value, and lower the possibility of generate actions. Even generate actions afterwards, how the balancer knows which table is the most unbalanced?
Balancing by table can ensure every table regions be distributed evenly. Hope I can fully get your ideas, I'm a little confused about some problems, can you tell me if you have used balance by table in your clusters, and how many tables whose region count is smaller than the online RS count? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] clarax edited a comment on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
clarax edited a comment on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840342636
Is there really a case with only one table? Balancer actually looks at both system tables and user tables. By default we already have some system tables such as meta table. if we don't want to calculate table skew, as I mentioned, we can set the weight of this cost function to 0 and no calculation will be done or disable regioCoutSkewCostfunction by setting the weight to 0. I don't think we can count on the possibility that we have only one table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] clarax commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
clarax commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840342636
Is there really a case with only one table? Balancer actually looks at both system tables and user tables. By default we already have some system tables such as meta table. if we don't want to calculate table shew, as I mentioned, we can set the weight of this cost function to 0 and no calculation will be done or disable regioCoutSkewCostfunction. I don't think we can count on the possibility that we have only one table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-839664978
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 55s | master passed |
| +1 :green_heart: | compile | 3m 13s | master passed |
| +1 :green_heart: | checkstyle | 1m 4s | master passed |
| +1 :green_heart: | spotbugs | 2m 3s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 35s | the patch passed |
| +1 :green_heart: | compile | 3m 13s | the patch passed |
| +1 :green_heart: | javac | 3m 13s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 3s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 18m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 13s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| | | 47m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4efd97bb3d0c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 630c73fda4 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] clarax commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
clarax commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840371592
> > @sunhelly We used it on a production cluster with about 7 user table and a test cluster with one user table. Both have meta table which has smaller number of regions than the number of nodes.Since I use the same aggregation as regionCountSkewCostFunction, for tables with smaller number of regions say [1,0,0,0], the cost is min so no calculation would be triggered.
> > I simply add the scaled cost for each table so balancer will keep moving until all tables are balanced enough.
>
> Yes, the TableSkewCostFunction should use similar aggregation calculation like RegionCountSkewCostFunction, while it should calculate from the perspective of each table. But I think the calculation may be of low efficiency and won't have big impact on subsequent balancer. For example, if there are 100 tables on a cluster, 98 tables of them have balanced layout, while only 2 is imbalanced. The TableSkewCostFunction can perceive this problem, but the balancer can not choose regions of these two regions to generate actions, and only 2% chose regions belong to these 2 tables. After max steps, if these 2 tables still imbalanced, the balancer should keep generate actions in the next cycle as you have mentioned before.
> This is a problem of gradual balance, it's hard to know when these two tables can be balanced, and what a proper number of minCostNeedBalance should be...I think trying to set "hbase.master.loadbalance.bytable" be true can solve this problem simply...
That is a valid concern. I am working on choosing the proper minCostNeedBalance. Please see the umbrella Jira https://issues.apache.org/jira/browse/HBASE-25697 and your input is welcome. byTable option doesn't work though. Can we move the discussion in the other jiras since it is beyond the scope of this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
sunhelly commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840380126
> > > @sunhelly We used it on a production cluster with about 7 user table and a test cluster with one user table. Both have meta table which has smaller number of regions than the number of nodes.Since I use the same aggregation as regionCountSkewCostFunction, for tables with smaller number of regions say [1,0,0,0], the cost is min so no calculation would be triggered.
> > > I simply add the scaled cost for each table so balancer will keep moving until all tables are balanced enough.
> >
> >
> > Yes, the TableSkewCostFunction should use similar aggregation calculation like RegionCountSkewCostFunction, while it should calculate from the perspective of each table. But I think the calculation may be of low efficiency and won't have big impact on subsequent balancer. For example, if there are 100 tables on a cluster, 98 tables of them have balanced layout, while only 2 is imbalanced. The TableSkewCostFunction can perceive this problem, but the balancer can not choose regions of these two regions to generate actions, and only 2% chose regions belong to these 2 tables. After max steps, if these 2 tables still imbalanced, the balancer should keep generate actions in the next cycle as you have mentioned before.
> > This is a problem of gradual balance, it's hard to know when these two tables can be balanced, and what a proper number of minCostNeedBalance should be...I think trying to set "hbase.master.loadbalance.bytable" be true can solve this problem simply...
>
> That is a valid concern. I am working on choosing the proper minCostNeedBalance. Please see the umbrella Jira https://issues.apache.org/jira/browse/HBASE-25697 and your input is welcome. byTable option doesn't work though. Can we move the discussion in the other jiras since it is beyond the scope of this pr?
Yes, of course. I have watched the issue you referred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-839842524
I think even with only 1 table, we could still compute the table skew count?
For example, we have a table with 100 regions, and there are 3 region servers, but all the regions are placed on a single server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
sunhelly commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840367641
> @sunhelly We used it on a production cluster with about 7 user table and a test cluster with one user table. Both have meta table which has smaller number of regions than the number of nodes.Since I use the same aggregation as regionCountSkewCostFunction, for tables with smaller number of regions say [1,0,0,0], the cost is min so no calculation would be triggered.
>
> I simply add the scaled cost for each table so balancer will keep moving until all tables are balanced enough.
Yes, the TableSkewCostFunction should use similar aggregation calculation like RegionCountSkewCostFunction, while it should calculate from the perspective of each table. But I think the calculation may be of low efficiency and won't have big impact on subsequent balancer. For example, if there are 100 tables on a cluster, 98 tables of them have balanced layout, while only 2 is imbalanced. The TableSkewCostFunction can perceive this problem, but the balancer can not choose regions of these two regions to generate actions, and only 2% chose regions belong to these 2 tables. After max steps, if these 2 tables still imbalanced, the balancer should keep generate actions in the next cycle as you have mentioned before.
This is a problem of gradual balance, it's hard to know when these two tables can be balanced, and what a proper number of minCostNeedBalance should be...I think trying to set "hbase.master.loadbalance.bytable" be true can solve this problem simply...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-842822527
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 42s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 36s | master passed |
| +1 :green_heart: | compile | 4m 22s | master passed |
| +1 :green_heart: | checkstyle | 1m 31s | master passed |
| +1 :green_heart: | spotbugs | 2m 52s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 49s | the patch passed |
| +1 :green_heart: | compile | 4m 6s | the patch passed |
| +1 :green_heart: | javac | 4m 6s | the patch passed |
| -0 :warning: | checkstyle | 1m 21s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 24m 55s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 9s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 63m 59s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6dc46cd7ddbc 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 741b4b4674 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] clarax commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
clarax commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840346195
@sunhelly We used it on a production cluster with about 7 user table and a test cluster with one user table. Both have meta table which has smaller number of regions than the number of nodes.Since I use the same aggregation as regionCountSkewCostFunction, for tables with smaller number of regions say [1,0,0,0], the cost is min so no calculation would be triggered.
I simply add the scaled cost for each table so balancer will keep moving until all tables are balanced enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-839800029
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 9s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 19s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | shadedjars | 8m 57s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 4s | the patch passed |
| +1 :green_heart: | compile | 1m 6s | the patch passed |
| +1 :green_heart: | javac | 1m 6s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 2s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 224m 38s | hbase-server in the patch passed. |
| | | 257m 33s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux ad5fd2c7a9d9 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 630c73fda4 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/testReport/ |
| Max. process+thread count | 3365 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
sunhelly commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-840352608
> > > I think even with only 1 table, we could still compute the table skew count?
> > > For example, we have a table with 100 regions, and there are 3 region servers, but all the regions are placed on a single server?
> >
> >
> > Hi, @Apache9 , when there is only 1 table, just like balance by table (the cluster state only contains distribution info of exactly one table and all the online RSes), the RegionCountSkewCostFunction can replace the TableSkewCostFunction. For the example of [100,0,0], the cost of RegionCountSkewCostFunction is 1, which will generate balance actions in the next steps.
> > But when the table regions has already been evenly distributed, for example [1,0,0,0], the cost of TableSkewCostFunction will not be 0, it is 1(the cost of RegionCountSkewCostFunction is 0), but the table does not need to be balanced, it brings many redundant computation steps of actions.
>
> OK, got your point. So you mean if there is only one table, RegionCountSkewCostFunction is enough, and TableSkewCostFunction will generate unnecesary actions.
>
> @clarax WDYT? Is this a valid point?
Yes, and one table is just a general scenario when we set "hbase.master.loadbalance.bytable" be true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3260: HBASE-25882 TableSkewCostFunction may cost unnecessary calculation st…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3260:
URL: https://github.com/apache/hbase/pull/3260#issuecomment-900295668
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 8s | master passed |
| +1 :green_heart: | compile | 0m 21s | master passed |
| +1 :green_heart: | shadedjars | 9m 6s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | master passed |
| -0 :warning: | patch | 9m 35s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 46s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 2s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 11m 36s | hbase-balancer in the patch passed. |
| | | 43m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3260 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 8cb984fe9c8f 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 721cb96f8c |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/testReport/ |
| Max. process+thread count | 219 (vs. ulimit of 30000) |
| modules | C: hbase-balancer U: hbase-balancer |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3260/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org