You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2022/09/23 16:26:09 UTC
[GitHub] [hbase] ragarkar opened a new pull request, #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
ragarkar opened a new pull request, #4799:
URL: https://github.com/apache/hbase/pull/4799
…lding bucket cache before moving regions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1107232429
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +558,61 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio()[region][server];
+ }
+
+ private float[][] getOrComputeRegionPrefetchRatio() {
+ if (regionServerWithBestPrefetchRatio == null || regionServerPrefetchRatio == null) {
+ computeRegionServerPrefetchRatio();
+ }
+ return regionServerPrefetchRatio;
+ }
+
+ public int[] getOrComputeServerWithBestPrefetchRatio() {
+ if (regionServerWithBestPrefetchRatio == null || regionServerPrefetchRatio == null) {
+ computeRegionServerPrefetchRatio();
+ }
+ return regionServerWithBestPrefetchRatio;
+ }
+
+ private void computeRegionServerPrefetchRatio() {
+ regionServerPrefetchRatio = new float[numRegions][numServers];
+ regionServerWithBestPrefetchRatio = new int[numRegions];
+
+ for (int region = 0; region < numRegions; region++) {
+ float bestPrefetchRatio = 0.0f;
+ int serverWithBestPrefetchRatio = 0;
+ for (int server = 0; server < numServers; server++) {
Review Comment:
Thanks for the explanation, makes sense to me now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1107288610
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
Ok, so `prefetchRatio` here means the current RS hosting the region prefetch ratio. Then, extra question is, when would `bestPrefetchRatio` be 0 here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1397093394
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 7s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 37s | master passed |
| +1 :green_heart: | compile | 2m 18s | master passed |
| +1 :green_heart: | shadedjars | 3m 51s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 21s | master passed |
| -0 :warning: | patch | 5m 49s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 30s | the patch passed |
| +1 :green_heart: | compile | 2m 16s | the patch passed |
| +1 :green_heart: | javac | 2m 16s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 46s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 5s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 42s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 210m 32s | hbase-server in the patch passed. |
| | | 251m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d31127b6e883 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/testReport/ |
| Max. process+thread count | 2721 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1387227748
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 33s | master passed |
| +1 :green_heart: | compile | 4m 29s | master passed |
| +1 :green_heart: | checkstyle | 1m 11s | master passed |
| +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 10s | master passed |
| -0 :warning: | patch | 1m 52s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | the patch passed |
| +1 :green_heart: | compile | 4m 26s | the patch passed |
| +1 :green_heart: | cc | 4m 26s | the patch passed |
| -0 :warning: | javac | 0m 18s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| +1 :green_heart: | checkstyle | 1m 12s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 9m 44s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 40s | the patch passed |
| +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 49m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux f6757abb2e69 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| Max. process+thread count | 81 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1482815953
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 2s | master passed |
| +1 :green_heart: | compile | 1m 37s | master passed |
| +1 :green_heart: | shadedjars | 4m 34s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 2s | master passed |
| -0 :warning: | patch | 6m 7s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 20s | root in the patch failed. |
| -1 :x: | compile | 0m 34s | hbase-server in the patch failed. |
| -0 :warning: | javac | 0m 34s | hbase-server in the patch failed. |
| -1 :x: | shadedjars | 3m 25s | patch has 16 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 40s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 7s | hbase-client in the patch passed. |
| -1 :x: | unit | 18m 34s | hbase-balancer in the patch failed. |
| -1 :x: | unit | 0m 34s | hbase-server in the patch failed. |
| | | 44m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 24acc7d85472 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 230fdc0b50 |
| Default Java | Temurin-1.8.0_352-b08 |
| mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt |
| shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-balancer.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/testReport/ |
| Max. process+thread count | 167 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1492402531
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 24s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 49s | master passed |
| +1 :green_heart: | compile | 2m 38s | master passed |
| +1 :green_heart: | shadedjars | 4m 39s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | master passed |
| -0 :warning: | patch | 6m 16s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 14s | the patch passed |
| +1 :green_heart: | compile | 1m 55s | the patch passed |
| +1 :green_heart: | javac | 1m 55s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 29s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 32s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 0s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 23s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 12m 5s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 201m 40s | hbase-server in the patch passed. |
| | | 247m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 72ff7edd3c45 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 143e9b4ff6 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/testReport/ |
| Max. process+thread count | 2703 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1398043539
rebuild
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1256682143
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 41s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 1s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 21s | master passed |
| +1 :green_heart: | compile | 6m 6s | master passed |
| +1 :green_heart: | checkstyle | 1m 21s | master passed |
| +1 :green_heart: | spotless | 0m 52s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 7m 36s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 13s | the patch passed |
| +1 :green_heart: | compile | 3m 54s | the patch passed |
| +1 :green_heart: | cc | 3m 54s | the patch passed |
| -0 :warning: | javac | 2m 12s | hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193) |
| +1 :green_heart: | checkstyle | 0m 51s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 8m 3s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 25s | the patch passed |
| +1 :green_heart: | spotless | 0m 36s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 50m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 41a300156642 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 748cad655c |
| Default Java | Temurin-1.8.0_345-b01 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.7.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1120400988
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:
##########
@@ -452,7 +459,10 @@ protected List<RegionPlan> balanceTable(TableName tableName,
// Allow turning this feature off if the locality cost is not going to
// be used in any computations.
RegionHDFSBlockLocationFinder finder = null;
- if ((this.localityCost != null) || (this.rackLocalityCost != null)) {
+ if (
+ (this.localityCost != null) || (this.rackLocalityCost != null)
+ || (this.prefetchCacheCost != null)
Review Comment:
But setting the finder is just important if we are interested in calculate hdfs blocks locality, so why setting it for the prefetch function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1452429494
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 52s | master passed |
| +1 :green_heart: | compile | 4m 26s | master passed |
| +1 :green_heart: | checkstyle | 1m 13s | master passed |
| +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 9s | master passed |
| -0 :warning: | patch | 1m 53s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 35s | the patch passed |
| +1 :green_heart: | compile | 4m 30s | the patch passed |
| +1 :green_heart: | cc | 4m 30s | the patch passed |
| -0 :warning: | javac | 0m 17s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| +1 :green_heart: | checkstyle | 1m 18s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 53s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 57s | the patch passed |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 59m 25s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 5713e01763ff 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1257406730
Mind explaining more about the algorithm here?
I guess the problem here is that, when moving a region from rs A to rs B, the block cache on A is useless now and then on rs B, we need to reload the block cache of this region, and it will evict other regions data?
What is the algo here to measure this cost?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1485567530
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 51s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 22s | master passed |
| +1 :green_heart: | compile | 2m 16s | master passed |
| +1 :green_heart: | shadedjars | 4m 19s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | master passed |
| -0 :warning: | patch | 6m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 16s | the patch passed |
| +1 :green_heart: | compile | 2m 15s | the patch passed |
| +1 :green_heart: | javac | 2m 15s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 23s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 4s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 21s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 30s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 214m 30s | hbase-server in the patch passed. |
| | | 257m 16s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 6f2ea392284e 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a6dad700db |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/testReport/ |
| Max. process+thread count | 2855 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1256469822
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 24s | master passed |
| +1 :green_heart: | compile | 1m 34s | master passed |
| +1 :green_heart: | shadedjars | 4m 8s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 9s | the patch passed |
| +1 :green_heart: | compile | 1m 29s | the patch passed |
| +1 :green_heart: | javac | 1m 29s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 5s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. |
| -1 :x: | unit | 10m 49s | hbase-balancer in the patch failed. |
| -1 :x: | unit | 13m 33s | hbase-server in the patch failed. |
| | | 46m 33s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9bb1c29a6ec3 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 748cad655c |
| Default Java | Temurin-1.8.0_345-b01 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-balancer.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/testReport/ |
| Max. process+thread count | 899 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1491393729
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 51s | master passed |
| +1 :green_heart: | compile | 4m 28s | master passed |
| +1 :green_heart: | checkstyle | 1m 9s | master passed |
| +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 3s | master passed |
| -0 :warning: | patch | 1m 49s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 40s | the patch passed |
| +1 :green_heart: | compile | 4m 23s | the patch passed |
| +1 :green_heart: | cc | 4m 23s | the patch passed |
| +1 :green_heart: | javac | 4m 23s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 8s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 33s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 55s | the patch passed |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 49s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 56m 39s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 59e793620f8d 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e5620e26a2 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1482975869
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 4m 56s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 27s | master passed |
| +1 :green_heart: | compile | 2m 16s | master passed |
| +1 :green_heart: | shadedjars | 4m 23s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 21s | master passed |
| -0 :warning: | patch | 6m 25s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 19s | the patch passed |
| +1 :green_heart: | compile | 2m 17s | the patch passed |
| +1 :green_heart: | javac | 2m 17s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 23s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 7s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 26s | hbase-client in the patch passed. |
| -1 :x: | unit | 17m 50s | hbase-balancer in the patch failed. |
| -1 :x: | unit | 10m 54s | hbase-server in the patch failed. |
| | | 64m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 1baba8e233b8 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 735fb43388 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-balancer.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/testReport/ |
| Max. process+thread count | 597 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1387149777
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 24s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 57s | master passed |
| +1 :green_heart: | compile | 3m 28s | master passed |
| +1 :green_heart: | shadedjars | 5m 44s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 36s | master passed |
| -0 :warning: | patch | 7m 55s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 19s | the patch passed |
| +1 :green_heart: | compile | 2m 14s | the patch passed |
| +1 :green_heart: | javac | 2m 14s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 45s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 39s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 13s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 22s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 34s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 207m 47s | hbase-server in the patch passed. |
| | | 253m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 791f06bbc546 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/testReport/ |
| Max. process+thread count | 2733 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1387136101
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 20s | master passed |
| +1 :green_heart: | compile | 1m 39s | master passed |
| +1 :green_heart: | shadedjars | 4m 0s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | master passed |
| -0 :warning: | patch | 5m 35s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 6s | the patch passed |
| +1 :green_heart: | compile | 1m 39s | the patch passed |
| +1 :green_heart: | javac | 1m 39s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 59s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 54s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 206m 55s | hbase-server in the patch passed. |
| | | 244m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d8de9ea15f48 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/testReport/ |
| Max. process+thread count | 2671 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1396540567
recheck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1396733383
rebuild
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1400279869
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 25s | master passed |
| +1 :green_heart: | compile | 2m 15s | master passed |
| +1 :green_heart: | shadedjars | 4m 23s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 19s | master passed |
| -0 :warning: | patch | 6m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 15s | the patch passed |
| +1 :green_heart: | compile | 2m 15s | the patch passed |
| +1 :green_heart: | javac | 2m 15s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 20s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 19s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 6s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 30s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 232m 37s | hbase-server in the patch passed. |
| | | 275m 31s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 5db1539dcd1c 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 83d450d5b5 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/testReport/ |
| Max. process+thread count | 2415 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1123032103
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
Review Comment:
Done. Added javadoc comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1256465753
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 26s | master passed |
| +1 :green_heart: | compile | 3m 57s | master passed |
| +1 :green_heart: | checkstyle | 0m 53s | master passed |
| +1 :green_heart: | spotless | 0m 38s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 4s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 25s | the patch passed |
| +1 :green_heart: | compile | 5m 7s | the patch passed |
| +1 :green_heart: | cc | 5m 7s | the patch passed |
| +1 :green_heart: | javac | 5m 7s | the patch passed |
| -0 :warning: | checkstyle | 0m 9s | hbase-balancer: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| -0 :warning: | checkstyle | 0m 31s | hbase-server: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | hadoopcheck | 5m 44s | The patch causes 10 errors with Hadoop v3.3.4. |
| +1 :green_heart: | hbaseprotoc | 2m 1s | the patch passed |
| -1 :x: | spotless | 0m 21s | patch has 33 errors when running spotless:check, run spotless:apply to fix. |
| -1 :x: | spotbugs | 2m 7s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. |
| | | 41m 28s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hbase-server |
| | Integral division result cast to double or float in org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegion, ClusterStatusProtos$RegionLoad$Builder, HBaseProtos$RegionSpecifier$Builder) At HRegionServer.java:double or float in org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegion, ClusterStatusProtos$RegionLoad$Builder, HBaseProtos$RegionSpecifier$Builder) At HRegionServer.java:[line 1546] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux f4625729b680 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 748cad655c |
| Default Java | Temurin-1.8.0_345-b01 |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-general-check/output/patch-javac-3.3.4.txt |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-general-check/output/patch-spotless.txt |
| spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.7.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1453568650
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 23s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 44s | master passed |
| +1 :green_heart: | compile | 4m 28s | master passed |
| +1 :green_heart: | checkstyle | 1m 13s | master passed |
| +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 3s | master passed |
| -0 :warning: | patch | 1m 50s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 30s | the patch passed |
| +1 :green_heart: | compile | 4m 21s | the patch passed |
| +1 :green_heart: | cc | 4m 21s | the patch passed |
| -0 :warning: | javac | 0m 17s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| +1 :green_heart: | checkstyle | 1m 7s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 55s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 54s | the patch passed |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 43s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 56m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 29d82f520c69 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1124412915
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ /**
+ * Enables or disables the prefetch cache cost function depending on the parameter
+ * PREFETCH_PERSISTENCE_PATH_KEY. If set, this parameter enables the prefetched file list
+ * persistence. If this parameter is not set this means that the cache persistence is disabled
+ * which means that the prefetch ratios of regions on region servers cannot be calculated and
+ * hence the regions should be moved based on how much they have been prefetched on a region
+ * server. The prefetch cache cost function is disabled if the multiplier is set to 0.
Review Comment:
Done. Made the suggested changes to javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1452698500
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 56s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 36s | master passed |
| +1 :green_heart: | compile | 2m 16s | master passed |
| +1 :green_heart: | shadedjars | 4m 24s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 21s | master passed |
| -0 :warning: | patch | 6m 28s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 18s | the patch passed |
| +1 :green_heart: | compile | 2m 15s | the patch passed |
| +1 :green_heart: | javac | 2m 15s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 24s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 21s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 7s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 34s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 28s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 241m 11s | hbase-server in the patch passed. |
| | | 284m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a92594fd137e 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/testReport/ |
| Max. process+thread count | 2970 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1386860589
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 49s | master passed |
| +1 :green_heart: | compile | 4m 28s | master passed |
| +1 :green_heart: | checkstyle | 1m 12s | master passed |
| +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 17s | master passed |
| -0 :warning: | patch | 1m 51s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | the patch passed |
| +1 :green_heart: | compile | 4m 26s | the patch passed |
| +1 :green_heart: | cc | 4m 26s | the patch passed |
| -0 :warning: | javac | 0m 17s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| -0 :warning: | checkstyle | 0m 8s | hbase-balancer: The patch generated 6 new + 1 unchanged - 0 fixed = 7 total (was 1) |
| -0 :warning: | checkstyle | 0m 31s | hbase-server: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 9m 53s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 41s | the patch passed |
| -1 :x: | spotless | 0m 18s | patch has 19 errors when running spotless:check, run spotless:apply to fix. |
| +1 :green_heart: | spotbugs | 5m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 49m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 7eca0ea16fac 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 80 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/3/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1107288610
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
Ok, so `prefetchRatio` here means the current RS hosting the region prefetch ratio. Then, extra question is, when would bestPrefetchRatio be 0 here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1123020734
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:
##########
@@ -452,7 +459,10 @@ protected List<RegionPlan> balanceTable(TableName tableName,
// Allow turning this feature off if the locality cost is not going to
// be used in any computations.
RegionHDFSBlockLocationFinder finder = null;
- if ((this.localityCost != null) || (this.rackLocalityCost != null)) {
+ if (
+ (this.localityCost != null) || (this.rackLocalityCost != null)
+ || (this.prefetchCacheCost != null)
Review Comment:
Removed this code as it is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1121338193
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
The bestPrefetchRatio could be 0 if a region is not yet hosted on any region server which may happen while initialising the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1121272883
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +569,108 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio(region, server);
+ }
+
+ protected float getRegionServerPrefetchRatio(int region, int regionServerIndex) {
+ // Cost this server has from RegionLoad
+ float prefetchRatio = 0.0f;
+
+ // Get the prefetch ratio if the region is currently hosted on this server
+ for (int regionIndex : regionsPerServer[regionServerIndex]) {
+ if (region != regionIndex) {
+ continue;
+ }
+ Deque<BalancerRegionLoad> regionLoadList = regionLoads[regionIndex];
+
+ // The region is currently hosted on this region server. Now, get the prefetch cache ratio
+ // for this region on this region server
+ prefetchRatio =
+ regionLoadList == null ? 0.0f : regionLoadList.getLast().getPrefetchCacheRatio();
+
+ return prefetchRatio;
+ }
+
+ // Region is not currently hosted on this server. Check if the region was prefetched on this
+ // server earlier. This can happen when the server was shutdown and the cache was persisted.
+ // Seartch using the index name and server name and not the index id and server id as these ids
+ // may change when a server is marked as dead or a new server is added.
+ String regionNameAsString = regions[region].getRegionNameAsString();
+ String serverNameAsString = servers[regionServerIndex].getServerName();
+ if (
+ historicalRegionServerPrefetchRatio != null
+ && historicalRegionServerPrefetchRatio.containsKey(regionNameAsString)
+ ) {
+ Map<String, Float> serverPrefetchRatio =
+ historicalRegionServerPrefetchRatio.get(regionNameAsString);
+ if (serverPrefetchRatio.containsKey(serverNameAsString)) {
+ prefetchRatio = serverPrefetchRatio.get(serverNameAsString);
+
+ // The old prefetch cache ratio has been accounted for and hence, clear up this information
+ historicalRegionServerPrefetchRatio.remove(regionNameAsString, serverPrefetchRatio);
+ }
+ }
+ return prefetchRatio;
Review Comment:
This historical prefetch ratio is just an indicator of how much a region was prefetched on a region server before it was moved (because of the region server going down or some other reason) to another region server. This information is mainly used to calculate the difference between the prefetch ratio of this region on a new region server and how much this region was prefetched on the old region server (if any). If the region's prefetch ratio is higher on the currently hosting region server, the region will not be moved back to the old region server anyways. However, this historical prefetch ratio is useful if the region is only partially prefetched on the new region server while the historical prefetch ratio is higher to the extent that it still makes sense to move the region back to the old region server assuming that the region's blocks were persisted in the cache while shutting down the region server. In absence of this metric, it will not be possible to find out how much a re
gion was historically prefetched on a region server. Without this metric, this cost function will never move the regions based on the prefetch ratio as it will not have any historical information to compare with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1483339003
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 28s | master passed |
| +1 :green_heart: | compile | 4m 21s | master passed |
| +1 :green_heart: | checkstyle | 1m 12s | master passed |
| +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 4s | master passed |
| -0 :warning: | patch | 1m 53s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 34s | the patch passed |
| +1 :green_heart: | compile | 4m 25s | the patch passed |
| +1 :green_heart: | cc | 4m 25s | the patch passed |
| +1 :green_heart: | javac | 4m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 8s | hbase-balancer: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 38s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 2m 0s | the patch passed |
| -1 :x: | spotless | 0m 17s | patch has 22 errors when running spotless:check, run spotless:apply to fix. |
| +1 :green_heart: | spotbugs | 5m 43s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 55m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 291d394af8a6 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 735fb43388 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 81 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1492071971
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 29s | master passed |
| +1 :green_heart: | compile | 4m 23s | master passed |
| +1 :green_heart: | checkstyle | 1m 10s | master passed |
| +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 2s | master passed |
| -0 :warning: | patch | 1m 49s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 28s | the patch passed |
| +1 :green_heart: | compile | 4m 22s | the patch passed |
| +1 :green_heart: | cc | 4m 22s | the patch passed |
| +1 :green_heart: | javac | 4m 22s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 11s | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 38s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 55s | the patch passed |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 56m 5s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux d8e008c1e858 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 143e9b4ff6 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1256813017
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 15s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 48s | master passed |
| +1 :green_heart: | compile | 2m 0s | master passed |
| +1 :green_heart: | shadedjars | 3m 55s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 8s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 35s | the patch passed |
| +1 :green_heart: | compile | 2m 0s | the patch passed |
| +1 :green_heart: | javac | 2m 0s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 45s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 11m 46s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 204m 10s | hbase-server in the patch passed. |
| | | 241m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 650e5b89e9e0 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 748cad655c |
| Default Java | Eclipse Adoptium-11.0.16.1+1 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/testReport/ |
| Max. process+thread count | 2708 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r985399745
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private String prefetchedFileListPath;
+ private float prefetchRatio;
+ private float bestPrefetchRatio;
+
+ public static final String PREFETCH_PERSISTENCE_PATH_KEY = "hbase.prefetch.file-list.path";
Review Comment:
Done. I have made this change. It will be visible in the next patch.
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private String prefetchedFileListPath;
+ private float prefetchRatio;
+ private float bestPrefetchRatio;
+
+ public static final String PREFETCH_PERSISTENCE_PATH_KEY = "hbase.prefetch.file-list.path";
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(PREFETCH_PERSISTENCE_PATH_KEY);
+ this.setMultiplier(prefetchedFileListPath == null ? 0 : 1);
Review Comment:
Yes. The whole logic of this cost function depends on the capability of persisting the list of files already prefetched on the server and hence, if the configuration "hbase.prefetch.file-list.path" isn't set, then the cost function will be disabled.
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:
##########
@@ -452,7 +459,10 @@ protected List<RegionPlan> balanceTable(TableName tableName,
// Allow turning this feature off if the locality cost is not going to
// be used in any computations.
RegionHDFSBlockLocationFinder finder = null;
- if ((this.localityCost != null) || (this.rackLocalityCost != null)) {
+ if (
+ (this.localityCost != null) || (this.rackLocalityCost != null)
+ || (this.prefetchCacheCost != null)
Review Comment:
I have put this code just in case, the localityCost and rackLocalityCost functions are disabled by the user configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1448139983
> An update on testing. In addition to adding a unit test which considers different combinations of prefetch ratio on the currently hosting region server as well as historical prefetch ratio, these changes were tested on a real cluster. However, it was only tested for happy condition viz. if the prefetch ratio on a region server was good, then the region was not moved back to the old region server even though it was hosted and prefetched on the older region server in the part. This test was done by disabling all other cost functions by setting their multiplier values to 0. Every cost function provides a parameter which can be set to 0 to disable that particular cost function.
Thanks for the updates! I was looking at the existing functions and based on the respective multipliers default values, I guess region skewness and locality are currently considered the most relevant factors for StochasticLoadBalancer.
The prefetch ratio function would be most relevant for non-hdfs deployments, which then makes locality irrelevant. In those cases, prefetch and region skewness would be competing to prevail over the final balancer decision. We should also document this and mention that the behaviour is configurable via the function multiplier property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1123027979
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +569,108 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio(region, server);
+ }
+
+ protected float getRegionServerPrefetchRatio(int region, int regionServerIndex) {
+ // Cost this server has from RegionLoad
+ float prefetchRatio = 0.0f;
+
+ // Get the prefetch ratio if the region is currently hosted on this server
+ for (int regionIndex : regionsPerServer[regionServerIndex]) {
+ if (region != regionIndex) {
+ continue;
+ }
+ Deque<BalancerRegionLoad> regionLoadList = regionLoads[regionIndex];
+
+ // The region is currently hosted on this region server. Now, get the prefetch cache ratio
+ // for this region on this region server
+ prefetchRatio =
+ regionLoadList == null ? 0.0f : regionLoadList.getLast().getPrefetchCacheRatio();
+
+ return prefetchRatio;
+ }
+
+ // Region is not currently hosted on this server. Check if the region was prefetched on this
+ // server earlier. This can happen when the server was shutdown and the cache was persisted.
+ // Seartch using the index name and server name and not the index id and server id as these ids
+ // may change when a server is marked as dead or a new server is added.
+ String regionNameAsString = regions[region].getRegionNameAsString();
+ String serverNameAsString = servers[regionServerIndex].getServerName();
+ if (
+ historicalRegionServerPrefetchRatio != null
+ && historicalRegionServerPrefetchRatio.containsKey(regionNameAsString)
+ ) {
+ Map<String, Float> serverPrefetchRatio =
+ historicalRegionServerPrefetchRatio.get(regionNameAsString);
+ if (serverPrefetchRatio.containsKey(serverNameAsString)) {
+ prefetchRatio = serverPrefetchRatio.get(serverNameAsString);
+
+ // The old prefetch cache ratio has been accounted for and hence, clear up this information
+ historicalRegionServerPrefetchRatio.remove(regionNameAsString, serverPrefetchRatio);
+ }
+ }
+ return prefetchRatio;
+ }
+
+ private void computeRegionServerPrefetchRatio() {
+ regionIndexServerIndexPrefetchRatio = new HashMap<>();
+ regionServerIndexWithBestPrefetchRatio = new int[numRegions];
+
+ for (int region = 0; region < numRegions; region++) {
+ float bestPrefetchRatio = 0.0f;
+ int serverWithBestPrefetchRatio = 0;
+ for (int server = 0; server < numServers; server++) {
+ float prefetchRatio = getRegionServerPrefetchRatio(region, server);
+ if (prefetchRatio > 0.0f || server == regionIndexToServerIndex[region]) {
+ // A region with prefetch ratio of 0 on a server means nothing. Hence, just make a note
+ // of prefetch only if the prefetch ratio is greater than 0.
+ Map<Integer, Integer> tempMap = new HashMap<>();
Review Comment:
Replaced the Map with Pair. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1106697411
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
+ }
+
+ @Override
+ protected double cost() {
+ return 1 - prefetchRatio;
+ }
+
+ @Override
+ protected void regionMoved(int region, int oldServer, int newServer) {
+ float oldServerPrefetch = cluster.getOrComputeWeightedPrefetchRatio(region, oldServer);
+ float newServerPrefetch = cluster.getOrComputeWeightedPrefetchRatio(region, newServer);
+ float prefetchDelta = newServerPrefetch - oldServerPrefetch;
+ float normalizeDelta = bestPrefetchRatio == 0.0f ? 0.0f : prefetchDelta / bestPrefetchRatio;
+ prefetchRatio += normalizeDelta;
+ }
Review Comment:
This code is triggered when the plan calculated by the balancer cannot be applied as it will increase the overall cost. In this case, the plan is cleaned up and all the updates done to the costs are reverted back through this method implemented on each cost function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1482958873
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 45s | master passed |
| +1 :green_heart: | compile | 4m 26s | master passed |
| +1 :green_heart: | checkstyle | 1m 13s | master passed |
| +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 5s | master passed |
| -0 :warning: | patch | 1m 50s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 29s | the patch passed |
| +1 :green_heart: | compile | 4m 23s | the patch passed |
| +1 :green_heart: | cc | 4m 23s | the patch passed |
| +1 :green_heart: | javac | 4m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 7s | hbase-balancer: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 36s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 55s | the patch passed |
| -1 :x: | spotless | 0m 17s | patch has 22 errors when running spotless:check, run spotless:apply to fix. |
| +1 :green_heart: | spotbugs | 5m 45s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 56m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux b4c2cd994e05 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 735fb43388 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1485203423
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 32s | master passed |
| +1 :green_heart: | compile | 4m 19s | master passed |
| +1 :green_heart: | checkstyle | 1m 15s | master passed |
| +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 4m 52s | master passed |
| -0 :warning: | patch | 1m 46s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 18s | the patch passed |
| +1 :green_heart: | compile | 4m 18s | the patch passed |
| +1 :green_heart: | cc | 4m 18s | the patch passed |
| +1 :green_heart: | javac | 4m 18s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 9s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 12m 44s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 2m 7s | the patch passed |
| +1 :green_heart: | spotless | 0m 49s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 6m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 56m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux fd921b3fa0ff 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a6dad700db |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1264968926
> Also, does this new cost function always be enabled by default on the StochasticLoadBalancer?
The PrefetchCacheCostFunction will be enabled if the configuration "hbase.prefetch.file.list.path" is set. In addition to that, it will have it's own configuration "hbase.master.balancer.stochastic.prefetchCacheCost" that will define the multiplier value for this cost function. This is covered in the new patch that I will upload soon. If either "hbase.prefetch.file.list.path" is not set or "hbase.master.balancer.stochastic.prefetchCacheCost" is set to 0, the cost function will not be enabled. The default value of "hbase.master.balancer.stochastic.prefetchCacheCost" will be > 0 which will mean that the cost function will be enabled by default given that the configuration "hbase.prefetch.file.list.path" is set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r979840582
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private String prefetchedFileListPath;
+ private float prefetchRatio;
+ private float bestPrefetchRatio;
+
+ public static final String PREFETCH_PERSISTENCE_PATH_KEY = "hbase.prefetch.file-list.path";
Review Comment:
nit: import static CacheConfig.PREFETCH_PERSISTENCE_PATH_KEY
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +558,61 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio()[region][server];
+ }
+
+ private float[][] getOrComputeRegionPrefetchRatio() {
+ if (regionServerWithBestPrefetchRatio == null || regionServerPrefetchRatio == null) {
+ computeRegionServerPrefetchRatio();
+ }
+ return regionServerPrefetchRatio;
+ }
+
+ public int[] getOrComputeServerWithBestPrefetchRatio() {
+ if (regionServerWithBestPrefetchRatio == null || regionServerPrefetchRatio == null) {
+ computeRegionServerPrefetchRatio();
+ }
+ return regionServerWithBestPrefetchRatio;
+ }
+
+ private void computeRegionServerPrefetchRatio() {
+ regionServerPrefetchRatio = new float[numRegions][numServers];
+ regionServerWithBestPrefetchRatio = new int[numRegions];
+
+ for (int region = 0; region < numRegions; region++) {
+ float bestPrefetchRatio = 0.0f;
+ int serverWithBestPrefetchRatio = 0;
+ for (int server = 0; server < numServers; server++) {
Review Comment:
Shouldn't we be iterating through servers first, then going through the regions on each of the servers? The way we are doing now, we may be over iterating through servers that don't hold the current region.
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:
##########
@@ -256,6 +262,7 @@ protected void loadConf(Configuration conf) {
addCostFunction(new WriteRequestCostFunction(conf));
addCostFunction(new MemStoreSizeCostFunction(conf));
addCostFunction(new StoreFileCostFunction(conf));
+ addCostFunction(prefetchCacheCost);
Review Comment:
Please update the class JavaDoc to mention this additional cost function.
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private String prefetchedFileListPath;
+ private float prefetchRatio;
+ private float bestPrefetchRatio;
+
+ public static final String PREFETCH_PERSISTENCE_PATH_KEY = "hbase.prefetch.file-list.path";
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(PREFETCH_PERSISTENCE_PATH_KEY);
+ this.setMultiplier(prefetchedFileListPath == null ? 0 : 1);
Review Comment:
Is this effectively disabling the feature if "hbase.prefetch.file-list.path" isn't set?
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchBasedCandidateGenerator.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import java.util.Optional;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+class PrefetchBasedCandidateGenerator extends CandidateGenerator {
Review Comment:
How the StochasticLoadBalancer picks this generator? From my readings, it seems it's selecting random generators. If so, does this mean we'll not always have an accurate prefetch ratio for the related cost function?
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:
##########
@@ -452,7 +459,10 @@ protected List<RegionPlan> balanceTable(TableName tableName,
// Allow turning this feature off if the locality cost is not going to
// be used in any computations.
RegionHDFSBlockLocationFinder finder = null;
- if ((this.localityCost != null) || (this.rackLocalityCost != null)) {
+ if (
+ (this.localityCost != null) || (this.rackLocalityCost != null)
+ || (this.prefetchCacheCost != null)
Review Comment:
Do we need to care about locality for the prefetchCacheCost ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1387516515
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 40s | master passed |
| +1 :green_heart: | compile | 2m 4s | master passed |
| +1 :green_heart: | shadedjars | 3m 58s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | master passed |
| -0 :warning: | patch | 5m 37s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 24s | the patch passed |
| +1 :green_heart: | compile | 1m 57s | the patch passed |
| +1 :green_heart: | javac | 1m 57s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 57s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 0s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 20s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 49s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 202m 59s | hbase-server in the patch passed. |
| | | 242m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux b09d9fa6c0df 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/testReport/ |
| Max. process+thread count | 2632 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1387538902
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 21s | master passed |
| +1 :green_heart: | compile | 1m 41s | master passed |
| +1 :green_heart: | shadedjars | 4m 7s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | master passed |
| -0 :warning: | patch | 5m 43s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 9s | the patch passed |
| +1 :green_heart: | compile | 1m 40s | the patch passed |
| +1 :green_heart: | javac | 1m 40s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 5s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. |
| -1 :x: | unit | 20m 1s | hbase-balancer in the patch failed. |
| -1 :x: | unit | 207m 28s | hbase-server in the patch failed. |
| | | 254m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 74c6d5416c80 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Temurin-1.8.0_352-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-balancer.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/testReport/ |
| Max. process+thread count | 2647 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/4/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1399803589
The test reported failing in the PR seems to be failing inconsistently as it passed when tested on a local build. Also, the failure do not seem to be related to the code changes in this PR.
Attempting to rebuild this PR once again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1124330646
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ /**
+ * Enables or disables the prefetch cache cost function depending on the parameter
+ * PREFETCH_PERSISTENCE_PATH_KEY. If set, this parameter enables the prefetched file list
+ * persistence. If this parameter is not set this means that the cache persistence is disabled
+ * which means that the prefetch ratios of regions on region servers cannot be calculated and
+ * hence the regions should be moved based on how much they have been prefetched on a region
+ * server. The prefetch cache cost function is disabled if the multiplier is set to 0.
Review Comment:
Let's explicitly mention it's disabled by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1452333281
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 5m 0s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 28s | master passed |
| +1 :green_heart: | compile | 2m 17s | master passed |
| +1 :green_heart: | shadedjars | 4m 23s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 26s | master passed |
| -0 :warning: | patch | 6m 33s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 25s | the patch passed |
| +1 :green_heart: | compile | 2m 18s | the patch passed |
| +1 :green_heart: | javac | 2m 18s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 22s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 22s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 6s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 21s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 27s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 213m 20s | hbase-server in the patch passed. |
| | | 260m 22s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d1a59b7adee8 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/7/testReport/ |
| Max. process+thread count | 2475 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/7/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1482821883
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 1s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 42s | master passed |
| +1 :green_heart: | compile | 5m 43s | master passed |
| +1 :green_heart: | checkstyle | 1m 36s | master passed |
| +1 :green_heart: | spotless | 0m 49s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 6m 25s | master passed |
| -0 :warning: | patch | 2m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 2m 12s | root in the patch failed. |
| -1 :x: | compile | 1m 27s | hbase-server in the patch failed. |
| -0 :warning: | cc | 1m 27s | hbase-server in the patch failed. |
| -0 :warning: | javac | 0m 19s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| -0 :warning: | javac | 1m 27s | hbase-server in the patch failed. |
| +1 :green_heart: | checkstyle | 1m 18s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | hadoopcheck | 2m 34s | The patch causes 16 errors with Hadoop v3.2.4. |
| -1 :x: | hadoopcheck | 6m 15s | The patch causes 16 errors with Hadoop v3.3.4. |
| -1 :x: | hbaseprotoc | 0m 34s | hbase-server in the patch failed. |
| +1 :green_heart: | spotless | 0m 48s | patch has no errors when running spotless:check. |
| -1 :x: | spotbugs | 0m 33s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 46m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 98beefd25078 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 230fdc0b50 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-compile-hbase-server.txt |
| cc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-compile-hbase-server.txt |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-compile-hbase-server.txt |
| hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-javac-3.2.4.txt |
| hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-javac-3.3.4.txt |
| hbaseprotoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-hbaseprotoc-hbase-server.txt |
| spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1452686216
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 24s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 50s | master passed |
| +1 :green_heart: | compile | 1m 41s | master passed |
| +1 :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | master passed |
| -0 :warning: | patch | 6m 10s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 1s | the patch passed |
| +1 :green_heart: | compile | 1m 43s | the patch passed |
| +1 :green_heart: | javac | 1m 43s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 37s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 56s | hbase-balancer in the patch passed. |
| -1 :x: | unit | 239m 1s | hbase-server in the patch failed. |
| | | 278m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 6fd2e2c9f20d 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Temurin-1.8.0_352-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/testReport/ |
| Max. process+thread count | 2413 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/8/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1453850863
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 54s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 13s | master passed |
| +1 :green_heart: | compile | 2m 18s | master passed |
| +1 :green_heart: | shadedjars | 4m 22s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | master passed |
| -0 :warning: | patch | 6m 24s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 18s | the patch passed |
| +1 :green_heart: | compile | 2m 20s | the patch passed |
| +1 :green_heart: | javac | 2m 20s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 21s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 5s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 21s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 29s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 214m 4s | hbase-server in the patch passed. |
| | | 256m 49s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux c3af046b35c1 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/testReport/ |
| Max. process+thread count | 2652 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1483597868
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 58s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 29s | master passed |
| +1 :green_heart: | compile | 2m 16s | master passed |
| +1 :green_heart: | shadedjars | 4m 26s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 22s | master passed |
| -0 :warning: | patch | 6m 30s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 14s | the patch passed |
| +1 :green_heart: | compile | 2m 19s | the patch passed |
| +1 :green_heart: | javac | 2m 19s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 25s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 6s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 40s | hbase-balancer in the patch passed. |
| -1 :x: | unit | 240m 45s | hbase-server in the patch failed. |
| | | 284m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux ee7668bb657d 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 735fb43388 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/testReport/ |
| Max. process+thread count | 2590 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1483565804
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 51s | master passed |
| +1 :green_heart: | compile | 1m 43s | master passed |
| +1 :green_heart: | shadedjars | 4m 40s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | master passed |
| -0 :warning: | patch | 6m 16s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 47s | the patch passed |
| +1 :green_heart: | compile | 1m 43s | the patch passed |
| +1 :green_heart: | javac | 1m 43s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 43s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 10s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 56s | hbase-balancer in the patch passed. |
| -1 :x: | unit | 214m 37s | hbase-server in the patch failed. |
| | | 254m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d47d78c6e82c 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 735fb43388 |
| Default Java | Temurin-1.8.0_352-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/testReport/ |
| Max. process+thread count | 2645 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/12/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1482953951
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 41s | master passed |
| +1 :green_heart: | compile | 1m 38s | master passed |
| +1 :green_heart: | shadedjars | 4m 32s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 2s | master passed |
| -0 :warning: | patch | 6m 4s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 43s | the patch passed |
| +1 :green_heart: | compile | 1m 38s | the patch passed |
| +1 :green_heart: | javac | 1m 38s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 31s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 26s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 43s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 7s | hbase-client in the patch passed. |
| -1 :x: | unit | 18m 30s | hbase-balancer in the patch failed. |
| -1 :x: | unit | 8m 11s | hbase-server in the patch failed. |
| | | 53m 22s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 0b7cd8b4da48 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 735fb43388 |
| Default Java | Temurin-1.8.0_352-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-balancer.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/testReport/ |
| Max. process+thread count | 571 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/11/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1491644165
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 56s | master passed |
| +1 :green_heart: | compile | 1m 56s | master passed |
| +1 :green_heart: | shadedjars | 4m 33s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | master passed |
| -0 :warning: | patch | 6m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 16s | the patch passed |
| +1 :green_heart: | compile | 1m 57s | the patch passed |
| +1 :green_heart: | javac | 1m 57s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 29s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 30s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 56s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 18s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 43s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 196m 44s | hbase-server in the patch passed. |
| | | 238m 30s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 0006a10edb96 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e5620e26a2 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/testReport/ |
| Max. process+thread count | 2616 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1491649853
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 24s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 5s | master passed |
| +1 :green_heart: | compile | 1m 43s | master passed |
| +1 :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 2s | master passed |
| -0 :warning: | patch | 6m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 46s | the patch passed |
| +1 :green_heart: | compile | 1m 39s | the patch passed |
| +1 :green_heart: | javac | 1m 39s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 37s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 23s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 8s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 50s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 204m 16s | hbase-server in the patch passed. |
| | | 243m 16s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 75f73591994c 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e5620e26a2 |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/testReport/ |
| Max. process+thread count | 2683 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/14/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1400005646
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 44s | master passed |
| +1 :green_heart: | compile | 4m 24s | master passed |
| +1 :green_heart: | checkstyle | 1m 9s | master passed |
| +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 16s | master passed |
| -0 :warning: | patch | 1m 52s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 20s | the patch passed |
| +1 :green_heart: | compile | 4m 25s | the patch passed |
| +1 :green_heart: | cc | 4m 25s | the patch passed |
| -0 :warning: | javac | 0m 18s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| +1 :green_heart: | checkstyle | 1m 12s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 33s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 5m 40s | the patch passed |
| +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 49s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 60m 30s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 1dd8a772fa6d 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 83d450d5b5 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1106698856
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
+ }
+
+ @Override
+ protected double cost() {
+ return 1 - prefetchRatio;
+ }
Review Comment:
Yes. That is correct. The overall cost is calculated after reviewing all the cost functions. If the cost is below a minimum threshold, no action is taken. To be in agreement with this, the prefetch based cost function calculates the cost this way viz. a higher prefetch means that the cluster is more stable and hence, the overall cost of running the cluster in this state is lower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1264965698
> I do not fully understand why higher prefetched ratio leads to lower moving cost, if a region's hfiles have all been cached, after moving we need to fetched all the hfiles again, which is costly?
If we are moving a region from server A to server B, if the prefetch ratio on B is higher than that on A, then the cost of moving the region is low. During region movement, there is a likelihood that the file is already prefetched on server B and hence there will be no need to prefetch it again. The changes done in HBASE-27313 already maintain a list of files which are already prefetched on the server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1386790468
rebuild
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1396837398
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 1s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 17s | master passed |
| +1 :green_heart: | compile | 5m 17s | master passed |
| +1 :green_heart: | checkstyle | 1m 26s | master passed |
| +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 6m 11s | master passed |
| -0 :warning: | patch | 2m 19s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 8s | the patch passed |
| +1 :green_heart: | compile | 5m 13s | the patch passed |
| +1 :green_heart: | cc | 5m 13s | the patch passed |
| -0 :warning: | javac | 0m 22s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| +1 :green_heart: | checkstyle | 1m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 10m 49s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 2m 8s | the patch passed |
| +1 :green_heart: | spotless | 0m 48s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 7m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. |
| | | 57m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux 577325c8d88b 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| Max. process+thread count | 81 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1397095868
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 14s | master passed |
| +1 :green_heart: | compile | 1m 42s | master passed |
| +1 :green_heart: | shadedjars | 4m 31s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 15s | master passed |
| -0 :warning: | patch | 6m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | the patch passed |
| +1 :green_heart: | compile | 1m 45s | the patch passed |
| +1 :green_heart: | javac | 1m 45s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 16s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 24s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 47s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 36s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 10m 12s | hbase-balancer in the patch passed. |
| -1 :x: | unit | 212m 48s | hbase-server in the patch failed. |
| | | 252m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 42d364c15fc4 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ad8f28e297 |
| Default Java | Temurin-1.8.0_352-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/testReport/ |
| Max. process+thread count | 2883 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/5/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1257548111
> Mind explaining more about the algorithm here?
>
> I guess the problem here is that, when moving a region from rs A to rs B, the block cache on A is useless now and then on rs B, we need to reload the block cache of this region, and it will evict other regions data?
>
> What is the algo here to measure this cost?
>
> Thanks.
Hello,
All the region servers maintain a list of all the HFiles which are already cached. This change was done as part of HBASE-27313.
The current JIRA uses this information to find out the ratio of files already cached to the total number of files in the region server and puts this information in the region specific metric.
The stochastic load balancer uses this information to find out the cost of moving a region from one region server to the other by comparing the ratio of files already prefetched. Higher the ratio of files prefetched on a region server, lower the cost of moving the region.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1257718013
I do not fully understand why higher prefetched ratio leads to lower moving cost, if a region's hfiles have all been cached, after moving we need to fetched all the hfiles again, which is costly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1452008032
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 46s | master passed |
| +1 :green_heart: | compile | 4m 25s | master passed |
| +1 :green_heart: | checkstyle | 1m 10s | master passed |
| +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 5m 3s | master passed |
| -0 :warning: | patch | 1m 51s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 25s | the patch passed |
| +1 :green_heart: | compile | 4m 24s | the patch passed |
| +1 :green_heart: | cc | 4m 24s | the patch passed |
| -0 :warning: | javac | 0m 18s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) |
| +1 :green_heart: | checkstyle | 1m 10s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 48s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | hbaseprotoc | 1m 55s | the patch passed |
| -1 :x: | spotless | 0m 17s | patch has 56 errors when running spotless:check, run spotless:apply to fix. |
| +1 :green_heart: | spotbugs | 5m 41s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 57m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool |
| uname | Linux bd7689fa4c91 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/7/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/7/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/7/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1124335455
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ /**
+ * Enables or disables the prefetch cache cost function depending on the parameter
+ * PREFETCH_PERSISTENCE_PATH_KEY. If set, this parameter enables the prefetched file list
+ * persistence. If this parameter is not set this means that the cache persistence is disabled
+ * which means that the prefetch ratios of regions on region servers cannot be calculated and
+ * hence the regions should be moved based on how much they have been prefetched on a region
+ * server. The prefetch cache cost function is disabled if the multiplier is set to 0.
Review Comment:
Let's also mention this:
The prefetch ratio function would be most relevant for non-hdfs deployments, which then makes locality irrelevant. In those cases, prefetch and region skewness would be competing to prevail over the final balancer decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1117095821
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchBasedCandidateGenerator.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import java.util.Optional;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+class PrefetchBasedCandidateGenerator extends CandidateGenerator {
Review Comment:
Ok, so generators weights are updated by each cost function, and the higher the weight of the generator, the higher the likelihood for it being picked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1400250507
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 44s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 3s | master passed |
| +1 :green_heart: | compile | 1m 42s | master passed |
| +1 :green_heart: | shadedjars | 4m 39s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | master passed |
| -0 :warning: | patch | 6m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 43s | the patch passed |
| +1 :green_heart: | compile | 1m 42s | the patch passed |
| +1 :green_heart: | javac | 1m 42s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 42s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 53s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 210m 57s | hbase-server in the patch passed. |
| | | 250m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9690dcd94577 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 83d450d5b5 |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/testReport/ |
| Max. process+thread count | 2641 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/6/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1107288610
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
Ok, so `prefetchRation` here means the current RS hosting the region prefetch ratio.
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
Ok, so `prefetchRatio` here means the current RS hosting the region prefetch ratio.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1492398138
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 39s | master passed |
| +1 :green_heart: | compile | 1m 38s | master passed |
| +1 :green_heart: | shadedjars | 4m 34s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | master passed |
| -0 :warning: | patch | 6m 4s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 40s | the patch passed |
| +1 :green_heart: | compile | 1m 37s | the patch passed |
| +1 :green_heart: | javac | 1m 37s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 32s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 24s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 7s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 9m 53s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 204m 14s | hbase-server in the patch passed. |
| | | 242m 39s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 23280f0d1d2a 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 143e9b4ff6 |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/testReport/ |
| Max. process+thread count | 2514 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/15/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1485579026
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 15s | master passed |
| +1 :green_heart: | compile | 2m 2s | master passed |
| +1 :green_heart: | shadedjars | 5m 20s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 15s | master passed |
| -0 :warning: | patch | 7m 12s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 24s | the patch passed |
| +1 :green_heart: | compile | 2m 4s | the patch passed |
| +1 :green_heart: | javac | 2m 4s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 2s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 15s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 59s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 22s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 10m 23s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 223m 43s | hbase-server in the patch passed. |
| | | 267m 44s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux b8535d67f74e 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a6dad700db |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/testReport/ |
| Max. process+thread count | 2697 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/13/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1256810272
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 4s | master passed |
| +1 :green_heart: | compile | 1m 26s | master passed |
| +1 :green_heart: | shadedjars | 4m 7s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 5s | the patch passed |
| +1 :green_heart: | compile | 1m 27s | the patch passed |
| +1 :green_heart: | javac | 1m 27s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 5s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 8m 48s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 202m 22s | hbase-server in the patch passed. |
| | | 232m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux e7ca5ee4f10e 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 748cad655c |
| Default Java | Temurin-1.8.0_345-b01 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/testReport/ |
| Max. process+thread count | 2579 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1256474072
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 51s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 37s | master passed |
| +1 :green_heart: | compile | 1m 40s | master passed |
| +1 :green_heart: | shadedjars | 3m 56s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | the patch passed |
| +1 :green_heart: | compile | 1m 41s | the patch passed |
| +1 :green_heart: | javac | 1m 41s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 53s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 19s | hbase-client in the patch passed. |
| -1 :x: | unit | 12m 1s | hbase-balancer in the patch failed. |
| -1 :x: | unit | 14m 53s | hbase-server in the patch failed. |
| | | 49m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9329e847d468 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 748cad655c |
| Default Java | Eclipse Adoptium-11.0.16.1+1 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-balancer.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/testReport/ |
| Max. process+thread count | 961 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1115759540
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
+ }
+
+ @Override
+ protected double cost() {
+ return 1 - prefetchRatio;
+ }
+
+ @Override
+ protected void regionMoved(int region, int oldServer, int newServer) {
+ float oldServerPrefetch = cluster.getOrComputeWeightedPrefetchRatio(region, oldServer);
+ float newServerPrefetch = cluster.getOrComputeWeightedPrefetchRatio(region, newServer);
+ float prefetchDelta = newServerPrefetch - oldServerPrefetch;
+ float normalizeDelta = bestPrefetchRatio == 0.0f ? 0.0f : prefetchDelta / bestPrefetchRatio;
+ prefetchRatio += normalizeDelta;
+ }
Review Comment:
Actually, this method is called every time we are evaluating a possible region move. The `bestPrefetchRatio` is only calculated at `init` time, and if it's 0, it means the original assignment is already the best one, so this region move would be a high cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1099092422
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +569,108 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio(region, server);
+ }
+
+ protected float getRegionServerPrefetchRatio(int region, int regionServerIndex) {
+ // Cost this server has from RegionLoad
+ float prefetchRatio = 0.0f;
+
+ // Get the prefetch ratio if the region is currently hosted on this server
+ for (int regionIndex : regionsPerServer[regionServerIndex]) {
+ if (region != regionIndex) {
+ continue;
+ }
+ Deque<BalancerRegionLoad> regionLoadList = regionLoads[regionIndex];
+
+ // The region is currently hosted on this region server. Now, get the prefetch cache ratio
+ // for this region on this region server
+ prefetchRatio =
+ regionLoadList == null ? 0.0f : regionLoadList.getLast().getPrefetchCacheRatio();
+
+ return prefetchRatio;
+ }
+
+ // Region is not currently hosted on this server. Check if the region was prefetched on this
+ // server earlier. This can happen when the server was shutdown and the cache was persisted.
+ // Seartch using the index name and server name and not the index id and server id as these ids
+ // may change when a server is marked as dead or a new server is added.
+ String regionNameAsString = regions[region].getRegionNameAsString();
+ String serverNameAsString = servers[regionServerIndex].getServerName();
+ if (
+ historicalRegionServerPrefetchRatio != null
+ && historicalRegionServerPrefetchRatio.containsKey(regionNameAsString)
+ ) {
+ Map<String, Float> serverPrefetchRatio =
+ historicalRegionServerPrefetchRatio.get(regionNameAsString);
+ if (serverPrefetchRatio.containsKey(serverNameAsString)) {
+ prefetchRatio = serverPrefetchRatio.get(serverNameAsString);
+
+ // The old prefetch cache ratio has been accounted for and hence, clear up this information
+ historicalRegionServerPrefetchRatio.remove(regionNameAsString, serverPrefetchRatio);
+ }
+ }
+ return prefetchRatio;
+ }
+
+ private void computeRegionServerPrefetchRatio() {
+ regionIndexServerIndexPrefetchRatio = new HashMap<>();
+ regionServerIndexWithBestPrefetchRatio = new int[numRegions];
+
+ for (int region = 0; region < numRegions; region++) {
+ float bestPrefetchRatio = 0.0f;
+ int serverWithBestPrefetchRatio = 0;
+ for (int server = 0; server < numServers; server++) {
+ float prefetchRatio = getRegionServerPrefetchRatio(region, server);
+ if (prefetchRatio > 0.0f || server == regionIndexToServerIndex[region]) {
+ // A region with prefetch ratio of 0 on a server means nothing. Hence, just make a note
+ // of prefetch only if the prefetch ratio is greater than 0.
+ Map<Integer, Integer> tempMap = new HashMap<>();
Review Comment:
Do we really need a map here? Couldn't this be a pair?
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +569,108 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio(region, server);
+ }
+
+ protected float getRegionServerPrefetchRatio(int region, int regionServerIndex) {
+ // Cost this server has from RegionLoad
+ float prefetchRatio = 0.0f;
+
+ // Get the prefetch ratio if the region is currently hosted on this server
+ for (int regionIndex : regionsPerServer[regionServerIndex]) {
+ if (region != regionIndex) {
+ continue;
+ }
+ Deque<BalancerRegionLoad> regionLoadList = regionLoads[regionIndex];
+
+ // The region is currently hosted on this region server. Now, get the prefetch cache ratio
+ // for this region on this region server
+ prefetchRatio =
+ regionLoadList == null ? 0.0f : regionLoadList.getLast().getPrefetchCacheRatio();
+
+ return prefetchRatio;
+ }
+
+ // Region is not currently hosted on this server. Check if the region was prefetched on this
+ // server earlier. This can happen when the server was shutdown and the cache was persisted.
+ // Seartch using the index name and server name and not the index id and server id as these ids
+ // may change when a server is marked as dead or a new server is added.
+ String regionNameAsString = regions[region].getRegionNameAsString();
+ String serverNameAsString = servers[regionServerIndex].getServerName();
+ if (
+ historicalRegionServerPrefetchRatio != null
+ && historicalRegionServerPrefetchRatio.containsKey(regionNameAsString)
+ ) {
+ Map<String, Float> serverPrefetchRatio =
+ historicalRegionServerPrefetchRatio.get(regionNameAsString);
+ if (serverPrefetchRatio.containsKey(serverNameAsString)) {
+ prefetchRatio = serverPrefetchRatio.get(serverNameAsString);
+
+ // The old prefetch cache ratio has been accounted for and hence, clear up this information
+ historicalRegionServerPrefetchRatio.remove(regionNameAsString, serverPrefetchRatio);
+ }
+ }
+ return prefetchRatio;
Review Comment:
Is this something we can really trust? Because this may not be the real prefetch ratio anymore, we are relying on region metrics from the past, but what if this RS has already evicted some of this region's blocks?
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +569,108 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio(region, server);
+ }
+
+ protected float getRegionServerPrefetchRatio(int region, int regionServerIndex) {
+ // Cost this server has from RegionLoad
+ float prefetchRatio = 0.0f;
+
+ // Get the prefetch ratio if the region is currently hosted on this server
+ for (int regionIndex : regionsPerServer[regionServerIndex]) {
+ if (region != regionIndex) {
+ continue;
+ }
+ Deque<BalancerRegionLoad> regionLoadList = regionLoads[regionIndex];
+
+ // The region is currently hosted on this region server. Now, get the prefetch cache ratio
+ // for this region on this region server
+ prefetchRatio =
+ regionLoadList == null ? 0.0f : regionLoadList.getLast().getPrefetchCacheRatio();
+
+ return prefetchRatio;
+ }
+
+ // Region is not currently hosted on this server. Check if the region was prefetched on this
+ // server earlier. This can happen when the server was shutdown and the cache was persisted.
+ // Seartch using the index name and server name and not the index id and server id as these ids
+ // may change when a server is marked as dead or a new server is added.
+ String regionNameAsString = regions[region].getRegionNameAsString();
+ String serverNameAsString = servers[regionServerIndex].getServerName();
+ if (
+ historicalRegionServerPrefetchRatio != null
+ && historicalRegionServerPrefetchRatio.containsKey(regionNameAsString)
+ ) {
+ Map<String, Float> serverPrefetchRatio =
+ historicalRegionServerPrefetchRatio.get(regionNameAsString);
+ if (serverPrefetchRatio.containsKey(serverNameAsString)) {
+ prefetchRatio = serverPrefetchRatio.get(serverNameAsString);
+
+ // The old prefetch cache ratio has been accounted for and hence, clear up this information
+ historicalRegionServerPrefetchRatio.remove(regionNameAsString, serverPrefetchRatio);
+ }
+ }
+ return prefetchRatio;
+ }
+
+ private void computeRegionServerPrefetchRatio() {
+ regionIndexServerIndexPrefetchRatio = new HashMap<>();
+ regionServerIndexWithBestPrefetchRatio = new int[numRegions];
+
+ for (int region = 0; region < numRegions; region++) {
+ float bestPrefetchRatio = 0.0f;
+ int serverWithBestPrefetchRatio = 0;
+ for (int server = 0; server < numServers; server++) {
+ float prefetchRatio = getRegionServerPrefetchRatio(region, server);
+ if (prefetchRatio > 0.0f || server == regionIndexToServerIndex[region]) {
+ // A region with prefetch ratio of 0 on a server means nothing. Hence, just make a note
+ // of prefetch only if the prefetch ratio is greater than 0.
Review Comment:
I'm not following these comments. Clearly, we only put a prefetch ratio of 0 if it's the server hosting the region. It sounds one more reason to iterate through the RS level first (see my comments from Sep above).
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
I'm a bit confused here. When would prefetchRatio be different from bestPrefetchRatio?
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
+ }
+
+ @Override
+ protected double cost() {
+ return 1 - prefetchRatio;
+ }
Review Comment:
So higher prefetch means lower cost. I guess this is to influence the decision made by StochasticLoadBalancer.needsBalance() method.
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
Review Comment:
Please explain this in the class javadoc.
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
+ }
+
+ @Override
+ protected double cost() {
+ return 1 - prefetchRatio;
+ }
+
+ @Override
+ protected void regionMoved(int region, int oldServer, int newServer) {
+ float oldServerPrefetch = cluster.getOrComputeWeightedPrefetchRatio(region, oldServer);
+ float newServerPrefetch = cluster.getOrComputeWeightedPrefetchRatio(region, newServer);
+ float prefetchDelta = newServerPrefetch - oldServerPrefetch;
+ float normalizeDelta = bestPrefetchRatio == 0.0f ? 0.0f : prefetchDelta / bestPrefetchRatio;
+ prefetchRatio += normalizeDelta;
+ }
Review Comment:
This is also difficult to follow. Shouldn't this just repeat same calculations from `prepare()`, but using the newServerPrefetch in the place of ` cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1123031306
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:
##########
@@ -256,6 +262,7 @@ protected void loadConf(Configuration conf) {
addCostFunction(new WriteRequestCostFunction(conf));
addCostFunction(new MemStoreSizeCostFunction(conf));
addCostFunction(new StoreFileCostFunction(conf));
+ addCostFunction(prefetchCacheCost);
Review Comment:
Done. Added references to prefetchCacheCostFunction to the javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "wchevreuil (via GitHub)" <gi...@apache.org>.
wchevreuil commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1124331055
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
Gotcha, thanks for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1453839189
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 23s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 2s | master passed |
| +1 :green_heart: | compile | 1m 40s | master passed |
| +1 :green_heart: | shadedjars | 4m 34s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | master passed |
| -0 :warning: | patch | 6m 8s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 46s | the patch passed |
| +1 :green_heart: | compile | 1m 40s | the patch passed |
| +1 :green_heart: | javac | 1m 40s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 33s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 24s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 42s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 8s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 11m 51s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 206m 46s | hbase-server in the patch passed. |
| | | 247m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 010a7742cc21 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d2b0074f7a |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/testReport/ |
| Max. process+thread count | 2652 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/9/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1482813950
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 55s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 27s | master passed |
| +1 :green_heart: | compile | 2m 16s | master passed |
| +1 :green_heart: | shadedjars | 4m 20s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | master passed |
| -0 :warning: | patch | 6m 23s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 49s | root in the patch failed. |
| -1 :x: | compile | 0m 46s | hbase-server in the patch failed. |
| -0 :warning: | javac | 0m 46s | hbase-server in the patch failed. |
| -1 :x: | shadedjars | 3m 20s | patch has 16 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 12s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 1m 19s | hbase-client in the patch passed. |
| -1 :x: | unit | 14m 7s | hbase-balancer in the patch failed. |
| -1 :x: | unit | 0m 53s | hbase-server in the patch failed. |
| | | 42m 59s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4799 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 00777f250ee4 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 230fdc0b50 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt |
| javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt |
| shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-balancer.txt |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/testReport/ |
| Max. process+thread count | 198 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4799/10/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1106703131
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java:
##########
@@ -553,6 +558,61 @@ enum LocalityType {
RACK
}
+ public float getOrComputeWeightedPrefetchRatio(int region, int server) {
+ return getRegionSizeMB(region) * getOrComputeRegionPrefetchRatio()[region][server];
+ }
+
+ private float[][] getOrComputeRegionPrefetchRatio() {
+ if (regionServerWithBestPrefetchRatio == null || regionServerPrefetchRatio == null) {
+ computeRegionServerPrefetchRatio();
+ }
+ return regionServerPrefetchRatio;
+ }
+
+ public int[] getOrComputeServerWithBestPrefetchRatio() {
+ if (regionServerWithBestPrefetchRatio == null || regionServerPrefetchRatio == null) {
+ computeRegionServerPrefetchRatio();
+ }
+ return regionServerWithBestPrefetchRatio;
+ }
+
+ private void computeRegionServerPrefetchRatio() {
+ regionServerPrefetchRatio = new float[numRegions][numServers];
+ regionServerWithBestPrefetchRatio = new int[numRegions];
+
+ for (int region = 0; region < numRegions; region++) {
+ float bestPrefetchRatio = 0.0f;
+ int serverWithBestPrefetchRatio = 0;
+ for (int server = 0; server < numServers; server++) {
Review Comment:
We are iterating through the regions first instead of servers because, we also need to find out if the region was previously hosted on that region server. If we go the other way i.e. iterate through the servers and then iterate through the regions hosted on that server, we will have to iterate through all the servers again to find out if this region was previously hosted on any of these region servers. Hence, IMO, it's better to iterate through the regions before iterating through servers. This way, in the same iteration, we can find out whether is region is currently hosted on the given region server or if it was previously hosted on this region server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on PR #4799:
URL: https://github.com/apache/hbase/pull/4799#issuecomment-1430828692
An update on testing. In addition to adding a unit test which considers different combinations of prefetch ratio on the currently hosting region server as well as historical prefetch ratio, these changes were tested on a real cluster. However, it was only tested for happy condition viz. if the prefetch ratio on a region server was good, then the region was not moved back to the old region server even though it was hosted and prefetched on the older region server in the part. This test was done by disabling all other cost functions by setting their multiplier values to 0. Every cost function provides a parameter which can be set to 0 to disable that particular cost function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ragarkar commented on a diff in pull request #4799: HBASE-27389 Add cost function in balancer to consider the cost of bui…
Posted by "ragarkar (via GitHub)" <gi...@apache.org>.
ragarkar commented on code in PR #4799:
URL: https://github.com/apache/hbase/pull/4799#discussion_r1106700557
##########
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/PrefetchCacheCostFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Compute the cost of a potential cluster configuration based on the number of HFile's already
+ * cached in the bucket cache
+ */
+@InterfaceAudience.Private
+public class PrefetchCacheCostFunction extends CostFunction {
+ private static final String PREFETCH_CACHE_COST_KEY =
+ "hbase.master.balancer.stochastic.prefetchCacheCost";
+ private static final float DEFAULT_PREFETCH_COST = 500;
+
+ private String prefetchedFileListPath;
+ private double prefetchRatio;
+ private float bestPrefetchRatio;
+
+ PrefetchCacheCostFunction(Configuration conf) {
+ prefetchedFileListPath = conf.get(HConstants.PREFETCH_PERSISTENCE_PATH_KEY);
+ // Disable the prefetch cache cost function if the prefetched file list persistence is not
+ // enabled
+ this.setMultiplier(prefetchedFileListPath == null
+ ? 0.0f
+ : conf.getFloat(PREFETCH_CACHE_COST_KEY, DEFAULT_PREFETCH_COST));
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+ }
+
+ @Override
+ void prepare(BalancerClusterState cluster) {
+ super.prepare(cluster);
+ prefetchRatio = 0.0f;
+ bestPrefetchRatio = 0.0f;
+
+ for (int region = 0; region < cluster.numRegions; region++) {
+ prefetchRatio +=
+ cluster.getOrComputeWeightedPrefetchRatio(region, cluster.regionIndexToServerIndex[region]);
+ bestPrefetchRatio += cluster.getOrComputeWeightedPrefetchRatio(region,
+ cluster.getOrComputeServerWithBestPrefetchRatio()[region]);
+ }
+ prefetchRatio = bestPrefetchRatio == 0.0f ? 1.0f : prefetchRatio / bestPrefetchRatio;
Review Comment:
The prefetch ratio and the bestPrefetchRatio will be different in case the region was hosted previously on some other region server. For example, if the region was previously hosted and prefetched 100% on RS-A and later moved to RS-B and it was partially prefetched there, the best prefetch ratio will be different than the prefetch ratio on individual nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org