You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mihir6692 (via GitHub)" <gi...@apache.org> on 2024/02/10 04:11:59 UTC
[PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
mihir6692 opened a new pull request, #5675:
URL: https://github.com/apache/hbase/pull/5675
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "mihir6692 (via GitHub)" <gi...@apache.org>.
mihir6692 commented on code in PR #5675:
URL: https://github.com/apache/hbase/pull/5675#discussion_r1485211063
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java:
##########
@@ -510,19 +510,38 @@ public Void call() {
private Void readColumnFamily(Table table, ColumnFamilyDescriptor column) {
byte[] startKey = null;
- Get get = null;
Scan scan = null;
ResultScanner rs = null;
StopWatch stopWatch = new StopWatch();
startKey = region.getStartKey();
// Can't do a get on empty start row so do a Scan of first element if any instead.
if (startKey.length > 0) {
- get = new Get(startKey);
+ Get get = new Get(startKey);
get.setCacheBlocks(false);
get.setFilter(new FirstKeyOnlyFilter());
get.addFamily(column.getName());
+ // Converting get object to scan to enable RAW SCAN.
+ // This will work for all the regions of the HBase tables except first region of the table.
+ scan = new Scan(get);
+ scan.setRaw(rawScanEnabled);
} else {
scan = new Scan();
+ // In case of first region of the HBase Table, we do not have start-key for the region.
+ // For Region Canary, we only need scan a single row/cell in the region to make sure that
+ // region is accessible.
+ //
+ // When HBase table has more than 1 empty regions at start of the row-key space, Canary will
+ // create multiple scan object to find first available row in the table by scanning all the
+ // regions in sequence until it can find first available row.
+ //
+ // This could result in multiple millions of scans based on the size of table and number of
+ // empty regions in sequence. In test environment, A table no data and 1000 empty regions,
+ // Single canary run was creating close to half million to 1 million scans to successfully
+ // do canary run for the table.
+ //
+ // Since First region of the table doesn't have any start key, We should set End Key as
+ // stop row and set inclusive=false to limit scan to single region only.
Review Comment:
Added some more comments as well as TODO for future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1936863167
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 1s | master passed |
| +1 :green_heart: | compile | 2m 33s | master passed |
| +1 :green_heart: | checkstyle | 0m 38s | master passed |
| +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 33s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 44s | the patch passed |
| +1 :green_heart: | compile | 2m 27s | the patch passed |
| +1 :green_heart: | javac | 2m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 4m 49s | Patch does not cause any errors with Hadoop 3.3.6. |
| +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. |
| | | 28m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/5675 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux c623837cefdc 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b3ffc4acb6 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 82 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "mihir6692 (via GitHub)" <gi...@apache.org>.
mihir6692 commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1937110196
Updated Release note on the JIRA @virajjasani
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1937293892
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 46s | master passed |
| +1 :green_heart: | compile | 0m 38s | master passed |
| +1 :green_heart: | shadedjars | 5m 33s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 25s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 225m 51s | hbase-server in the patch passed. |
| | | 248m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/5675 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 3354719fc5f1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b3ffc4acb6 |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/testReport/ |
| Max. process+thread count | 5597 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1936960778
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 33s | master passed |
| +1 :green_heart: | compile | 1m 6s | master passed |
| +1 :green_heart: | shadedjars | 6m 42s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 54s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 315m 48s | hbase-server in the patch passed. |
| | | 346m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/5675 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 5f9debf74642 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b3ffc4acb6 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/testReport/ |
| Max. process+thread count | 4623 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #5675:
URL: https://github.com/apache/hbase/pull/5675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1936963686
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 57s | master passed |
| +1 :green_heart: | compile | 0m 50s | master passed |
| +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 20s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 17s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 330m 19s | hbase-server in the patch failed. |
| | | 358m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/5675 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 8262b6cf7abc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b3ffc4acb6 |
| Default Java | Temurin-1.8.0_352-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/testReport/ |
| Max. process+thread count | 4416 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #5675:
URL: https://github.com/apache/hbase/pull/5675#discussion_r1484947970
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java:
##########
@@ -510,19 +510,38 @@ public Void call() {
private Void readColumnFamily(Table table, ColumnFamilyDescriptor column) {
byte[] startKey = null;
- Get get = null;
Scan scan = null;
ResultScanner rs = null;
StopWatch stopWatch = new StopWatch();
startKey = region.getStartKey();
// Can't do a get on empty start row so do a Scan of first element if any instead.
if (startKey.length > 0) {
- get = new Get(startKey);
+ Get get = new Get(startKey);
get.setCacheBlocks(false);
get.setFilter(new FirstKeyOnlyFilter());
get.addFamily(column.getName());
+ // Converting get object to scan to enable RAW SCAN.
+ // This will work for all the regions of the HBase tables except first region of the table.
+ scan = new Scan(get);
+ scan.setRaw(rawScanEnabled);
} else {
scan = new Scan();
+ // In case of first region of the HBase Table, we do not have start-key for the region.
+ // For Region Canary, we only need scan a single row/cell in the region to make sure that
+ // region is accessible.
+ //
+ // When HBase table has more than 1 empty regions at start of the row-key space, Canary will
+ // create multiple scan object to find first available row in the table by scanning all the
+ // regions in sequence until it can find first available row.
+ //
+ // This could result in multiple millions of scans based on the size of table and number of
+ // empty regions in sequence. In test environment, A table no data and 1000 empty regions,
Review Comment:
nit: A table with no data
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java:
##########
@@ -510,19 +510,38 @@ public Void call() {
private Void readColumnFamily(Table table, ColumnFamilyDescriptor column) {
byte[] startKey = null;
- Get get = null;
Scan scan = null;
ResultScanner rs = null;
StopWatch stopWatch = new StopWatch();
startKey = region.getStartKey();
// Can't do a get on empty start row so do a Scan of first element if any instead.
if (startKey.length > 0) {
- get = new Get(startKey);
+ Get get = new Get(startKey);
get.setCacheBlocks(false);
get.setFilter(new FirstKeyOnlyFilter());
get.addFamily(column.getName());
+ // Converting get object to scan to enable RAW SCAN.
+ // This will work for all the regions of the HBase tables except first region of the table.
+ scan = new Scan(get);
+ scan.setRaw(rawScanEnabled);
} else {
scan = new Scan();
+ // In case of first region of the HBase Table, we do not have start-key for the region.
+ // For Region Canary, we only need scan a single row/cell in the region to make sure that
Review Comment:
nit: we only need to scan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1937093930
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 5s | master passed |
| +1 :green_heart: | compile | 2m 27s | master passed |
| +1 :green_heart: | checkstyle | 0m 32s | master passed |
| +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 26s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 42s | the patch passed |
| +1 :green_heart: | compile | 2m 23s | the patch passed |
| +1 :green_heart: | javac | 2m 23s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 34s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 5m 3s | Patch does not cause any errors with Hadoop 3.3.6. |
| +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 32s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. |
| | | 27m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/5675 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 85380ad2a503 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b3ffc4acb6 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 77 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1937304198
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 2s | master passed |
| +1 :green_heart: | compile | 0m 48s | master passed |
| +1 :green_heart: | shadedjars | 5m 9s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 51s | the patch passed |
| +1 :green_heart: | compile | 0m 48s | the patch passed |
| +1 :green_heart: | javac | 0m 48s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 229m 38s | hbase-server in the patch passed. |
| | | 253m 35s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/5675 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 4e4140ac47df 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b3ffc4acb6 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/testReport/ |
| Max. process+thread count | 5134 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "d-c-manning (via GitHub)" <gi...@apache.org>.
d-c-manning commented on code in PR #5675:
URL: https://github.com/apache/hbase/pull/5675#discussion_r1484986773
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java:
##########
@@ -510,19 +510,38 @@ public Void call() {
private Void readColumnFamily(Table table, ColumnFamilyDescriptor column) {
byte[] startKey = null;
- Get get = null;
Scan scan = null;
ResultScanner rs = null;
StopWatch stopWatch = new StopWatch();
startKey = region.getStartKey();
// Can't do a get on empty start row so do a Scan of first element if any instead.
if (startKey.length > 0) {
- get = new Get(startKey);
+ Get get = new Get(startKey);
get.setCacheBlocks(false);
get.setFilter(new FirstKeyOnlyFilter());
get.addFamily(column.getName());
+ // Converting get object to scan to enable RAW SCAN.
+ // This will work for all the regions of the HBase tables except first region of the table.
+ scan = new Scan(get);
+ scan.setRaw(rawScanEnabled);
} else {
scan = new Scan();
+ // In case of first region of the HBase Table, we do not have start-key for the region.
+ // For Region Canary, we only need scan a single row/cell in the region to make sure that
+ // region is accessible.
+ //
+ // When HBase table has more than 1 empty regions at start of the row-key space, Canary will
+ // create multiple scan object to find first available row in the table by scanning all the
+ // regions in sequence until it can find first available row.
+ //
+ // This could result in multiple millions of scans based on the size of table and number of
+ // empty regions in sequence. In test environment, A table no data and 1000 empty regions,
+ // Single canary run was creating close to half million to 1 million scans to successfully
+ // do canary run for the table.
+ //
+ // Since First region of the table doesn't have any start key, We should set End Key as
+ // stop row and set inclusive=false to limit scan to single region only.
Review Comment:
We could even add a comment that we think this would be the better approach for all regions (not only the first region of the table), but who knows what performance people may expect from the canary, and whether they are counting on most Get requests (now Scans) not returning any data, and therefore performing extra fast (?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "mihir6692 (via GitHub)" <gi...@apache.org>.
mihir6692 commented on code in PR #5675:
URL: https://github.com/apache/hbase/pull/5675#discussion_r1485211027
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java:
##########
@@ -510,19 +510,38 @@ public Void call() {
private Void readColumnFamily(Table table, ColumnFamilyDescriptor column) {
byte[] startKey = null;
- Get get = null;
Scan scan = null;
ResultScanner rs = null;
StopWatch stopWatch = new StopWatch();
startKey = region.getStartKey();
// Can't do a get on empty start row so do a Scan of first element if any instead.
if (startKey.length > 0) {
- get = new Get(startKey);
+ Get get = new Get(startKey);
get.setCacheBlocks(false);
get.setFilter(new FirstKeyOnlyFilter());
get.addFamily(column.getName());
+ // Converting get object to scan to enable RAW SCAN.
+ // This will work for all the regions of the HBase tables except first region of the table.
+ scan = new Scan(get);
+ scan.setRaw(rawScanEnabled);
} else {
scan = new Scan();
+ // In case of first region of the HBase Table, we do not have start-key for the region.
+ // For Region Canary, we only need scan a single row/cell in the region to make sure that
+ // region is accessible.
+ //
+ // When HBase table has more than 1 empty regions at start of the row-key space, Canary will
+ // create multiple scan object to find first available row in the table by scanning all the
+ // regions in sequence until it can find first available row.
+ //
+ // This could result in multiple millions of scans based on the size of table and number of
+ // empty regions in sequence. In test environment, A table no data and 1000 empty regions,
Review Comment:
Done
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java:
##########
@@ -510,19 +510,38 @@ public Void call() {
private Void readColumnFamily(Table table, ColumnFamilyDescriptor column) {
byte[] startKey = null;
- Get get = null;
Scan scan = null;
ResultScanner rs = null;
StopWatch stopWatch = new StopWatch();
startKey = region.getStartKey();
// Can't do a get on empty start row so do a Scan of first element if any instead.
if (startKey.length > 0) {
- get = new Get(startKey);
+ Get get = new Get(startKey);
get.setCacheBlocks(false);
get.setFilter(new FirstKeyOnlyFilter());
get.addFamily(column.getName());
+ // Converting get object to scan to enable RAW SCAN.
+ // This will work for all the regions of the HBase tables except first region of the table.
+ scan = new Scan(get);
+ scan.setRaw(rawScanEnabled);
} else {
scan = new Scan();
+ // In case of first region of the HBase Table, we do not have start-key for the region.
+ // For Region Canary, we only need scan a single row/cell in the region to make sure that
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "mihir6692 (via GitHub)" <gi...@apache.org>.
mihir6692 commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1937087080
> At least this change does not change any behavior of the Canary, other than avoiding unnecessary scans for empty tables/regions, and avoiding a Get of a very large deleted row, if it exists. Seems to be a strict improvement in that sense.
>
> If anyone was relying on the RPCs being Gets, by using separate RPC handlers for Gets vs. Scans, this will change that behavior. But that seems like weird behavior to rely on.
@d-c-manning 's suggestion is valid. Should we include this as JIRA release note? So it's documented for every branch that has this JIRA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5675:
URL: https://github.com/apache/hbase/pull/5675#issuecomment-1937093892
> > At least this change does not change any behavior of the Canary, other than avoiding unnecessary scans for empty tables/regions, and avoiding a Get of a very large deleted row, if it exists. Seems to be a strict improvement in that sense.
> > If anyone was relying on the RPCs being Gets, by using separate RPC handlers for Gets vs. Scans, this will change that behavior. But that seems like weird behavior to rely on.
>
> @d-c-manning 's suggestion is valid. Should we include this as JIRA release note? So it's documented for every branch that has this JIRA? @virajjasani
Sounds good, feel free to update release note on the Jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org