You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/02/06 01:48:49 UTC
[GitHub] [hbase] taklwu opened a new pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
taklwu opened a new pull request #2931:
URL: https://github.com/apache/hbase/pull/2931
Create a new wrapper store engine on top of the default store engine, as well as, introducing a new store file manager that keep tracking path of HFiles of user tables into a hbase self-managed system table (further will have a different tasks to merge into hbase:meta table)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585247597
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
##########
@@ -203,5 +213,21 @@ public double getCompactionPressure() {
public Comparator<HStoreFile> getStoreFileComparator() {
return storeFileComparator;
}
+
+ void setStorefiles(ImmutableList<HStoreFile> storefiles) {
Review comment:
refactor to use a abstract base class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-774812029
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 17s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 31s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 8s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 2m 6s | root in the patch failed. |
| -1 :x: | compile | 0m 39s | hbase-server in the patch failed. |
| -0 :warning: | javac | 0m 39s | hbase-server in the patch failed. |
| -1 :x: | shadedjars | 5m 1s | patch has 18 errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 24s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | hbase-common in the patch passed. |
| -1 :x: | unit | 0m 41s | hbase-server in the patch failed. |
| | | 27m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 048cd619fb5c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt |
| javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/testReport/ |
| Max. process+thread count | 264 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-797289324
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 4m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 55s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 10s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 37s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 51s | HBASE-24749 passed |
| -0 :warning: | patch | 2m 23s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | the patch passed |
| +1 :green_heart: | compile | 4m 3s | the patch passed |
| +1 :green_heart: | javac | 4m 3s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 25s | The patch passed checkstyle in hbase-common |
| +1 :green_heart: | checkstyle | 1m 10s | hbase-server: The patch generated 0 new + 107 unchanged - 2 fixed = 107 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 10s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| | | 57m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cee86fadbce9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-789441913
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 29s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 11s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 27s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 49s | HBASE-24749 passed |
| -0 :warning: | patch | 2m 19s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 38s | the patch passed |
| +1 :green_heart: | compile | 4m 1s | the patch passed |
| +1 :green_heart: | javac | 4m 1s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 26s | The patch passed checkstyle in hbase-common |
| +1 :green_heart: | checkstyle | 1m 4s | hbase-server: The patch generated 0 new + 107 unchanged - 2 fixed = 107 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 21m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 57m 28s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d4ad85180df4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-788628157
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 46s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 53s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 3s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 38s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 48s | HBASE-24749 passed |
| -0 :warning: | patch | 2m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 53s | the patch passed |
| +1 :green_heart: | compile | 4m 6s | the patch passed |
| +1 :green_heart: | javac | 4m 6s | the patch passed |
| -0 :warning: | checkstyle | 1m 13s | hbase-server: The patch generated 2 new + 107 unchanged - 2 fixed = 109 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 2s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| -1 :x: | spotbugs | 2m 19s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
| | | 56m 2s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hbase-server |
| | Format string should use %n rather than n in org.apache.hadoop.hbase.regionserver.PersistedStoreFileManager.updatePathListToTracker(StoreFilePathUpdate) At PersistedStoreFileManager.java:rather than n in org.apache.hadoop.hbase.regionserver.PersistedStoreFileManager.updatePathListToTracker(StoreFilePathUpdate) At PersistedStoreFileManager.java:[line 156] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 689c3f5a9e4f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-797427736
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 56s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 7s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 36s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 54s | HBASE-24749 passed |
| -0 :warning: | patch | 2m 26s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 48s | the patch passed |
| +1 :green_heart: | compile | 4m 6s | the patch passed |
| +1 :green_heart: | javac | 4m 6s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 24s | The patch passed checkstyle in hbase-common |
| +1 :green_heart: | checkstyle | 1m 11s | hbase-server: The patch generated 0 new + 107 unchanged - 2 fixed = 107 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| | | 54m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2ef22a36da35 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-797392952
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 55s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 5s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 21s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 7m 13s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 31s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 54s | the patch passed |
| +1 :green_heart: | compile | 1m 20s | the patch passed |
| +1 :green_heart: | javac | 1m 20s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 8s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 208m 44s | hbase-server in the patch passed. |
| | | 244m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 780c2f556e5c 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/testReport/ |
| Max. process+thread count | 3329 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-797669061
thanks @anmolnar , @joshelser do you have any other comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585151505
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @param storeFilePathUpdate Updates to be persisted
+ * @throws IOException if a remote or network exception occurs during write
+ */
+ void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) throws IOException;
+
+ /**
+ * Delete storefile paths for a tracking column family, normally used when a region-store is
+ * completely removed due to region split or merge
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @throws IOException if a remote or network exception occurs during delete
+ */
+ void deleteStoreFilePaths(final String tableName, final String regionName, final String storeName)
+ throws IOException;
+
+ /**
+ * Returns the families being tracked in the storefile tracking data for the given
+ * table/region
Review comment:
mmm, we want all stores of a region server to be tracked, maybe just this comment is confusing , I will update it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-774384451
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 17s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 6s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 4s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 38s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 52s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 51s | the patch passed |
| +1 :green_heart: | compile | 4m 6s | the patch passed |
| +1 :green_heart: | javac | 4m 6s | the patch passed |
| -0 :warning: | checkstyle | 1m 14s | hbase-server: The patch generated 22 new + 109 unchanged - 0 fixed = 131 total (was 109) |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 18m 59s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 10s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| | | 54m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0f9d9cb6d522 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu merged pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu merged pull request #2931:
URL: https://github.com/apache/hbase/pull/2931
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-775395190
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 3s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 18s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 37s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 57s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 56s | the patch passed |
| +1 :green_heart: | compile | 4m 4s | the patch passed |
| +1 :green_heart: | javac | 4m 4s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 24s | The patch passed checkstyle in hbase-common |
| +1 :green_heart: | checkstyle | 1m 13s | hbase-server: The patch generated 0 new + 107 unchanged - 2 fixed = 107 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 10s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
| | | 56m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3e6564be7caa 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-790188641
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 2s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 30s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 48s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 17s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 2s | the patch passed |
| +1 :green_heart: | compile | 1m 30s | the patch passed |
| +1 :green_heart: | javac | 1m 30s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | hbase-common in the patch passed. |
| -1 :x: | unit | 135m 36s | hbase-server in the patch failed. |
| | | 167m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 896a135986ed 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/testReport/ |
| Max. process+thread count | 4328 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585247400
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessorTestBase.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.hamcrest.Matchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TestName;
+
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+public abstract class StoreFilePathAccessorTestBase {
+
+ @Rule
+ public TestName name = new TestName();
+
+ @Rule
+ public ExpectedException expectedException = ExpectedException.none();
+
+ protected static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
+ protected StoreFilePathAccessor storeFilePathAccessor;
+ protected static String REGION_NAME = UUID.randomUUID().toString().replaceAll("-", "");
+ protected static String STORE_NAME = UUID.randomUUID().toString();
+ protected static List<Path> EMPTY_PATH = Collections.emptyList();
+ protected static List<Path> INCLUDE_EXAMPLE_PATH =
+ Lists.newArrayList(new Path("hdfs://foo/bar1"), new Path("hdfs://foo/bar2"));
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anmolnar commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
anmolnar commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-796780015
@taklwu I'm checking your commit which introduced the abstract class `AbstractStoreFileManager`. You introduced 2 things here: an abstract base class and hooks. Inheritence graph looks like AbstractStoreFileManager -> DefaultStoreFileManager -> PersistedStoreFileManager. On the top of that you essentially moved all code from the Default impl to the Abstract.
Revisiting my idea about the abstract class I think we don't need that. Let's revert to your original DefaultStoreFileManager -> PersistedStoreFileManager inheritence with the following modifications:
- PersistedStoreFileManager needs to override method loadFiles(), call the super at the beginning and do what you have in the hook now. No need to hook here.
- Similarly in insertNewFilesHook() the extra logic is at the end, so instead of the hook, just override -> call super() -> do the rest.
- addCompactionResultsHook() - This is where the hook magic comes in handy, because the extra logic happens in the middle. Nice as it is now.
- loadInitialFiles() - This method has to be completely overriden by PersistedStoreFileManager, no shared logic here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-789497356
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 23s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 38s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 57s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 9s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 27s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 11s | the patch passed |
| +1 :green_heart: | compile | 1m 34s | the patch passed |
| +1 :green_heart: | javac | 1m 34s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 46s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 36s | hbase-common in the patch passed. |
| -1 :x: | unit | 136m 54s | hbase-server in the patch failed. |
| | | 169m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 3a361f1a2948 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/testReport/ |
| Max. process+thread count | 3997 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-774812273
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 48s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 5s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 36s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 48s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 56s | root in the patch failed. |
| -1 :x: | compile | 0m 41s | hbase-server in the patch failed. |
| -0 :warning: | javac | 0m 41s | hbase-server in the patch failed. |
| +1 :green_heart: | checkstyle | 0m 23s | The patch passed checkstyle in hbase-common |
| +1 :green_heart: | checkstyle | 1m 11s | hbase-server: The patch generated 0 new + 107 unchanged - 2 fixed = 107 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | hadoopcheck | 2m 2s | The patch causes 22 errors with Hadoop v3.1.2. |
| -1 :x: | hadoopcheck | 4m 9s | The patch causes 22 errors with Hadoop v3.2.1. |
| -1 :x: | hadoopcheck | 6m 9s | The patch causes 22 errors with Hadoop v3.3.0. |
| -1 :x: | spotbugs | 0m 36s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
| | | 28m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 157e14233ae1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/patch-compile-hbase-server.txt |
| javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/patch-compile-hbase-server.txt |
| hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/patch-javac-3.1.2.txt |
| hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/patch-javac-3.2.1.txt |
| hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/patch-javac-3.3.0.txt |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r586079300
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractStoreFilePathAccessor.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Joiner;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+
+@InterfaceAudience.Private
+public abstract class AbstractStoreFilePathAccessor implements StoreFilePathAccessor {
+
+ public static final String STOREFILE_INCLUDED_STR = "included";
+
+ protected static final String LIST_SEPARATOR = ";";
+ protected final Configuration conf;
+
+ public AbstractStoreFilePathAccessor(Configuration conf) {
+ this.conf = conf;
+ }
+
+ abstract String getSeparator();
+
+ abstract List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String columnName) throws IOException;
+
+ @Override
+ public abstract void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException;
+
+ @Override
+ public List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ return getStoreFilePaths(tableName, regionName, storeName, STOREFILE_INCLUDED_STR);
+ }
+
+ protected static byte[] storeFileListToByteArray(List<Path> storeFilePaths) {
+ return Bytes.toBytes(Joiner.on(LIST_SEPARATOR).join(storeFilePaths));
+ }
+
+ protected static List<Path> byteToStoreFileList(byte[] data) {
+ List<Path> paths = new ArrayList<>();
+ if (data != null && data.length != 0) {
+ String pathString = Bytes.toString(data);
+ String[] pathStrings = pathString.split(LIST_SEPARATOR);
Review comment:
fixed and used `StringUtils.split(pathString, LIST_SEPARATOR)`
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathUpdate.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.List;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
Review comment:
I don't think so, and I confirmed by searching the code base and the shading setting.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.filter.RegexStringComparator;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
+
+@InterfaceAudience.Private
+public class HTableStoreFilePathAccessor extends AbstractStoreFilePathAccessor {
+
+ public static final byte[] STOREFILE_FAMILY_INCLUDED = Bytes.toBytes(STOREFILE_INCLUDED_STR);
+
+ private static final String DASH_SEPARATOR = "-";
+ private static final String STOREFILE_QUALIFIER_STR = "filepaths";
+ private static final byte[] STOREFILE_QUALIFIER = Bytes.toBytes(STOREFILE_QUALIFIER_STR);
+ private static final int STOREFILE_TABLE_VERSIONS = 3;
+
+ // TODO find a way for system table to support region split at table creation or remove this
+ // comment when we merge into hbase:meta table
+ public static final TableDescriptor STOREFILE_TABLE_DESC =
+ TableDescriptorBuilder.newBuilder(TableName.STOREFILE_TABLE_NAME)
+ .setColumnFamily(
+ ColumnFamilyDescriptorBuilder.newBuilder(STOREFILE_FAMILY_INCLUDED)
+ .setMaxVersions(STOREFILE_TABLE_VERSIONS)
+ .setInMemory(true)
+ .build())
+ .setRegionSplitPolicyClassName(BusyRegionSplitPolicy.class.getName())
+ .build();
+
+ private Connection connection;
+
+ public HTableStoreFilePathAccessor(Configuration conf, Connection connection) {
+ super(conf);
+ Preconditions.checkNotNull(connection, "connection cannot be null");
+ this.connection = connection;
+ }
+
+ @Override
+ public void initialize(final MasterServices masterServices) throws IOException {
+ StorefileTrackingUtils.init(masterServices);
+ }
+
+ @Override
+ List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String colFamily) throws IOException {
+ validate(tableName, regionName, storeName, colFamily);
+ byte[] colFamilyBytes = Bytes.toBytes(colFamily);
+ Get get =
+ new Get(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ get.addColumn(colFamilyBytes, STOREFILE_QUALIFIER);
+ Result result = doGet(get);
+ if (result == null || result.isEmpty()) {
+ return new ArrayList<>();
+ }
+ return byteToStoreFileList(result.getValue(colFamilyBytes, STOREFILE_QUALIFIER));
+ }
+
+ @Override
+ public void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException {
+ validate(tableName, regionName, storeName, storeFilePathUpdate);
+ Put put = generatePutForStoreFilePaths(tableName, regionName, storeName, storeFilePathUpdate);
+ doPut(put);
+ }
+
+
+ private Put generatePutForStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) {
+ Put put = new Put(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ if (storeFilePathUpdate.hasStoreFilesUpdate()) {
+ put.addColumn(Bytes.toBytes(STOREFILE_INCLUDED_STR), STOREFILE_QUALIFIER,
+ storeFileListToByteArray(storeFilePathUpdate.getStoreFiles()));
+ }
+ return put;
+ }
+
+ @Override
+ public void deleteStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ validate(tableName, regionName, storeName);
+ Delete delete = new Delete(
+ Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ delete.addColumns(STOREFILE_FAMILY_INCLUDED, STOREFILE_QUALIFIER);
+ doDelete(Lists.newArrayList(delete));
+ }
+
+ @Override
+ public void deleteRegion(String regionName) throws IOException {
+ Scan scan = getScanWithFilter(regionName);
+ List<Delete> familiesToDelete = new ArrayList<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ Delete delete = new Delete(Bytes.toBytes(rowKey));
+ familiesToDelete.add(delete);
+ }
+ doDelete(familiesToDelete);
+ }
+
+ @Override
+ public Set<String> getTrackedFamilies(String tableName, String regionName)
+ throws IOException {
+ // find all rows by regionName
+ Scan scan = getScanWithFilter(regionName);
+
+ Set<String> families = new HashSet<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ String family =
+ StorefileTrackingUtils.getFamilyFromKey(rowKey, tableName, regionName, getSeparator());
+ families.add(family);
+ }
+ return families;
+ }
+
+ @Override
+ String getSeparator() {
+ return DASH_SEPARATOR;
+ }
+
+ private Result doGet(final Get get) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ return table.get(get);
+ }
+ }
+
+ private void doPut(final Put put) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ table.put(put);
+ }
+ }
+
+ private void doDelete(final List<Delete> delete) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ table.delete(delete);
+ }
+ }
+
+ private ResultScanner getResultScanner(final Scan scan) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ return table.getScanner(scan);
+ }
+ }
+
+ private Connection getConnection() throws IOException {
+ if (connection == null) {
+ throw new IOException("Connection should be provided by region server "
+ + "and should not be null after initialized.");
+ }
Review comment:
fixed and removed this `getConnection()` function after making `connection` final
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.filter.RegexStringComparator;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
+
+@InterfaceAudience.Private
+public class HTableStoreFilePathAccessor extends AbstractStoreFilePathAccessor {
+
+ public static final byte[] STOREFILE_FAMILY_INCLUDED = Bytes.toBytes(STOREFILE_INCLUDED_STR);
+
+ private static final String DASH_SEPARATOR = "-";
+ private static final String STOREFILE_QUALIFIER_STR = "filepaths";
+ private static final byte[] STOREFILE_QUALIFIER = Bytes.toBytes(STOREFILE_QUALIFIER_STR);
+ private static final int STOREFILE_TABLE_VERSIONS = 3;
+
+ // TODO find a way for system table to support region split at table creation or remove this
+ // comment when we merge into hbase:meta table
+ public static final TableDescriptor STOREFILE_TABLE_DESC =
+ TableDescriptorBuilder.newBuilder(TableName.STOREFILE_TABLE_NAME)
+ .setColumnFamily(
+ ColumnFamilyDescriptorBuilder.newBuilder(STOREFILE_FAMILY_INCLUDED)
+ .setMaxVersions(STOREFILE_TABLE_VERSIONS)
+ .setInMemory(true)
+ .build())
+ .setRegionSplitPolicyClassName(BusyRegionSplitPolicy.class.getName())
+ .build();
+
+ private Connection connection;
+
+ public HTableStoreFilePathAccessor(Configuration conf, Connection connection) {
+ super(conf);
+ Preconditions.checkNotNull(connection, "connection cannot be null");
+ this.connection = connection;
+ }
+
+ @Override
+ public void initialize(final MasterServices masterServices) throws IOException {
+ StorefileTrackingUtils.init(masterServices);
+ }
+
+ @Override
+ List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String colFamily) throws IOException {
+ validate(tableName, regionName, storeName, colFamily);
+ byte[] colFamilyBytes = Bytes.toBytes(colFamily);
+ Get get =
+ new Get(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ get.addColumn(colFamilyBytes, STOREFILE_QUALIFIER);
+ Result result = doGet(get);
+ if (result == null || result.isEmpty()) {
Review comment:
from the API java doc it should be non-null and `result.isEmpty()` should be the indicator for no result for a given `GET`, I will remove the null check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r577899781
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractStoreFilePathAccessor.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Joiner;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+
+@InterfaceAudience.Private
+public abstract class AbstractStoreFilePathAccessor implements StoreFilePathAccessor {
+
+ public static final String STOREFILE_INCLUDED_STR = "included";
+
+ protected static final String LIST_SEPARATOR = ";";
+ protected final Configuration conf;
+
+ public AbstractStoreFilePathAccessor(Configuration conf) {
+ this.conf = conf;
+ }
+
+ abstract String getSeparator();
+
+ abstract List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String columnName) throws IOException;
+
+ @Override
+ public abstract void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException;
+
+ @Override
+ public List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ return getStoreFilePaths(tableName, regionName, storeName, STOREFILE_INCLUDED_STR);
+ }
+
+ protected static byte[] storeFileListToByteArray(List<Path> storeFilePaths) {
+ return Bytes.toBytes(Joiner.on(LIST_SEPARATOR).join(storeFilePaths));
+ }
+
+ protected static List<Path> byteToStoreFileList(byte[] data) {
+ List<Path> paths = new ArrayList<>();
+ if (data != null && data.length != 0) {
+ String pathString = Bytes.toString(data);
+ String[] pathStrings = pathString.split(LIST_SEPARATOR);
Review comment:
http://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringUtils.html#split-java.lang.String-char- would be what I'd recommend to use (StringTokenizer is considered "legacy" looking at its javadoc)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anmolnar commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r577021789
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
##########
@@ -203,5 +213,21 @@ public double getCompactionPressure() {
public Comparator<HStoreFile> getStoreFileComparator() {
return storeFileComparator;
}
+
+ void setStorefiles(ImmutableList<HStoreFile> storefiles) {
+ this.storefiles = storefiles;
+ }
+
+ void setCompactedfiles(ImmutableList<HStoreFile> compactedfiles) {
Review comment:
Hm. My previous review has been lost in Github.
I would rather create an abstract base class for DefaultStoreFileManager and PersistedStoreFileManager and put everything which is common in there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-799918388
fixed the commit from [fd649ce544553a7b2a2cfe9df0ea8a9bc49fa571](https://github.com/apache/hbase/commit/fd649ce544553a7b2a2cfe9df0ea8a9bc49fa571) to [7b58ada2ff3e06b2a4550d69aa121e0c1c8f3c67](https://github.com/apache/hbase/commit/7b58ada2ff3e06b2a4550d69aa121e0c1c8f3c67) with adding `Reviewed-by:` and `Signed-off-by:` to branch HBASE-24749
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-775481982
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 4m 49s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 18s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 37s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 59s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 36s | the patch passed |
| +1 :green_heart: | javac | 1m 36s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 23s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 8s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 137m 38s | hbase-server in the patch passed. |
| | | 175m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux bb1399090ab8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/testReport/ |
| Max. process+thread count | 4503 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-799913582
> Minor comments, approved.
Hey Zach @z-york since your comment are minor and I will implement them in the upcoming JIRA, I will push the commit AS IS to save another around of CI build
In addition, @joshelser I didn't hear back from you so I think the reviewed by @anmolnar should be good enough, and if you want to come back and ask more change, we can have it in a addendum PR/JIRA. I'm pushing this to the feature branch HBASE-24749 and continues on other PRs this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585188596
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPersistedStoreEngine.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.hamcrest.core.Is.isA;
+import static org.junit.Assert.assertEquals;
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.CellComparatorImpl;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TestName;
+import org.mockito.Mockito;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestPersistedStoreEngine {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestPersistedStoreEngine.class);
+
+ @Rule
+ public TestName name = new TestName();
+
+ @Rule
+ public ExpectedException expectedException = ExpectedException.none();
+
+ private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
+ private static final byte[] DEFAULT_STORE_BYTE = TEST_UTIL.fam1;
+
+ private TableName tableName;
+ private Configuration conf;
+ private HStore store;
+
+ @BeforeClass
+ public static void setUpCluster() throws Exception {
+ TEST_UTIL.getConfiguration().setBoolean(HConstants.STOREFILE_TRACKING_PERSIST_ENABLED, true);
+ TEST_UTIL.getConfiguration().set(StoreEngine.STORE_ENGINE_CLASS_KEY,
+ PersistedStoreEngine.class.getName());
+ TEST_UTIL.startMiniCluster();
+ }
+
+ @Before
+ public void setup() throws Exception {
+ store = Mockito.mock(HStore.class);
+ StoreContext context = new StoreContext.Builder().build();
+
+ conf = TEST_UTIL.getConfiguration();
+ Mockito.when(store.getStoreContext()).thenReturn(context);
+ Mockito.when(store.getRegionInfo()).thenReturn(RegionInfoBuilder.FIRST_META_REGIONINFO);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ for (TableDescriptor htd: TEST_UTIL.getAdmin().listTableDescriptors()) {
+ TEST_UTIL.deleteTable(htd.getTableName());
+ }
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+ // use killMiniHBaseCluster before shutdownMiniCluster because shutdownMiniCluster puts a RS
+ // that host hbase:meta to a stopped state that prevents meta lookup the table state of
+ // hbase:storefile and hangs the last write to hbase:storefile when any tracking region closes.
+ //
+ // this should be rarely happened on a real cluster and RS with meta table should be the last
+ // to be shutdown normally.
+ TEST_UTIL.killMiniHBaseCluster();
Review comment:
sorry, we can remove this now.
it was something old from my local branch that tried to write one last record for compactedFiles to `hbase:storefile` table, but we have simplified the logic in this commit that only storefiles are being tracked. The test here has already flushed and should not have any new update to the table `hbase:storefile` when closing.
if we merge into hbase:meta (will be done in HBASE-25397) , this will definitely not a issue anymore as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-774811671
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 44s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 20s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 46s | root in the patch failed. |
| -1 :x: | compile | 0m 37s | hbase-server in the patch failed. |
| -0 :warning: | javac | 0m 37s | hbase-server in the patch failed. |
| -1 :x: | shadedjars | 4m 50s | patch has 22 errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 37s | hbase-server generated 1 new + 16 unchanged - 0 fixed = 17 total (was 16) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hbase-common in the patch passed. |
| -1 :x: | unit | 0m 37s | hbase-server in the patch failed. |
| | | 25m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a0f770152949 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt |
| javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] z-york commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
z-york commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r593400626
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
##########
@@ -1646,6 +1647,18 @@
*/
public static final int BATCH_ROWS_THRESHOLD_DEFAULT = 5000;
+ /**
+ * Configuration for storefile tracking feature
+ */
+ public static final String STOREFILE_TRACKING_PERSIST_ENABLED =
+ "hbase.storefile.tracking.persist.enabled";
+ public static final boolean DEFAULT_STOREFILE_TRACKING_PERSIST_ENABLED = false;
+
+ public static final String STOREFILE_TRACKING_INIT_TIMEOUT =
+ "hbase.storefile.tracking.init.timeout";
+ public static final long DEFAULT_STOREFILE_TRACKING_INIT_TIMEOUT =
+ TimeUnit.MINUTES.toMillis(5);
+
Review comment:
I think this is a bit of an anti-pattern now to put constants in HConstants, but not a major issue IMO.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
Review comment:
The in-memory tmp tracking won't touch the Accessor as nothing will be persisted. I think you can remove this.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private final StoreFilePathAccessor accessor;
+ private final Configuration conf;
+ // only uses for warmupHRegion
+ private final boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ // update with a sorted store files
+ super.loadFiles(storeFiles);
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(getStorefiles()).build());
+ }
+
+ @Override
+ public void insertNewFiles(Collection<HStoreFile> sfs) throws IOException {
+ // concatenate the new store files
+ super.insertNewFiles(sfs);
+ // return in case of empty store files as it is a No-op, here empty files are expected
+ // during region close
+ if (CollectionUtils.isEmpty(getStorefiles())) {
+ return;
+ }
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(getStorefiles()).build());
+ }
+
+ @Override
+ protected void addCompactionResultsHook(ImmutableList<HStoreFile> storeFiles)
+ throws IOException {
+ Preconditions.checkNotNull(storeFiles, "storeFiles cannot be null");
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storeFiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ // this logic is totally different from the default implementation in DefaultStoreFileManager
+
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
+ if (isEmptyInPersistedFilePaths) {
+ // When the path accessor is returning empty result, we scan the
+ // the file storage and see if there is any existing HFiles should be loaded.
+ // the scan is a one time process when store open during region assignment.
+ //
+ // this is especially used for region and store open
+ // 1. First time migration from a filesystem based e.g. DefaultStoreFileEngine
+ // 2. After region split and merge
+ // 3. After table clone and create new HFiles directly into data directory
+ //
+ // Also we don't handle the inconsistency between storefile tracking and file system, which
+ // will be handled by a HBCK command
Review comment:
JIRA?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private final StoreFilePathAccessor accessor;
+ private final Configuration conf;
+ // only uses for warmupHRegion
+ private final boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ // update with a sorted store files
+ super.loadFiles(storeFiles);
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(getStorefiles()).build());
+ }
+
+ @Override
+ public void insertNewFiles(Collection<HStoreFile> sfs) throws IOException {
+ // concatenate the new store files
+ super.insertNewFiles(sfs);
+ // return in case of empty store files as it is a No-op, here empty files are expected
+ // during region close
+ if (CollectionUtils.isEmpty(getStorefiles())) {
+ return;
+ }
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(getStorefiles()).build());
+ }
+
+ @Override
+ protected void addCompactionResultsHook(ImmutableList<HStoreFile> storeFiles)
+ throws IOException {
+ Preconditions.checkNotNull(storeFiles, "storeFiles cannot be null");
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storeFiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ // this logic is totally different from the default implementation in DefaultStoreFileManager
+
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
+ if (isEmptyInPersistedFilePaths) {
+ // When the path accessor is returning empty result, we scan the
+ // the file storage and see if there is any existing HFiles should be loaded.
+ // the scan is a one time process when store open during region assignment.
+ //
+ // this is especially used for region and store open
+ // 1. First time migration from a filesystem based e.g. DefaultStoreFileEngine
+ // 2. After region split and merge
+ // 3. After table clone and create new HFiles directly into data directory
+ //
+ // Also we don't handle the inconsistency between storefile tracking and file system, which
+ // will be handled by a HBCK command
+ LOG.info("Cannot find tracking paths ({}) for store {} in region {} of "
+ + "table {}, fall back to scan the storage to get a list of storefiles to be opened"
+ , isEmptyInPersistedFilePaths, storeName, regionName,
+ tableName);
+ return getRegionFs().getStoreFiles(getFamilyName());
+ }
+ ArrayList<StoreFileInfo> storeFiles = new ArrayList<>();
+ for (Path storeFilePath : pathList) {
+ if (!StoreFileInfo.isValid(getRegionFs().getFileSystem().getFileStatus(storeFilePath))) {
+ LOG.warn("Invalid StoreFile: {}, and archiving it", storeFilePath);
Review comment:
nit: remove 'and'
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
+
+@InterfaceAudience.Private
+public class HTableStoreFilePathAccessor extends AbstractStoreFilePathAccessor {
+
+ public static final byte[] STOREFILE_FAMILY_INCLUDED = Bytes.toBytes(STOREFILE_INCLUDED_STR);
+
+ private static final String DASH_SEPARATOR = "-";
+ private static final String STOREFILE_QUALIFIER_STR = "filepaths";
+ private static final byte[] STOREFILE_QUALIFIER = Bytes.toBytes(STOREFILE_QUALIFIER_STR);
+ private static final int STOREFILE_TABLE_VERSIONS = 3;
+
+ // TODO find a way for system table to support region split at table creation or remove this
Review comment:
link to a JIRA?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r577756352
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractStoreFilePathAccessor.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Joiner;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+
+@InterfaceAudience.Private
+public abstract class AbstractStoreFilePathAccessor implements StoreFilePathAccessor {
+
+ public static final String STOREFILE_INCLUDED_STR = "included";
+
+ protected static final String LIST_SEPARATOR = ";";
+ protected final Configuration conf;
+
+ public AbstractStoreFilePathAccessor(Configuration conf) {
+ this.conf = conf;
+ }
+
+ abstract String getSeparator();
+
+ abstract List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String columnName) throws IOException;
+
+ @Override
+ public abstract void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException;
+
+ @Override
+ public List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ return getStoreFilePaths(tableName, regionName, storeName, STOREFILE_INCLUDED_STR);
+ }
+
+ protected static byte[] storeFileListToByteArray(List<Path> storeFilePaths) {
+ return Bytes.toBytes(Joiner.on(LIST_SEPARATOR).join(storeFilePaths));
+ }
+
+ protected static List<Path> byteToStoreFileList(byte[] data) {
+ List<Path> paths = new ArrayList<>();
+ if (data != null && data.length != 0) {
+ String pathString = Bytes.toString(data);
+ String[] pathStrings = pathString.split(LIST_SEPARATOR);
Review comment:
Regex based splits were (if memory serves) at least 10x slower than a tokenizer-based string splitting. We would need to be super-aware of that as this is down in the critical path for serving data out of HBase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r574909689
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StorefileTrackingUtils.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to support persistent storefile tracking
+ */
+@InterfaceAudience.Private
+public final class StorefileTrackingUtils {
+
+ private static Logger LOG = LoggerFactory.getLogger(StorefileTrackingUtils.class);
+ public static final long SLEEP_DELTA_MS = TimeUnit.MILLISECONDS.toMillis(100);
+
+ private StorefileTrackingUtils() {
+ // private for utility class
+ }
+
+ public static boolean isStorefileTrackingPersistEnabled(Configuration conf) {
+ boolean isStoreTrackingPersistEnabled =
+ conf.getBoolean(HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ HConstants.DEFAULT_STOREFILE_TRACKING_PERSIST_ENABLED);
+ boolean isPersistedStoreEngineSet =
+ conf.get(StoreEngine.STORE_ENGINE_CLASS_KEY, DefaultStoreEngine.class.getName())
+ .equals(PersistedStoreEngine.class.getName());
+ boolean isFeatureEnabled = isStoreTrackingPersistEnabled && isPersistedStoreEngineSet;
+ if (isStoreTrackingPersistEnabled ^ isPersistedStoreEngineSet) {
+ // check if both configuration are correct.
+ String errorMessage = String.format("please set %s to true and set store engine key %s to %s "
+ + "to enable persist storefile tracking",
+ HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ StoreEngine.STORE_ENGINE_CLASS_KEY, PersistedStoreEngine.class.getName());
+ throw new IllegalArgumentException(errorMessage);
+ }
+ return isFeatureEnabled;
+ }
+
+ /**
+ * if storefile tracking feature is configured, Initialize hbase:storefile table and wait for it
+ * to be online. Otherwise, look for hbase:storefile table and remove it
+ *
+ * @param masterServices masterServices
+ * @throws IOException if hbase:storefile table cannot be initialized and be online
+ */
+ public static void init(MasterServices masterServices) throws IOException {
+ createStorefileTable(masterServices);
+ waitForStoreFileTableOnline(masterServices);
+ }
+
+ /**
+ * Cleans up all storefile related state on the cluster. disable and delete hbase:storefile
+ * if found
+ * @param masterServices {@link MasterServices}
+ * @throws IOException if failures
+ */
+ private static void cleanup(MasterServices masterServices) throws IOException {
+ try {
+ masterServices.getConnection().getAdmin().disableTable(TableName.STOREFILE_TABLE_NAME);
+ masterServices.getConnection().getAdmin().deleteTable(TableName.STOREFILE_TABLE_NAME);
+ } catch (IOException ex) {
+ final String message = "Failed disable and deleting table " +
+ TableName.STOREFILE_TABLE_NAME.getNameAsString();
+ LOG.error(message);
+ throw new IOException(message, ex);
+ }
+ }
+
+ public static StoreFilePathAccessor createStoreFilePathAccessor(Configuration conf,
+ Connection connection) {
+ return new HTableStoreFilePathAccessor(conf, connection);
+ }
+
+ public static List<Path> convertStoreFilesToPaths(Collection<HStoreFile> storeFiles) {
+ return storeFiles.stream().map(HStoreFile::getPath).collect(Collectors.toList());
+ }
+
+ private static void createStorefileTable(MasterServices masterServices)
+ throws IOException {
+ if (MetaTableAccessor.getTableState(masterServices.getConnection(),
+ TableName.STOREFILE_TABLE_NAME) == null) {
+ LOG.info("{} table not found. Creating...", TableName.STOREFILE_TABLE_NAME);
+ masterServices.createSystemTable(HTableStoreFilePathAccessor.STOREFILE_TABLE_DESC);
+ }
+ }
+
+ private static void waitForStoreFileTableOnline(MasterServices masterServices)
+ throws IOException {
+ try {
+ long startTime = EnvironmentEdgeManager.currentTime();
+ long timeout = masterServices.getConfiguration()
+ .getLong(HConstants.STOREFILE_TRACKING_INIT_TIMEOUT,
+ HConstants.DEFAULT_STOREFILE_TRACKING_INIT_TIMEOUT);
+ while (!isStoreFileTableAssignedAndEnabled(masterServices)) {
Review comment:
agreed. this is an optional feature but if this is enabled, the order will be master-region > hbase:meta -> hbase:storefile > any other tables, so the best way is to merge into hbase:meta (will be done in [HBASE-25397](https://issues.apache.org/jira/browse/HBASE-25397) ).
the only blocker/concern is if hbase:meta will be splittable by the time we hit that task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585151505
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @param storeFilePathUpdate Updates to be persisted
+ * @throws IOException if a remote or network exception occurs during write
+ */
+ void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) throws IOException;
+
+ /**
+ * Delete storefile paths for a tracking column family, normally used when a region-store is
+ * completely removed due to region split or merge
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @throws IOException if a remote or network exception occurs during delete
+ */
+ void deleteStoreFilePaths(final String tableName, final String regionName, final String storeName)
+ throws IOException;
+
+ /**
+ * Returns the families being tracked in the storefile tracking data for the given
+ * table/region
Review comment:
mmm, we want all stores of a region server to be tracked e.g. when supporting snapshot and restoring region from the tracking data, [we need to get it per family](https://github.com/apache/hbase/blob/01876071b990063899e0839cdda9745783fe814c/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java#L504) , maybe just this comment is confusing , I will update it and said this will be used by snapshot feature.
```
for (Path familyDir: FSUtils.getFamilyDirs(fs, regionDir)) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585167999
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
##########
@@ -203,5 +213,21 @@ public double getCompactionPressure() {
public Comparator<HStoreFile> getStoreFileComparator() {
return storeFileComparator;
}
+
+ void setStorefiles(ImmutableList<HStoreFile> storefiles) {
+ this.storefiles = storefiles;
+ }
+
+ void setCompactedfiles(ImmutableList<HStoreFile> compactedfiles) {
Review comment:
abstract base class is good idea, I will give a try as a last thing (another commit at the end) as that will take longer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-789415912
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 13s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 38s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 50s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 18s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 3s | the patch passed |
| +1 :green_heart: | compile | 1m 34s | the patch passed |
| +1 :green_heart: | javac | 1m 34s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 136m 7s | hbase-server in the patch passed. |
| | | 168m 20s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 1e4e1ebbab82 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/testReport/ |
| Max. process+thread count | 4426 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-775649940
@joshelser this patch is ready for review, and if you have any person in your mind could review it, please tag them as well, thanks a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu edited a comment on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu edited a comment on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-799918388
fixed the commit from fd649ce544553a7b2a2cfe9df0ea8a9bc49fa571 to cca4cbf173f4320dcb9262e9a97deba27f7de15e with adding `Reviewed-by:` and `Signed-off-by:` to branch HBASE-24749 , and rebased on master 5457554826519232c7d81ab382a5f345da0c0da2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anmolnar commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r574754171
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractStoreFilePathAccessor.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Joiner;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+
+@InterfaceAudience.Private
+public abstract class AbstractStoreFilePathAccessor implements StoreFilePathAccessor {
+
+ public static final String STOREFILE_INCLUDED_STR = "included";
+
+ protected static final String LIST_SEPARATOR = ";";
+ protected final Configuration conf;
+
+ public AbstractStoreFilePathAccessor(Configuration conf) {
+ this.conf = conf;
+ }
+
+ abstract String getSeparator();
+
+ abstract List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String columnName) throws IOException;
+
+ @Override
+ public abstract void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException;
+
+ @Override
+ public List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ return getStoreFilePaths(tableName, regionName, storeName, STOREFILE_INCLUDED_STR);
+ }
+
+ protected static byte[] storeFileListToByteArray(List<Path> storeFilePaths) {
+ return Bytes.toBytes(Joiner.on(LIST_SEPARATOR).join(storeFilePaths));
+ }
+
+ protected static List<Path> byteToStoreFileList(byte[] data) {
+ List<Path> paths = new ArrayList<>();
+ if (data != null && data.length != 0) {
+ String pathString = Bytes.toString(data);
+ String[] pathStrings = pathString.split(LIST_SEPARATOR);
Review comment:
What's the problem with regex split? I can think of the following options which might be more performant:
- compiled regex Pattern and split()
- StringTokenizer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-788701355
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 7s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 33s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 7m 35s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | HBASE-24749 passed |
| -0 :warning: | patch | 9m 3s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 10s | the patch passed |
| +1 :green_heart: | compile | 1m 34s | the patch passed |
| +1 :green_heart: | javac | 1m 34s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 45s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 33s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 160m 20s | hbase-server in the patch passed. |
| | | 194m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 6da49f79faaa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/testReport/ |
| Max. process+thread count | 3850 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-774398556
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 41s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 21s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 34s | the patch passed |
| +1 :green_heart: | compile | 1m 21s | the patch passed |
| +1 :green_heart: | javac | 1m 21s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 35s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 145m 39s | hbase-server in the patch passed. |
| | | 175m 41s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux bb2a09814507 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/testReport/ |
| Max. process+thread count | 4932 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r573100029
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StorefileTrackingUtils.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to support persistent storefile tracking
+ */
+@InterfaceAudience.Private
+public final class StorefileTrackingUtils {
+
+ private static Logger LOG = LoggerFactory.getLogger(StorefileTrackingUtils.class);
+ public static final long SLEEP_DELTA_MS = TimeUnit.MILLISECONDS.toMillis(100);
+
+ private StorefileTrackingUtils() {
+ // private for utility class
+ }
+
+ public static boolean isStorefileTrackingPersistEnabled(Configuration conf) {
+ boolean isStoreTrackingPersistEnabled =
+ conf.getBoolean(HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ HConstants.DEFAULT_STOREFILE_TRACKING_PERSIST_ENABLED);
+ boolean isPersistedStoreEngineSet =
+ conf.get(StoreEngine.STORE_ENGINE_CLASS_KEY, DefaultStoreEngine.class.getName())
+ .equals(PersistedStoreEngine.class.getName());
+ boolean isFeatureEnabled = isStoreTrackingPersistEnabled && isPersistedStoreEngineSet;
+ if (isStoreTrackingPersistEnabled ^ isPersistedStoreEngineSet) {
+ // check if both configuration are correct.
+ String errorMessage = String.format("please set %s to true and set store engine key %s to %s "
+ + "to enable persist storefile tracking",
+ HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ StoreEngine.STORE_ENGINE_CLASS_KEY, PersistedStoreEngine.class.getName());
+ throw new IllegalArgumentException(errorMessage);
+ }
+ return isFeatureEnabled;
+ }
+
+ /**
+ * if storefile tracking feature is configured, Initialize hbase:storefile table and wait for it
+ * to be online. Otherwise, look for hbase:storefile table and remove it
+ *
+ * @param masterServices masterServices
+ * @throws IOException if hbase:storefile table cannot be initialized and be online
+ */
+ public static void init(MasterServices masterServices) throws IOException {
+ createStorefileTable(masterServices);
+ waitForStoreFileTableOnline(masterServices);
+ }
+
+ /**
+ * Cleans up all storefile related state on the cluster. disable and delete hbase:storefile
+ * if found
+ * @param masterServices {@link MasterServices}
+ * @throws IOException if failures
+ */
+ private static void cleanup(MasterServices masterServices) throws IOException {
+ try {
+ masterServices.getConnection().getAdmin().disableTable(TableName.STOREFILE_TABLE_NAME);
+ masterServices.getConnection().getAdmin().deleteTable(TableName.STOREFILE_TABLE_NAME);
+ } catch (IOException ex) {
+ final String message = "Failed disable and deleting table " +
+ TableName.STOREFILE_TABLE_NAME.getNameAsString();
+ LOG.error(message);
+ throw new IOException(message, ex);
+ }
+ }
+
+ public static StoreFilePathAccessor createStoreFilePathAccessor(Configuration conf,
+ Connection connection) {
+ return new HTableStoreFilePathAccessor(conf, connection);
+ }
+
+ public static List<Path> convertStoreFilesToPaths(Collection<HStoreFile> storeFiles) {
+ return storeFiles.stream().map(HStoreFile::getPath).collect(Collectors.toList());
+ }
+
+ private static void createStorefileTable(MasterServices masterServices)
+ throws IOException {
+ if (MetaTableAccessor.getTableState(masterServices.getConnection(),
+ TableName.STOREFILE_TABLE_NAME) == null) {
+ LOG.info("{} table not found. Creating...", TableName.STOREFILE_TABLE_NAME);
+ masterServices.createSystemTable(HTableStoreFilePathAccessor.STOREFILE_TABLE_DESC);
+ }
+ }
+
+ private static void waitForStoreFileTableOnline(MasterServices masterServices)
+ throws IOException {
+ try {
+ long startTime = EnvironmentEdgeManager.currentTime();
+ long timeout = masterServices.getConfiguration()
+ .getLong(HConstants.STOREFILE_TRACKING_INIT_TIMEOUT,
+ HConstants.DEFAULT_STOREFILE_TRACKING_INIT_TIMEOUT);
+ while (!isStoreFileTableAssignedAndEnabled(masterServices)) {
Review comment:
I'm having nightmares about the hbase:namespace table not getting assigned and getting into cycles where the HBase master would just crashloop when the table wasn't getting assigned.
Not something that needs to be handled right now, but we should make sure we think about this later (rather than reintroduce old problems)
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
Review comment:
nit: Get
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @param storeFilePathUpdate Updates to be persisted
+ * @throws IOException if a remote or network exception occurs during write
+ */
+ void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) throws IOException;
+
+ /**
+ * Delete storefile paths for a tracking column family, normally used when a region-store is
+ * completely removed due to region split or merge
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @throws IOException if a remote or network exception occurs during delete
+ */
+ void deleteStoreFilePaths(final String tableName, final String regionName, final String storeName)
Review comment:
How does this method relate to `deleteRegion`? Would this be used for a `truncate` like operation?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
##########
@@ -203,5 +213,21 @@ public double getCompactionPressure() {
public Comparator<HStoreFile> getStoreFileComparator() {
return storeFileComparator;
}
+
+ void setStorefiles(ImmutableList<HStoreFile> storefiles) {
Review comment:
nit `setStoreFiles`
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.filter.RegexStringComparator;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
+
+@InterfaceAudience.Private
+public class HTableStoreFilePathAccessor extends AbstractStoreFilePathAccessor {
+
+ public static final byte[] STOREFILE_FAMILY_INCLUDED = Bytes.toBytes(STOREFILE_INCLUDED_STR);
+
+ private static final String DASH_SEPARATOR = "-";
+ private static final String STOREFILE_QUALIFIER_STR = "filepaths";
+ private static final byte[] STOREFILE_QUALIFIER = Bytes.toBytes(STOREFILE_QUALIFIER_STR);
+ private static final int STOREFILE_TABLE_VERSIONS = 3;
+
+ // TODO find a way for system table to support region split at table creation or remove this
+ // comment when we merge into hbase:meta table
+ public static final TableDescriptor STOREFILE_TABLE_DESC =
+ TableDescriptorBuilder.newBuilder(TableName.STOREFILE_TABLE_NAME)
+ .setColumnFamily(
+ ColumnFamilyDescriptorBuilder.newBuilder(STOREFILE_FAMILY_INCLUDED)
+ .setMaxVersions(STOREFILE_TABLE_VERSIONS)
+ .setInMemory(true)
+ .build())
+ .setRegionSplitPolicyClassName(BusyRegionSplitPolicy.class.getName())
+ .build();
+
+ private Connection connection;
+
+ public HTableStoreFilePathAccessor(Configuration conf, Connection connection) {
+ super(conf);
+ Preconditions.checkNotNull(connection, "connection cannot be null");
+ this.connection = connection;
+ }
+
+ @Override
+ public void initialize(final MasterServices masterServices) throws IOException {
+ StorefileTrackingUtils.init(masterServices);
+ }
+
+ @Override
+ List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String colFamily) throws IOException {
+ validate(tableName, regionName, storeName, colFamily);
+ byte[] colFamilyBytes = Bytes.toBytes(colFamily);
+ Get get =
+ new Get(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ get.addColumn(colFamilyBytes, STOREFILE_QUALIFIER);
+ Result result = doGet(get);
+ if (result == null || result.isEmpty()) {
+ return new ArrayList<>();
+ }
+ return byteToStoreFileList(result.getValue(colFamilyBytes, STOREFILE_QUALIFIER));
+ }
+
+ @Override
+ public void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException {
+ validate(tableName, regionName, storeName, storeFilePathUpdate);
+ Put put = generatePutForStoreFilePaths(tableName, regionName, storeName, storeFilePathUpdate);
+ doPut(put);
+ }
+
+
+ private Put generatePutForStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) {
+ Put put = new Put(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ if (storeFilePathUpdate.hasStoreFilesUpdate()) {
+ put.addColumn(Bytes.toBytes(STOREFILE_INCLUDED_STR), STOREFILE_QUALIFIER,
+ storeFileListToByteArray(storeFilePathUpdate.getStoreFiles()));
+ }
+ return put;
+ }
+
+ @Override
+ public void deleteStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ validate(tableName, regionName, storeName);
+ Delete delete = new Delete(
+ Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ delete.addColumns(STOREFILE_FAMILY_INCLUDED, STOREFILE_QUALIFIER);
+ doDelete(Lists.newArrayList(delete));
+ }
+
+ @Override
+ public void deleteRegion(String regionName) throws IOException {
+ Scan scan = getScanWithFilter(regionName);
+ List<Delete> familiesToDelete = new ArrayList<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ Delete delete = new Delete(Bytes.toBytes(rowKey));
+ familiesToDelete.add(delete);
+ }
+ doDelete(familiesToDelete);
+ }
+
+ @Override
+ public Set<String> getTrackedFamilies(String tableName, String regionName)
+ throws IOException {
+ // find all rows by regionName
+ Scan scan = getScanWithFilter(regionName);
+
+ Set<String> families = new HashSet<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ String family =
+ StorefileTrackingUtils.getFamilyFromKey(rowKey, tableName, regionName, getSeparator());
+ families.add(family);
+ }
+ return families;
+ }
+
+ @Override
+ String getSeparator() {
+ return DASH_SEPARATOR;
+ }
+
+ private Result doGet(final Get get) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ return table.get(get);
+ }
+ }
+
+ private void doPut(final Put put) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ table.put(put);
+ }
+ }
+
+ private void doDelete(final List<Delete> delete) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ table.delete(delete);
+ }
+ }
+
+ private ResultScanner getResultScanner(final Scan scan) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ return table.getScanner(scan);
+ }
+ }
+
+ private Connection getConnection() throws IOException {
+ if (connection == null) {
+ throw new IOException("Connection should be provided by region server "
+ + "and should not be null after initialized.");
+ }
+ return connection;
+ }
+
+
+ private Scan getScanWithFilter(String regionName) {
+ Scan scan = new Scan();
+ String regexPattern = "^" + regionName + getSeparator();
+ RowFilter rowFilter = new RowFilter(CompareOperator.EQUAL,
+ new RegexStringComparator(regexPattern));
Review comment:
Prefix filter would do the same thing without a regex cost, no?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathUpdate.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.List;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
Review comment:
Do we have commons-lang in hbase-thirdparty?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractStoreFilePathAccessor.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Joiner;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+
+@InterfaceAudience.Private
+public abstract class AbstractStoreFilePathAccessor implements StoreFilePathAccessor {
+
+ public static final String STOREFILE_INCLUDED_STR = "included";
+
+ protected static final String LIST_SEPARATOR = ";";
+ protected final Configuration conf;
+
+ public AbstractStoreFilePathAccessor(Configuration conf) {
+ this.conf = conf;
+ }
+
+ abstract String getSeparator();
+
+ abstract List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String columnName) throws IOException;
+
+ @Override
+ public abstract void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException;
+
+ @Override
+ public List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ return getStoreFilePaths(tableName, regionName, storeName, STOREFILE_INCLUDED_STR);
+ }
+
+ protected static byte[] storeFileListToByteArray(List<Path> storeFilePaths) {
+ return Bytes.toBytes(Joiner.on(LIST_SEPARATOR).join(storeFilePaths));
+ }
+
+ protected static List<Path> byteToStoreFileList(byte[] data) {
+ List<Path> paths = new ArrayList<>();
+ if (data != null && data.length != 0) {
+ String pathString = Bytes.toString(data);
+ String[] pathStrings = pathString.split(LIST_SEPARATOR);
Review comment:
This is doing a regex operation to split this. Should make sure we're doing something which avoids that.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StorefileTrackingUtils.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to support persistent storefile tracking
+ */
+@InterfaceAudience.Private
+public final class StorefileTrackingUtils {
+
+ private static Logger LOG = LoggerFactory.getLogger(StorefileTrackingUtils.class);
+ public static final long SLEEP_DELTA_MS = TimeUnit.MILLISECONDS.toMillis(100);
Review comment:
private instead of public?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.filter.RegexStringComparator;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
+
+@InterfaceAudience.Private
+public class HTableStoreFilePathAccessor extends AbstractStoreFilePathAccessor {
+
+ public static final byte[] STOREFILE_FAMILY_INCLUDED = Bytes.toBytes(STOREFILE_INCLUDED_STR);
+
+ private static final String DASH_SEPARATOR = "-";
+ private static final String STOREFILE_QUALIFIER_STR = "filepaths";
+ private static final byte[] STOREFILE_QUALIFIER = Bytes.toBytes(STOREFILE_QUALIFIER_STR);
+ private static final int STOREFILE_TABLE_VERSIONS = 3;
+
+ // TODO find a way for system table to support region split at table creation or remove this
+ // comment when we merge into hbase:meta table
+ public static final TableDescriptor STOREFILE_TABLE_DESC =
+ TableDescriptorBuilder.newBuilder(TableName.STOREFILE_TABLE_NAME)
+ .setColumnFamily(
+ ColumnFamilyDescriptorBuilder.newBuilder(STOREFILE_FAMILY_INCLUDED)
+ .setMaxVersions(STOREFILE_TABLE_VERSIONS)
+ .setInMemory(true)
+ .build())
+ .setRegionSplitPolicyClassName(BusyRegionSplitPolicy.class.getName())
+ .build();
+
+ private Connection connection;
+
+ public HTableStoreFilePathAccessor(Configuration conf, Connection connection) {
+ super(conf);
+ Preconditions.checkNotNull(connection, "connection cannot be null");
+ this.connection = connection;
+ }
+
+ @Override
+ public void initialize(final MasterServices masterServices) throws IOException {
+ StorefileTrackingUtils.init(masterServices);
+ }
+
+ @Override
+ List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String colFamily) throws IOException {
+ validate(tableName, regionName, storeName, colFamily);
+ byte[] colFamilyBytes = Bytes.toBytes(colFamily);
+ Get get =
+ new Get(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ get.addColumn(colFamilyBytes, STOREFILE_QUALIFIER);
+ Result result = doGet(get);
+ if (result == null || result.isEmpty()) {
Review comment:
meta-question: do we actually return `null` via a `Get` in some cases? That seems silly.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.filter.RegexStringComparator;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
+
+@InterfaceAudience.Private
+public class HTableStoreFilePathAccessor extends AbstractStoreFilePathAccessor {
+
+ public static final byte[] STOREFILE_FAMILY_INCLUDED = Bytes.toBytes(STOREFILE_INCLUDED_STR);
+
+ private static final String DASH_SEPARATOR = "-";
+ private static final String STOREFILE_QUALIFIER_STR = "filepaths";
+ private static final byte[] STOREFILE_QUALIFIER = Bytes.toBytes(STOREFILE_QUALIFIER_STR);
+ private static final int STOREFILE_TABLE_VERSIONS = 3;
+
+ // TODO find a way for system table to support region split at table creation or remove this
+ // comment when we merge into hbase:meta table
+ public static final TableDescriptor STOREFILE_TABLE_DESC =
+ TableDescriptorBuilder.newBuilder(TableName.STOREFILE_TABLE_NAME)
+ .setColumnFamily(
+ ColumnFamilyDescriptorBuilder.newBuilder(STOREFILE_FAMILY_INCLUDED)
+ .setMaxVersions(STOREFILE_TABLE_VERSIONS)
+ .setInMemory(true)
+ .build())
+ .setRegionSplitPolicyClassName(BusyRegionSplitPolicy.class.getName())
+ .build();
+
+ private Connection connection;
+
+ public HTableStoreFilePathAccessor(Configuration conf, Connection connection) {
+ super(conf);
+ Preconditions.checkNotNull(connection, "connection cannot be null");
+ this.connection = connection;
+ }
+
+ @Override
+ public void initialize(final MasterServices masterServices) throws IOException {
+ StorefileTrackingUtils.init(masterServices);
+ }
+
+ @Override
+ List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String colFamily) throws IOException {
+ validate(tableName, regionName, storeName, colFamily);
+ byte[] colFamilyBytes = Bytes.toBytes(colFamily);
+ Get get =
+ new Get(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ get.addColumn(colFamilyBytes, STOREFILE_QUALIFIER);
+ Result result = doGet(get);
+ if (result == null || result.isEmpty()) {
+ return new ArrayList<>();
+ }
+ return byteToStoreFileList(result.getValue(colFamilyBytes, STOREFILE_QUALIFIER));
+ }
+
+ @Override
+ public void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException {
+ validate(tableName, regionName, storeName, storeFilePathUpdate);
+ Put put = generatePutForStoreFilePaths(tableName, regionName, storeName, storeFilePathUpdate);
+ doPut(put);
+ }
+
+
+ private Put generatePutForStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) {
+ Put put = new Put(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ if (storeFilePathUpdate.hasStoreFilesUpdate()) {
+ put.addColumn(Bytes.toBytes(STOREFILE_INCLUDED_STR), STOREFILE_QUALIFIER,
+ storeFileListToByteArray(storeFilePathUpdate.getStoreFiles()));
+ }
+ return put;
+ }
+
+ @Override
+ public void deleteStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ validate(tableName, regionName, storeName);
+ Delete delete = new Delete(
+ Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ delete.addColumns(STOREFILE_FAMILY_INCLUDED, STOREFILE_QUALIFIER);
+ doDelete(Lists.newArrayList(delete));
+ }
+
+ @Override
+ public void deleteRegion(String regionName) throws IOException {
+ Scan scan = getScanWithFilter(regionName);
+ List<Delete> familiesToDelete = new ArrayList<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ Delete delete = new Delete(Bytes.toBytes(rowKey));
+ familiesToDelete.add(delete);
+ }
+ doDelete(familiesToDelete);
+ }
+
+ @Override
+ public Set<String> getTrackedFamilies(String tableName, String regionName)
+ throws IOException {
+ // find all rows by regionName
+ Scan scan = getScanWithFilter(regionName);
+
+ Set<String> families = new HashSet<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ String family =
+ StorefileTrackingUtils.getFamilyFromKey(rowKey, tableName, regionName, getSeparator());
+ families.add(family);
+ }
+ return families;
+ }
+
+ @Override
+ String getSeparator() {
+ return DASH_SEPARATOR;
+ }
+
+ private Result doGet(final Get get) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ return table.get(get);
+ }
+ }
+
+ private void doPut(final Put put) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ table.put(put);
+ }
+ }
+
+ private void doDelete(final List<Delete> delete) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ table.delete(delete);
+ }
+ }
+
+ private ResultScanner getResultScanner(final Scan scan) throws IOException {
+ try (Table table = getConnection().getTable(TableName.STOREFILE_TABLE_NAME)) {
+ return table.getScanner(scan);
+ }
+ }
+
+ private Connection getConnection() throws IOException {
+ if (connection == null) {
+ throw new IOException("Connection should be provided by region server "
+ + "and should not be null after initialized.");
+ }
Review comment:
Make `connection` final and then you can drop this.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableSortedSet;
+import org.apache.hbase.thirdparty.com.google.common.collect.Iterables;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private StoreFilePathAccessor accessor;
+ private Configuration conf;
+ // only uses for warmupHRegion
+ private boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ /**
+ * Loads the specified storeFiles to the StoreFileManager to be included in reads.
+ *
+ * @param storeFiles list of storefiles to be loaded, could be an empty list. throws exception
+ * if it's null.
+ */
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ ImmutableList<HStoreFile> sortedStorefiles =
+ ImmutableList.sortedCopyOf(getStoreFileComparator(), storeFiles);
+ setStorefiles(sortedStorefiles);
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(sortedStorefiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
+ if (isEmptyInPersistedFilePaths) {
+ // When the path accessor is returning empty result, we scan the
+ // the file storage and see if there is any existing HFiles should be loaded.
+ // the scan is a one time process when store open during region assignment.
+ //
+ // this is especially used for region and store open
+ // 1. First time migration from a filesystem based e.g. DefaultStoreFileEngine
+ // 2. After region split and merge
+ // 3. After table clone and create new HFiles directly into data directory
+ //
+ // Also we don't handle the inconsistency between storefile tracking and file system, which
+ // will be handled by a HBCK command
+ LOG.info("Cannot find tracking paths ({}) for store {} in region {} of "
+ + "table {}, fall back to scan the storage to get a list of storefiles to be opened"
+ , isEmptyInPersistedFilePaths, storeName, regionName,
+ tableName);
+ return getRegionFs().getStoreFiles(getFamilyName());
+ }
+ ArrayList<StoreFileInfo> storeFiles = new ArrayList<>();
+ for (Path storeFilePath : pathList) {
+ if (!StoreFileInfo.isValid(getRegionFs().getFileSystem().getFileStatus(storeFilePath))) {
+ // TODO add a comment how this file will be removed when we have cleaner
+ LOG.warn("Invalid StoreFile: {}", storeFilePath);
+ continue;
+ }
+ StoreFileInfo info = ServerRegionReplicaUtil
+ .getStoreFileInfo(conf, getRegionFs().getFileSystem(), regionInfo,
+ ServerRegionReplicaUtil.getRegionInfoForFs(regionInfo), getFamilyName(),
+ storeFilePath);
+ storeFiles.add(info);
+ }
+ return storeFiles;
+ }
+
+
+ @Override
+ public void insertNewFiles(Collection<HStoreFile> sfs) throws IOException {
+ // return in case of empty storeFiles as it is a No-op, empty files expected during region close
+ if (CollectionUtils.isEmpty(sfs)) {
+ return;
+ }
+ ImmutableList<HStoreFile> storefiles = ImmutableList
+ .sortedCopyOf(getStoreFileComparator(), Iterables.concat(getStorefiles(), sfs));
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storefiles).build());
+ setStorefiles(storefiles);
+ }
+
+ @Override
+ public void addCompactionResults(Collection<HStoreFile> newCompactedfiles,
Review comment:
nit `newCompactedFiles`
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HTableStoreFilePathAccessor.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.filter.RegexStringComparator;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+/**
+ * Helper class to interact with the hbase:storefile system table
+ *
+ * <pre>
+ * ROW-KEY FAMILY:QUALIFIER DATA VALUE
+ * region-store-table included:files List<Path> filesIncludedInRead
+ * </pre>
+ *
+ * The region encoded name is set as prefix for region split loading balance, and we use the
+ * target table name as suffix such that operator can identify the records per table.
+ *
+ * included:files is used for persisting storefiles of StoreFileManager in the cases of store
+ * opens and store closes. Meanwhile compactedFiles of StoreFileManager isn't being tracked
+ * off-memory, because the updated included:files contains compactedFiles and the leftover
+ * compactedFiles are either archived when a store closes or opens.
+ *
+ * TODO we will need a followup change to introduce in-memory temporarily file, such that further
+ * we can introduce a non-tracking temporarily storefiles left from a flush or compaction when
+ * a regionserver crashes without closing the store properly
+ */
+
+@InterfaceAudience.Private
+public class HTableStoreFilePathAccessor extends AbstractStoreFilePathAccessor {
+
+ public static final byte[] STOREFILE_FAMILY_INCLUDED = Bytes.toBytes(STOREFILE_INCLUDED_STR);
+
+ private static final String DASH_SEPARATOR = "-";
+ private static final String STOREFILE_QUALIFIER_STR = "filepaths";
+ private static final byte[] STOREFILE_QUALIFIER = Bytes.toBytes(STOREFILE_QUALIFIER_STR);
+ private static final int STOREFILE_TABLE_VERSIONS = 3;
+
+ // TODO find a way for system table to support region split at table creation or remove this
+ // comment when we merge into hbase:meta table
+ public static final TableDescriptor STOREFILE_TABLE_DESC =
+ TableDescriptorBuilder.newBuilder(TableName.STOREFILE_TABLE_NAME)
+ .setColumnFamily(
+ ColumnFamilyDescriptorBuilder.newBuilder(STOREFILE_FAMILY_INCLUDED)
+ .setMaxVersions(STOREFILE_TABLE_VERSIONS)
+ .setInMemory(true)
+ .build())
+ .setRegionSplitPolicyClassName(BusyRegionSplitPolicy.class.getName())
+ .build();
+
+ private Connection connection;
+
+ public HTableStoreFilePathAccessor(Configuration conf, Connection connection) {
+ super(conf);
+ Preconditions.checkNotNull(connection, "connection cannot be null");
+ this.connection = connection;
+ }
+
+ @Override
+ public void initialize(final MasterServices masterServices) throws IOException {
+ StorefileTrackingUtils.init(masterServices);
+ }
+
+ @Override
+ List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String colFamily) throws IOException {
+ validate(tableName, regionName, storeName, colFamily);
+ byte[] colFamilyBytes = Bytes.toBytes(colFamily);
+ Get get =
+ new Get(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ get.addColumn(colFamilyBytes, STOREFILE_QUALIFIER);
+ Result result = doGet(get);
+ if (result == null || result.isEmpty()) {
+ return new ArrayList<>();
+ }
+ return byteToStoreFileList(result.getValue(colFamilyBytes, STOREFILE_QUALIFIER));
+ }
+
+ @Override
+ public void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException {
+ validate(tableName, regionName, storeName, storeFilePathUpdate);
+ Put put = generatePutForStoreFilePaths(tableName, regionName, storeName, storeFilePathUpdate);
+ doPut(put);
+ }
+
+
+ private Put generatePutForStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) {
+ Put put = new Put(Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ if (storeFilePathUpdate.hasStoreFilesUpdate()) {
+ put.addColumn(Bytes.toBytes(STOREFILE_INCLUDED_STR), STOREFILE_QUALIFIER,
+ storeFileListToByteArray(storeFilePathUpdate.getStoreFiles()));
+ }
+ return put;
+ }
+
+ @Override
+ public void deleteStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ validate(tableName, regionName, storeName);
+ Delete delete = new Delete(
+ Bytes.toBytes(getKey(tableName, regionName, storeName)));
+ delete.addColumns(STOREFILE_FAMILY_INCLUDED, STOREFILE_QUALIFIER);
+ doDelete(Lists.newArrayList(delete));
+ }
+
+ @Override
+ public void deleteRegion(String regionName) throws IOException {
+ Scan scan = getScanWithFilter(regionName);
+ List<Delete> familiesToDelete = new ArrayList<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ Delete delete = new Delete(Bytes.toBytes(rowKey));
+ familiesToDelete.add(delete);
+ }
+ doDelete(familiesToDelete);
+ }
+
+ @Override
+ public Set<String> getTrackedFamilies(String tableName, String regionName)
+ throws IOException {
+ // find all rows by regionName
+ Scan scan = getScanWithFilter(regionName);
+
+ Set<String> families = new HashSet<>();
+ for (Result result : getResultScanner(scan)) {
+ String rowKey = Bytes.toString(result.getRow());
+ String family =
+ StorefileTrackingUtils.getFamilyFromKey(rowKey, tableName, regionName, getSeparator());
+ families.add(family);
Review comment:
Would be much better to do this without converting back to Strings.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
##########
@@ -203,5 +213,21 @@ public double getCompactionPressure() {
public Comparator<HStoreFile> getStoreFileComparator() {
return storeFileComparator;
}
+
+ void setStorefiles(ImmutableList<HStoreFile> storefiles) {
+ this.storefiles = storefiles;
+ }
+
+ void setCompactedfiles(ImmutableList<HStoreFile> compactedfiles) {
Review comment:
nit `setCompactedFiles`
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @param storeFilePathUpdate Updates to be persisted
+ * @throws IOException if a remote or network exception occurs during write
+ */
+ void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) throws IOException;
+
+ /**
+ * Delete storefile paths for a tracking column family, normally used when a region-store is
+ * completely removed due to region split or merge
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @throws IOException if a remote or network exception occurs during delete
+ */
+ void deleteStoreFilePaths(final String tableName, final String regionName, final String storeName)
+ throws IOException;
+
+ /**
+ * Returns the families being tracked in the storefile tracking data for the given
+ * table/region
Review comment:
For user-regions (not meta), is there a reason that we'd only want a subset of Stores to be using tracking data?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StorefileTrackingUtils.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to support persistent storefile tracking
+ */
+@InterfaceAudience.Private
+public final class StorefileTrackingUtils {
+
+ private static Logger LOG = LoggerFactory.getLogger(StorefileTrackingUtils.class);
+ public static final long SLEEP_DELTA_MS = TimeUnit.MILLISECONDS.toMillis(100);
+
+ private StorefileTrackingUtils() {
+ // private for utility class
+ }
+
+ public static boolean isStorefileTrackingPersistEnabled(Configuration conf) {
+ boolean isStoreTrackingPersistEnabled =
+ conf.getBoolean(HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ HConstants.DEFAULT_STOREFILE_TRACKING_PERSIST_ENABLED);
+ boolean isPersistedStoreEngineSet =
+ conf.get(StoreEngine.STORE_ENGINE_CLASS_KEY, DefaultStoreEngine.class.getName())
+ .equals(PersistedStoreEngine.class.getName());
+ boolean isFeatureEnabled = isStoreTrackingPersistEnabled && isPersistedStoreEngineSet;
+ if (isStoreTrackingPersistEnabled ^ isPersistedStoreEngineSet) {
+ // check if both configuration are correct.
+ String errorMessage = String.format("please set %s to true and set store engine key %s to %s "
+ + "to enable persist storefile tracking",
+ HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ StoreEngine.STORE_ENGINE_CLASS_KEY, PersistedStoreEngine.class.getName());
+ throw new IllegalArgumentException(errorMessage);
+ }
+ return isFeatureEnabled;
+ }
+
+ /**
+ * if storefile tracking feature is configured, Initialize hbase:storefile table and wait for it
+ * to be online. Otherwise, look for hbase:storefile table and remove it
+ *
+ * @param masterServices masterServices
+ * @throws IOException if hbase:storefile table cannot be initialized and be online
+ */
+ public static void init(MasterServices masterServices) throws IOException {
+ createStorefileTable(masterServices);
+ waitForStoreFileTableOnline(masterServices);
+ }
+
+ /**
+ * Cleans up all storefile related state on the cluster. disable and delete hbase:storefile
+ * if found
+ * @param masterServices {@link MasterServices}
+ * @throws IOException if failures
+ */
+ private static void cleanup(MasterServices masterServices) throws IOException {
+ try {
+ masterServices.getConnection().getAdmin().disableTable(TableName.STOREFILE_TABLE_NAME);
+ masterServices.getConnection().getAdmin().deleteTable(TableName.STOREFILE_TABLE_NAME);
+ } catch (IOException ex) {
+ final String message = "Failed disable and deleting table " +
+ TableName.STOREFILE_TABLE_NAME.getNameAsString();
+ LOG.error(message);
+ throw new IOException(message, ex);
+ }
+ }
+
+ public static StoreFilePathAccessor createStoreFilePathAccessor(Configuration conf,
+ Connection connection) {
+ return new HTableStoreFilePathAccessor(conf, connection);
+ }
+
+ public static List<Path> convertStoreFilesToPaths(Collection<HStoreFile> storeFiles) {
Review comment:
nit just call this `toPaths`
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableSortedSet;
+import org.apache.hbase.thirdparty.com.google.common.collect.Iterables;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private StoreFilePathAccessor accessor;
+ private Configuration conf;
+ // only uses for warmupHRegion
+ private boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ /**
+ * Loads the specified storeFiles to the StoreFileManager to be included in reads.
+ *
+ * @param storeFiles list of storefiles to be loaded, could be an empty list. throws exception
+ * if it's null.
+ */
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ ImmutableList<HStoreFile> sortedStorefiles =
+ ImmutableList.sortedCopyOf(getStoreFileComparator(), storeFiles);
+ setStorefiles(sortedStorefiles);
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(sortedStorefiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
+ if (isEmptyInPersistedFilePaths) {
+ // When the path accessor is returning empty result, we scan the
+ // the file storage and see if there is any existing HFiles should be loaded.
+ // the scan is a one time process when store open during region assignment.
+ //
+ // this is especially used for region and store open
+ // 1. First time migration from a filesystem based e.g. DefaultStoreFileEngine
+ // 2. After region split and merge
+ // 3. After table clone and create new HFiles directly into data directory
+ //
+ // Also we don't handle the inconsistency between storefile tracking and file system, which
+ // will be handled by a HBCK command
+ LOG.info("Cannot find tracking paths ({}) for store {} in region {} of "
+ + "table {}, fall back to scan the storage to get a list of storefiles to be opened"
+ , isEmptyInPersistedFilePaths, storeName, regionName,
+ tableName);
+ return getRegionFs().getStoreFiles(getFamilyName());
+ }
+ ArrayList<StoreFileInfo> storeFiles = new ArrayList<>();
+ for (Path storeFilePath : pathList) {
+ if (!StoreFileInfo.isValid(getRegionFs().getFileSystem().getFileStatus(storeFilePath))) {
+ // TODO add a comment how this file will be removed when we have cleaner
+ LOG.warn("Invalid StoreFile: {}", storeFilePath);
+ continue;
+ }
+ StoreFileInfo info = ServerRegionReplicaUtil
+ .getStoreFileInfo(conf, getRegionFs().getFileSystem(), regionInfo,
+ ServerRegionReplicaUtil.getRegionInfoForFs(regionInfo), getFamilyName(),
+ storeFilePath);
+ storeFiles.add(info);
+ }
+ return storeFiles;
+ }
+
+
+ @Override
+ public void insertNewFiles(Collection<HStoreFile> sfs) throws IOException {
+ // return in case of empty storeFiles as it is a No-op, empty files expected during region close
+ if (CollectionUtils.isEmpty(sfs)) {
+ return;
+ }
+ ImmutableList<HStoreFile> storefiles = ImmutableList
+ .sortedCopyOf(getStoreFileComparator(), Iterables.concat(getStorefiles(), sfs));
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storefiles).build());
+ setStorefiles(storefiles);
+ }
+
+ @Override
+ public void addCompactionResults(Collection<HStoreFile> newCompactedfiles,
+ Collection<HStoreFile> results) throws IOException {
+ Preconditions.checkNotNull(newCompactedfiles, "compactedFiles cannot be null");
+ Preconditions.checkNotNull(results, "compaction result cannot be null");
+ // only allow distinct path to be included, especially rerun after a compaction fails
+ ImmutableList<HStoreFile> storefiles = ImmutableList.sortedCopyOf(getStoreFileComparator(),
+ Iterables.concat(Iterables.filter(getStorefiles(), sf -> !newCompactedfiles.contains(sf)),
+ results)).asList();
+ Preconditions.checkArgument(!CollectionUtils.isEmpty(storefiles),
+ "storefiles cannot be empty when adding compaction results");
+
+ ImmutableList<HStoreFile> compactedfiles = ImmutableSortedSet
+ .copyOf(getStoreFileComparator(),
+ Iterables.concat(getCompactedfiles(), newCompactedfiles))
+ .asList();
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storefiles).build());
+ setStorefiles(storefiles);
+ setCompactedfiles(compactedfiles);
+ newCompactedfiles.forEach(HStoreFile::markCompactedAway);
+ }
+
+ void updatePathListToTracker(StoreFilePathUpdate storeFilePathUpdate) throws IOException {
+ try {
+ if (!readOnly) {
+ accessor.writeStoreFilePaths(tableName, regionName, storeName, storeFilePathUpdate);
+ }
+ } catch (IOException e) {
+ String message = "Failed to update Path list of " + tableName + "-" + regionName +
+ "-" + storeName + ", on " + TableName.STOREFILE_STR + ". The new files are not "
+ + "persistent and will be removed from " + regionName + "," + storeName +
+ ".\nFailed update: " + storeFilePathUpdate;
+ LOG.warn(message);
Review comment:
String.format or the parameterized log message, please.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableSortedSet;
+import org.apache.hbase.thirdparty.com.google.common.collect.Iterables;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private StoreFilePathAccessor accessor;
+ private Configuration conf;
+ // only uses for warmupHRegion
+ private boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ /**
+ * Loads the specified storeFiles to the StoreFileManager to be included in reads.
+ *
+ * @param storeFiles list of storefiles to be loaded, could be an empty list. throws exception
+ * if it's null.
+ */
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ ImmutableList<HStoreFile> sortedStorefiles =
+ ImmutableList.sortedCopyOf(getStoreFileComparator(), storeFiles);
+ setStorefiles(sortedStorefiles);
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(sortedStorefiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
Review comment:
Is there a way we can definitively know that our storefilepaths is uninitialized and we need to perform this action vs. an empty store?
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessorTestBase.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.hamcrest.Matchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TestName;
+
+import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
+public abstract class StoreFilePathAccessorTestBase {
+
+ @Rule
+ public TestName name = new TestName();
+
+ @Rule
+ public ExpectedException expectedException = ExpectedException.none();
+
+ protected static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
+ protected StoreFilePathAccessor storeFilePathAccessor;
+ protected static String REGION_NAME = UUID.randomUUID().toString().replaceAll("-", "");
+ protected static String STORE_NAME = UUID.randomUUID().toString();
+ protected static List<Path> EMPTY_PATH = Collections.emptyList();
+ protected static List<Path> INCLUDE_EXAMPLE_PATH =
+ Lists.newArrayList(new Path("hdfs://foo/bar1"), new Path("hdfs://foo/bar2"));
Review comment:
final?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
##########
@@ -203,5 +213,21 @@ public double getCompactionPressure() {
public Comparator<HStoreFile> getStoreFileComparator() {
return storeFileComparator;
}
+
+ void setStorefiles(ImmutableList<HStoreFile> storefiles) {
+ this.storefiles = storefiles;
+ }
+
+ void setCompactedfiles(ImmutableList<HStoreFile> compactedfiles) {
Review comment:
I'd challenge you to see if there's a better API we could add to DefaultStoreFileManager than this. It's a little brittle in that PersistedStoreFileManager has to be knowing when to call these setters. Maybe there's API in which there can be one method in this class which computes and sets the compactedFiles and "current" files which both this implementation and the new implementation can leverage?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
Review comment:
What is a "write" in this context? Defining what the total set of files for this Region+Store is supposed to be? Not clear from javadoc.
`storeFilePathUpdate`argument javadoc is also a little vague (i.e. what's an "update"?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585149492
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
Review comment:
thanks @anmolnar and you're right on the entity of StoreFilePathUpdate, I will adjust the comments.
also I will remove the `hasStoreFilesUpdate` (some old code left behind)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585247357
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StorefileTrackingUtils.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to support persistent storefile tracking
+ */
+@InterfaceAudience.Private
+public final class StorefileTrackingUtils {
+
+ private static Logger LOG = LoggerFactory.getLogger(StorefileTrackingUtils.class);
+ public static final long SLEEP_DELTA_MS = TimeUnit.MILLISECONDS.toMillis(100);
+
+ private StorefileTrackingUtils() {
+ // private for utility class
+ }
+
+ public static boolean isStorefileTrackingPersistEnabled(Configuration conf) {
+ boolean isStoreTrackingPersistEnabled =
+ conf.getBoolean(HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ HConstants.DEFAULT_STOREFILE_TRACKING_PERSIST_ENABLED);
+ boolean isPersistedStoreEngineSet =
+ conf.get(StoreEngine.STORE_ENGINE_CLASS_KEY, DefaultStoreEngine.class.getName())
+ .equals(PersistedStoreEngine.class.getName());
+ boolean isFeatureEnabled = isStoreTrackingPersistEnabled && isPersistedStoreEngineSet;
+ if (isStoreTrackingPersistEnabled ^ isPersistedStoreEngineSet) {
+ // check if both configuration are correct.
+ String errorMessage = String.format("please set %s to true and set store engine key %s to %s "
+ + "to enable persist storefile tracking",
+ HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ StoreEngine.STORE_ENGINE_CLASS_KEY, PersistedStoreEngine.class.getName());
+ throw new IllegalArgumentException(errorMessage);
+ }
+ return isFeatureEnabled;
+ }
+
+ /**
+ * if storefile tracking feature is configured, Initialize hbase:storefile table and wait for it
+ * to be online. Otherwise, look for hbase:storefile table and remove it
+ *
+ * @param masterServices masterServices
+ * @throws IOException if hbase:storefile table cannot be initialized and be online
+ */
+ public static void init(MasterServices masterServices) throws IOException {
+ createStorefileTable(masterServices);
+ waitForStoreFileTableOnline(masterServices);
+ }
+
+ /**
+ * Cleans up all storefile related state on the cluster. disable and delete hbase:storefile
+ * if found
+ * @param masterServices {@link MasterServices}
+ * @throws IOException if failures
+ */
+ private static void cleanup(MasterServices masterServices) throws IOException {
Review comment:
removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585247985
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @param storeFilePathUpdate Updates to be persisted
+ * @throws IOException if a remote or network exception occurs during write
+ */
+ void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) throws IOException;
+
+ /**
+ * Delete storefile paths for a tracking column family, normally used when a region-store is
+ * completely removed due to region split or merge
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @throws IOException if a remote or network exception occurs during delete
+ */
+ void deleteStoreFilePaths(final String tableName, final String regionName, final String storeName)
+ throws IOException;
+
+ /**
+ * Returns the families being tracked in the storefile tracking data for the given
+ * table/region
Review comment:
at the end, I removed it and will add it back when supporting snapshot (restore snapshot) feature
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-789418193
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 27s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 21s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | HBASE-24749 passed |
| -0 :warning: | patch | 7m 59s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 32s | the patch passed |
| +1 :green_heart: | compile | 1m 22s | the patch passed |
| +1 :green_heart: | javac | 1m 22s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 145m 39s | hbase-server in the patch passed. |
| | | 175m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 30010afa38cb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/testReport/ |
| Max. process+thread count | 4899 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-789380051
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 2s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 52s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 42s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 2m 2s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 3m 24s | HBASE-24749 passed |
| -0 :warning: | patch | 2m 43s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 24s | the patch passed |
| +1 :green_heart: | compile | 4m 41s | the patch passed |
| +1 :green_heart: | javac | 4m 41s | the patch passed |
| -0 :warning: | checkstyle | 1m 18s | hbase-server: The patch generated 2 new + 107 unchanged - 2 fixed = 109 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 21m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| | | 62m 49s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f01c3fc78d52 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/5/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-775486117
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 5s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 27s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 7m 20s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 4s | the patch passed |
| +1 :green_heart: | compile | 1m 31s | the patch passed |
| +1 :green_heart: | javac | 1m 31s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 18s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 27s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 149m 50s | hbase-server in the patch passed. |
| | | 182m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 5d1654fc43e8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/testReport/ |
| Max. process+thread count | 4111 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-774397788
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 44s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 34s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 44s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | HBASE-24749 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 13s | the patch passed |
| +1 :green_heart: | compile | 1m 37s | the patch passed |
| +1 :green_heart: | javac | 1m 37s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 34s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 135m 53s | hbase-server in the patch passed. |
| | | 168m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 4159204a74be 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/testReport/ |
| Max. process+thread count | 4239 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585156784
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @param storeFilePathUpdate Updates to be persisted
+ * @throws IOException if a remote or network exception occurs during write
+ */
+ void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final StoreFilePathUpdate storeFilePathUpdate) throws IOException;
+
+ /**
+ * Delete storefile paths for a tracking column family, normally used when a region-store is
+ * completely removed due to region split or merge
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @throws IOException if a remote or network exception occurs during delete
+ */
+ void deleteStoreFilePaths(final String tableName, final String regionName, final String storeName)
Review comment:
I will remove this function first in this PR, but this will be used by snapshot `restoreSnapshot` when we remove regions in [`regionsToRemove` and tracking data needs to be updated](https://github.com/apache/hbase/blob/01876071b990063899e0839cdda9745783fe814c/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RestoreSnapshotProcedure.java#L401-L404)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-788693264
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 7s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 31s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 22s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 47s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 11s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 48s | the patch passed |
| +1 :green_heart: | compile | 1m 26s | the patch passed |
| +1 :green_heart: | javac | 1m 26s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 56s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 147m 3s | hbase-server in the patch passed. |
| | | 179m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux e6df3ac3a0db 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/testReport/ |
| Max. process+thread count | 4289 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-797348104
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 17s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 34s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 50s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 4s | the patch passed |
| +1 :green_heart: | compile | 1m 31s | the patch passed |
| +1 :green_heart: | javac | 1m 31s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 41s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 46s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 136m 16s | hbase-server in the patch passed. |
| | | 168m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 17cdb9b459f5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/testReport/ |
| Max. process+thread count | 3855 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/8/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-790192310
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 31s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 19s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | HBASE-24749 passed |
| -0 :warning: | patch | 7m 59s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 27s | the patch passed |
| +1 :green_heart: | compile | 1m 22s | the patch passed |
| +1 :green_heart: | javac | 1m 22s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 144m 56s | hbase-server in the patch passed. |
| | | 174m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 572d1ef3cfc5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/testReport/ |
| Max. process+thread count | 4719 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-797495127
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 4m 6s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 34s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 21s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | HBASE-24749 passed |
| -0 :warning: | patch | 7m 57s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 32s | the patch passed |
| +1 :green_heart: | compile | 1m 21s | the patch passed |
| +1 :green_heart: | javac | 1m 21s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 157m 47s | hbase-server in the patch passed. |
| | | 191m 16s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux c0d9d2420496 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/testReport/ |
| Max. process+thread count | 4697 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-789503105
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 43s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 37s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 21s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 7m 7s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 27s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 26s | the patch passed |
| +1 :green_heart: | compile | 1m 21s | the patch passed |
| +1 :green_heart: | javac | 1m 21s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 30s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 149m 4s | hbase-server in the patch passed. |
| | | 179m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux e9b67fefad5b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/testReport/ |
| Max. process+thread count | 4776 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585250257
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableSortedSet;
+import org.apache.hbase.thirdparty.com.google.common.collect.Iterables;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private StoreFilePathAccessor accessor;
+ private Configuration conf;
+ // only uses for warmupHRegion
+ private boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ /**
+ * Loads the specified storeFiles to the StoreFileManager to be included in reads.
+ *
+ * @param storeFiles list of storefiles to be loaded, could be an empty list. throws exception
+ * if it's null.
+ */
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ ImmutableList<HStoreFile> sortedStorefiles =
+ ImmutableList.sortedCopyOf(getStoreFileComparator(), storeFiles);
+ setStorefiles(sortedStorefiles);
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(sortedStorefiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
Review comment:
IMO this is the cleanest by looking up the table for this store, and if the result is empty, it's definitely uninitialized because we don't allow writing empty list for a given store.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anmolnar commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r577614628
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPersistedStoreEngine.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.hamcrest.core.Is.isA;
+import static org.junit.Assert.assertEquals;
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.CellComparatorImpl;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TestName;
+import org.mockito.Mockito;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestPersistedStoreEngine {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestPersistedStoreEngine.class);
+
+ @Rule
+ public TestName name = new TestName();
+
+ @Rule
+ public ExpectedException expectedException = ExpectedException.none();
+
+ private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
+ private static final byte[] DEFAULT_STORE_BYTE = TEST_UTIL.fam1;
+
+ private TableName tableName;
+ private Configuration conf;
+ private HStore store;
+
+ @BeforeClass
+ public static void setUpCluster() throws Exception {
+ TEST_UTIL.getConfiguration().setBoolean(HConstants.STOREFILE_TRACKING_PERSIST_ENABLED, true);
+ TEST_UTIL.getConfiguration().set(StoreEngine.STORE_ENGINE_CLASS_KEY,
+ PersistedStoreEngine.class.getName());
+ TEST_UTIL.startMiniCluster();
+ }
+
+ @Before
+ public void setup() throws Exception {
+ store = Mockito.mock(HStore.class);
+ StoreContext context = new StoreContext.Builder().build();
+
+ conf = TEST_UTIL.getConfiguration();
+ Mockito.when(store.getStoreContext()).thenReturn(context);
+ Mockito.when(store.getRegionInfo()).thenReturn(RegionInfoBuilder.FIRST_META_REGIONINFO);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ for (TableDescriptor htd: TEST_UTIL.getAdmin().listTableDescriptors()) {
+ TEST_UTIL.deleteTable(htd.getTableName());
+ }
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+ // use killMiniHBaseCluster before shutdownMiniCluster because shutdownMiniCluster puts a RS
+ // that host hbase:meta to a stopped state that prevents meta lookup the table state of
+ // hbase:storefile and hangs the last write to hbase:storefile when any tracking region closes.
+ //
+ // this should be rarely happened on a real cluster and RS with meta table should be the last
+ // to be shutdown normally.
+ TEST_UTIL.killMiniHBaseCluster();
Review comment:
I think I understand the reasoning. Is that not the case in the other mini-cluster test `TestPersistedStoreFileManager` where you just shutdown the cluster without killing it?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
Review comment:
Is this method only used in tests?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StorefileTrackingUtils.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to support persistent storefile tracking
+ */
+@InterfaceAudience.Private
+public final class StorefileTrackingUtils {
+
+ private static Logger LOG = LoggerFactory.getLogger(StorefileTrackingUtils.class);
+ public static final long SLEEP_DELTA_MS = TimeUnit.MILLISECONDS.toMillis(100);
+
+ private StorefileTrackingUtils() {
+ // private for utility class
+ }
+
+ public static boolean isStorefileTrackingPersistEnabled(Configuration conf) {
+ boolean isStoreTrackingPersistEnabled =
+ conf.getBoolean(HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ HConstants.DEFAULT_STOREFILE_TRACKING_PERSIST_ENABLED);
+ boolean isPersistedStoreEngineSet =
+ conf.get(StoreEngine.STORE_ENGINE_CLASS_KEY, DefaultStoreEngine.class.getName())
+ .equals(PersistedStoreEngine.class.getName());
+ boolean isFeatureEnabled = isStoreTrackingPersistEnabled && isPersistedStoreEngineSet;
+ if (isStoreTrackingPersistEnabled ^ isPersistedStoreEngineSet) {
+ // check if both configuration are correct.
+ String errorMessage = String.format("please set %s to true and set store engine key %s to %s "
+ + "to enable persist storefile tracking",
+ HConstants.STOREFILE_TRACKING_PERSIST_ENABLED,
+ StoreEngine.STORE_ENGINE_CLASS_KEY, PersistedStoreEngine.class.getName());
+ throw new IllegalArgumentException(errorMessage);
+ }
+ return isFeatureEnabled;
+ }
+
+ /**
+ * if storefile tracking feature is configured, Initialize hbase:storefile table and wait for it
+ * to be online. Otherwise, look for hbase:storefile table and remove it
+ *
+ * @param masterServices masterServices
+ * @throws IOException if hbase:storefile table cannot be initialized and be online
+ */
+ public static void init(MasterServices masterServices) throws IOException {
+ createStorefileTable(masterServices);
+ waitForStoreFileTableOnline(masterServices);
+ }
+
+ /**
+ * Cleans up all storefile related state on the cluster. disable and delete hbase:storefile
+ * if found
+ * @param masterServices {@link MasterServices}
+ * @throws IOException if failures
+ */
+ private static void cleanup(MasterServices masterServices) throws IOException {
Review comment:
According to IntelliJ this method is not used.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathUpdate.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.List;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.fs.Path;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+@InterfaceAudience.Private
+final class StoreFilePathUpdate {
+
+ private final List<Path> storeFiles;
+ private final boolean hasStoreFilesUpdate;
+
+ private StoreFilePathUpdate(final List<Path> storeFiles, boolean hasStoreFilesUpdate) {
+ Preconditions.checkArgument(hasStoreFilesUpdate,
+ "StoreFilePathUpdate must include an update");
+ Preconditions.checkNotNull(storeFiles, "StoreFiles cannot be null");
+ if (hasStoreFilesUpdate) {
+ Preconditions
+ .checkArgument(CollectionUtils.isNotEmpty(storeFiles), "StoreFilePaths cannot be empty");
+ }
+ this.storeFiles = storeFiles;
+ this.hasStoreFilesUpdate = hasStoreFilesUpdate;
+ }
+
+ List<Path> getStoreFiles() {
+ return storeFiles;
+ }
+
+ boolean hasStoreFilesUpdate() {
+ return hasStoreFilesUpdate;
+ }
+
+ @Override
+ public String toString() {
+ return "StoreFilePathUpdate{" + "storeFiles=" + storeFiles + ", hasStoreFilesUpdate="
+ + hasStoreFilesUpdate + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ StoreFilePathUpdate that = (StoreFilePathUpdate) o;
+
+ return new EqualsBuilder().append(hasStoreFilesUpdate, that.hasStoreFilesUpdate)
+ .append(storeFiles, that.storeFiles).isEquals();
+ }
+
+ @Override
+ public int hashCode() {
+ return new HashCodeBuilder(17, 37).append(storeFiles)
+ .append(hasStoreFilesUpdate).toHashCode();
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ static class Builder {
+ private List<Path> storeFiles = ImmutableList.of();
+ private boolean hasStoreFilesUpdate;
+
+ Builder withStoreFiles(List<HStoreFile> storeFiles) {
+ Preconditions.checkArgument(!hasStoreFilesUpdate,
+ "Specify a Path List or File List, but not both");
+ this.storeFiles = StorefileTrackingUtils.convertStoreFilesToPaths(storeFiles);
+ this.hasStoreFilesUpdate = true;
+ return this;
+ }
+
+ Builder withStorePaths(List<Path> storeFiles) {
Review comment:
Testing only?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
+
+ /**
+ * GET storefile paths from the 'included' data set
+ * @param tableName name of the current table in String
+ * @param regionName name of the current region in String
+ * @param storeName name of the column family in String, to be combined with regionName to make
+ * the row key.
+ * @return list of StoreFile paths that should be included in reads in this store,
+ * returns an empty list if the target cell is empty or doesn't exist.
+ * @throws IOException if a remote or network exception occurs during Get
+ */
+ List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException;
+
+ /**
+ * Writes the specified updates to the tracking
Review comment:
In my understanding `StoreFilePathUpdate` represents a single entity which should be persisted by the Accessor. This could happen is some cases like results of a compaction, flush, recovered HFile, etc.
Essentially list of Paths decorated with `hasStoreFilesUpdate()` property which I don't fully understand what it's good for.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585171852
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathAccessor.java
##########
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Helper class to interact with the hbase storefile tracking data persisted as off-memory data
+ * from the {@link StoreFileManager}
+ *
+ * There is only a set of tracking storefiles, 'included'.
+ *
+ * e.g. list of storefile paths in 'included' should be the identical copy of the in-memory
+ * {@link HStoreFile}'s Path(s) and can be reused during region opens and region reassignment.
+ */
+@InterfaceAudience.Private
+public interface StoreFilePathAccessor {
+
+ /**
+ * Create the storefile tracking with the help of using the masterService
+ * @param masterServices instance of HMaster
+ * @throws IOException if Master is not running or connection has been lost
+ */
+ void initialize(final MasterServices masterServices) throws IOException;
Review comment:
yes, but after rethinking about along with `StorefileTrackingUtils.init(masterService);` we can remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-797487494
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 25s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 18s | HBASE-24749 passed |
| +1 :green_heart: | compile | 1m 32s | HBASE-24749 passed |
| +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | HBASE-24749 passed |
| -0 :warning: | patch | 8m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 30s | the patch passed |
| +1 :green_heart: | javac | 1m 30s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 142m 7s | hbase-server in the patch passed. |
| | | 177m 16s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 19e47e870020 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-11.0.6+10 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/testReport/ |
| Max. process+thread count | 3958 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/9/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-790124836
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24749 Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 56s | HBASE-24749 passed |
| +1 :green_heart: | compile | 4m 6s | HBASE-24749 passed |
| +1 :green_heart: | checkstyle | 1m 38s | HBASE-24749 passed |
| +1 :green_heart: | spotbugs | 2m 47s | HBASE-24749 passed |
| -0 :warning: | patch | 2m 20s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 56s | the patch passed |
| +1 :green_heart: | compile | 4m 6s | the patch passed |
| +1 :green_heart: | javac | 4m 6s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 25s | The patch passed checkstyle in hbase-common |
| +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 107 unchanged - 2 fixed = 107 total (was 109) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 3s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 11s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| | | 54m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2931 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 496df7cd78bf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24749 / 304379605a |
| Default Java | AdoptOpenJDK-1.8.0_232-b09 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2931/7/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585247304
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFilePathUpdate.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.List;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.fs.Path;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+@InterfaceAudience.Private
+final class StoreFilePathUpdate {
+
+ private final List<Path> storeFiles;
+ private final boolean hasStoreFilesUpdate;
+
+ private StoreFilePathUpdate(final List<Path> storeFiles, boolean hasStoreFilesUpdate) {
+ Preconditions.checkArgument(hasStoreFilesUpdate,
+ "StoreFilePathUpdate must include an update");
+ Preconditions.checkNotNull(storeFiles, "StoreFiles cannot be null");
+ if (hasStoreFilesUpdate) {
+ Preconditions
+ .checkArgument(CollectionUtils.isNotEmpty(storeFiles), "StoreFilePaths cannot be empty");
+ }
+ this.storeFiles = storeFiles;
+ this.hasStoreFilesUpdate = hasStoreFilesUpdate;
+ }
+
+ List<Path> getStoreFiles() {
+ return storeFiles;
+ }
+
+ boolean hasStoreFilesUpdate() {
+ return hasStoreFilesUpdate;
+ }
+
+ @Override
+ public String toString() {
+ return "StoreFilePathUpdate{" + "storeFiles=" + storeFiles + ", hasStoreFilesUpdate="
+ + hasStoreFilesUpdate + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ StoreFilePathUpdate that = (StoreFilePathUpdate) o;
+
+ return new EqualsBuilder().append(hasStoreFilesUpdate, that.hasStoreFilesUpdate)
+ .append(storeFiles, that.storeFiles).isEquals();
+ }
+
+ @Override
+ public int hashCode() {
+ return new HashCodeBuilder(17, 37).append(storeFiles)
+ .append(hasStoreFilesUpdate).toHashCode();
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ static class Builder {
+ private List<Path> storeFiles = ImmutableList.of();
+ private boolean hasStoreFilesUpdate;
+
+ Builder withStoreFiles(List<HStoreFile> storeFiles) {
+ Preconditions.checkArgument(!hasStoreFilesUpdate,
+ "Specify a Path List or File List, but not both");
+ this.storeFiles = StorefileTrackingUtils.convertStoreFilesToPaths(storeFiles);
+ this.hasStoreFilesUpdate = true;
+ return this;
+ }
+
+ Builder withStorePaths(List<Path> storeFiles) {
Review comment:
fixed with using `@RestrictedApi`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585248617
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StorefileTrackingUtils.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to support persistent storefile tracking
+ */
+@InterfaceAudience.Private
+public final class StorefileTrackingUtils {
+
+ private static Logger LOG = LoggerFactory.getLogger(StorefileTrackingUtils.class);
+ public static final long SLEEP_DELTA_MS = TimeUnit.MILLISECONDS.toMillis(100);
Review comment:
this need to used in HMaster, we can only use it via `public`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585247769
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableSortedSet;
+import org.apache.hbase.thirdparty.com.google.common.collect.Iterables;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private StoreFilePathAccessor accessor;
+ private Configuration conf;
+ // only uses for warmupHRegion
+ private boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ /**
+ * Loads the specified storeFiles to the StoreFileManager to be included in reads.
+ *
+ * @param storeFiles list of storefiles to be loaded, could be an empty list. throws exception
+ * if it's null.
+ */
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ ImmutableList<HStoreFile> sortedStorefiles =
+ ImmutableList.sortedCopyOf(getStoreFileComparator(), storeFiles);
+ setStorefiles(sortedStorefiles);
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(sortedStorefiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
+ if (isEmptyInPersistedFilePaths) {
+ // When the path accessor is returning empty result, we scan the
+ // the file storage and see if there is any existing HFiles should be loaded.
+ // the scan is a one time process when store open during region assignment.
+ //
+ // this is especially used for region and store open
+ // 1. First time migration from a filesystem based e.g. DefaultStoreFileEngine
+ // 2. After region split and merge
+ // 3. After table clone and create new HFiles directly into data directory
+ //
+ // Also we don't handle the inconsistency between storefile tracking and file system, which
+ // will be handled by a HBCK command
+ LOG.info("Cannot find tracking paths ({}) for store {} in region {} of "
+ + "table {}, fall back to scan the storage to get a list of storefiles to be opened"
+ , isEmptyInPersistedFilePaths, storeName, regionName,
+ tableName);
+ return getRegionFs().getStoreFiles(getFamilyName());
+ }
+ ArrayList<StoreFileInfo> storeFiles = new ArrayList<>();
+ for (Path storeFilePath : pathList) {
+ if (!StoreFileInfo.isValid(getRegionFs().getFileSystem().getFileStatus(storeFilePath))) {
+ // TODO add a comment how this file will be removed when we have cleaner
+ LOG.warn("Invalid StoreFile: {}", storeFilePath);
+ continue;
+ }
+ StoreFileInfo info = ServerRegionReplicaUtil
+ .getStoreFileInfo(conf, getRegionFs().getFileSystem(), regionInfo,
+ ServerRegionReplicaUtil.getRegionInfoForFs(regionInfo), getFamilyName(),
+ storeFilePath);
+ storeFiles.add(info);
+ }
+ return storeFiles;
+ }
+
+
+ @Override
+ public void insertNewFiles(Collection<HStoreFile> sfs) throws IOException {
+ // return in case of empty storeFiles as it is a No-op, empty files expected during region close
+ if (CollectionUtils.isEmpty(sfs)) {
+ return;
+ }
+ ImmutableList<HStoreFile> storefiles = ImmutableList
+ .sortedCopyOf(getStoreFileComparator(), Iterables.concat(getStorefiles(), sfs));
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storefiles).build());
+ setStorefiles(storefiles);
+ }
+
+ @Override
+ public void addCompactionResults(Collection<HStoreFile> newCompactedfiles,
Review comment:
fixed and refactored to use a abstract base class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#issuecomment-790220164
I checked locally with the jdk8 environment and it passed. I checked the code of the unit test `grantGlobal` should not be related since this should be using `DefaultStoreFileManager`
```
[ERROR] Failures:
[ERROR] TestRSGroupsWithACL.setupBeforeClass:135->SecureTestUtil.grantGlobal:345->SecureTestUtil.updateACLs:313 Waiting timed out after [10,000] msec
```
* local test result from a jdk8 build
```
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.645 s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anmolnar commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
anmolnar commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r577795682
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractStoreFilePathAccessor.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.base.Joiner;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+
+@InterfaceAudience.Private
+public abstract class AbstractStoreFilePathAccessor implements StoreFilePathAccessor {
+
+ public static final String STOREFILE_INCLUDED_STR = "included";
+
+ protected static final String LIST_SEPARATOR = ";";
+ protected final Configuration conf;
+
+ public AbstractStoreFilePathAccessor(Configuration conf) {
+ this.conf = conf;
+ }
+
+ abstract String getSeparator();
+
+ abstract List<Path> getStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, final String columnName) throws IOException;
+
+ @Override
+ public abstract void writeStoreFilePaths(final String tableName, final String regionName,
+ final String storeName, StoreFilePathUpdate storeFilePathUpdate)
+ throws IOException;
+
+ @Override
+ public List<Path> getIncludedStoreFilePaths(final String tableName, final String regionName,
+ final String storeName) throws IOException {
+ return getStoreFilePaths(tableName, regionName, storeName, STOREFILE_INCLUDED_STR);
+ }
+
+ protected static byte[] storeFileListToByteArray(List<Path> storeFilePaths) {
+ return Bytes.toBytes(Joiner.on(LIST_SEPARATOR).join(storeFilePaths));
+ }
+
+ protected static List<Path> byteToStoreFileList(byte[] data) {
+ List<Path> paths = new ArrayList<>();
+ if (data != null && data.length != 0) {
+ String pathString = Bytes.toString(data);
+ String[] pathStrings = pathString.split(LIST_SEPARATOR);
Review comment:
So, StringTokenizer would be the best?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #2931:
URL: https://github.com/apache/hbase/pull/2931#discussion_r585167490
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PersistedStoreFileManager.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableSortedSet;
+import org.apache.hbase.thirdparty.com.google.common.collect.Iterables;
+import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
+
+/**
+ * A Storefile manager that is used by {@link PersistedStoreEngine} that persists the in-memory
+ * storefile tracking to a persistent table hbase:storefile.
+ *
+ * We don't override the {@link #clearFiles()} from {@link DefaultStoreFileManager} and persist
+ * in-memory storefiles tracking, it will be reused when region reassigns on a different
+ * region server.
+ */
+@InterfaceAudience.Private
+public class PersistedStoreFileManager extends DefaultStoreFileManager {
+ private static final Logger LOG = LoggerFactory.getLogger(PersistedStoreFileManager.class);
+ private final RegionInfo regionInfo;
+ private final String tableName;
+ private final String regionName;
+ private final String storeName;
+ private StoreFilePathAccessor accessor;
+ private Configuration conf;
+ // only uses for warmupHRegion
+ private boolean readOnly;
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor, boolean readOnly) {
+ super(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, familyName);
+ this.conf = conf;
+ this.regionInfo = regionInfo;
+ this.tableName = regionInfo.getTable().getNameAsString();
+ this.regionName = regionInfo.getEncodedName();
+ this.storeName = familyName;
+ this.accessor = accessor;
+ this.readOnly = readOnly;
+ }
+
+ public PersistedStoreFileManager(CellComparator cellComparator,
+ Comparator<HStoreFile> storeFileComparator, Configuration conf,
+ CompactionConfiguration compactionConfiguration, HRegionFileSystem regionFs,
+ RegionInfo regionInfo, String familyName, StoreFilePathAccessor accessor) {
+ this(cellComparator, storeFileComparator, conf, compactionConfiguration, regionFs, regionInfo,
+ familyName, accessor, false);
+ }
+
+ /**
+ * Loads the specified storeFiles to the StoreFileManager to be included in reads.
+ *
+ * @param storeFiles list of storefiles to be loaded, could be an empty list. throws exception
+ * if it's null.
+ */
+ @Override
+ public void loadFiles(List<HStoreFile> storeFiles) throws IOException {
+ Preconditions.checkArgument(storeFiles != null, "store files cannot be "
+ + "null when loading");
+ if (storeFiles.isEmpty()) {
+ LOG.warn("Other than fresh region with no store files, store files should not be empty");
+ return;
+ }
+ ImmutableList<HStoreFile> sortedStorefiles =
+ ImmutableList.sortedCopyOf(getStoreFileComparator(), storeFiles);
+ setStorefiles(sortedStorefiles);
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(sortedStorefiles).build());
+ }
+
+ @Override
+ public Collection<StoreFileInfo> loadInitialFiles() throws IOException {
+ List<Path> pathList = accessor.getIncludedStoreFilePaths(tableName, regionName, storeName);
+ boolean isEmptyInPersistedFilePaths = CollectionUtils.isEmpty(pathList);
+ if (isEmptyInPersistedFilePaths) {
+ // When the path accessor is returning empty result, we scan the
+ // the file storage and see if there is any existing HFiles should be loaded.
+ // the scan is a one time process when store open during region assignment.
+ //
+ // this is especially used for region and store open
+ // 1. First time migration from a filesystem based e.g. DefaultStoreFileEngine
+ // 2. After region split and merge
+ // 3. After table clone and create new HFiles directly into data directory
+ //
+ // Also we don't handle the inconsistency between storefile tracking and file system, which
+ // will be handled by a HBCK command
+ LOG.info("Cannot find tracking paths ({}) for store {} in region {} of "
+ + "table {}, fall back to scan the storage to get a list of storefiles to be opened"
+ , isEmptyInPersistedFilePaths, storeName, regionName,
+ tableName);
+ return getRegionFs().getStoreFiles(getFamilyName());
+ }
+ ArrayList<StoreFileInfo> storeFiles = new ArrayList<>();
+ for (Path storeFilePath : pathList) {
+ if (!StoreFileInfo.isValid(getRegionFs().getFileSystem().getFileStatus(storeFilePath))) {
+ // TODO add a comment how this file will be removed when we have cleaner
+ LOG.warn("Invalid StoreFile: {}", storeFilePath);
+ continue;
+ }
+ StoreFileInfo info = ServerRegionReplicaUtil
+ .getStoreFileInfo(conf, getRegionFs().getFileSystem(), regionInfo,
+ ServerRegionReplicaUtil.getRegionInfoForFs(regionInfo), getFamilyName(),
+ storeFilePath);
+ storeFiles.add(info);
+ }
+ return storeFiles;
+ }
+
+
+ @Override
+ public void insertNewFiles(Collection<HStoreFile> sfs) throws IOException {
+ // return in case of empty storeFiles as it is a No-op, empty files expected during region close
+ if (CollectionUtils.isEmpty(sfs)) {
+ return;
+ }
+ ImmutableList<HStoreFile> storefiles = ImmutableList
+ .sortedCopyOf(getStoreFileComparator(), Iterables.concat(getStorefiles(), sfs));
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storefiles).build());
+ setStorefiles(storefiles);
+ }
+
+ @Override
+ public void addCompactionResults(Collection<HStoreFile> newCompactedfiles,
+ Collection<HStoreFile> results) throws IOException {
+ Preconditions.checkNotNull(newCompactedfiles, "compactedFiles cannot be null");
+ Preconditions.checkNotNull(results, "compaction result cannot be null");
+ // only allow distinct path to be included, especially rerun after a compaction fails
+ ImmutableList<HStoreFile> storefiles = ImmutableList.sortedCopyOf(getStoreFileComparator(),
+ Iterables.concat(Iterables.filter(getStorefiles(), sf -> !newCompactedfiles.contains(sf)),
+ results)).asList();
+ Preconditions.checkArgument(!CollectionUtils.isEmpty(storefiles),
+ "storefiles cannot be empty when adding compaction results");
+
+ ImmutableList<HStoreFile> compactedfiles = ImmutableSortedSet
+ .copyOf(getStoreFileComparator(),
+ Iterables.concat(getCompactedfiles(), newCompactedfiles))
+ .asList();
+ updatePathListToTracker(StoreFilePathUpdate.builder().withStoreFiles(storefiles).build());
+ setStorefiles(storefiles);
+ setCompactedfiles(compactedfiles);
+ newCompactedfiles.forEach(HStoreFile::markCompactedAway);
+ }
+
+ void updatePathListToTracker(StoreFilePathUpdate storeFilePathUpdate) throws IOException {
+ try {
+ if (!readOnly) {
+ accessor.writeStoreFilePaths(tableName, regionName, storeName, storeFilePathUpdate);
+ }
+ } catch (IOException e) {
+ String message = "Failed to update Path list of " + tableName + "-" + regionName +
+ "-" + storeName + ", on " + TableName.STOREFILE_STR + ". The new files are not "
+ + "persistent and will be removed from " + regionName + "," + storeName +
+ ".\nFailed update: " + storeFilePathUpdate;
+ LOG.warn(message);
Review comment:
:( sure (FYI we don't allow to use String.format because of performance concern)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org