You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "2005hithlj (via GitHub)" <gi...@apache.org> on 2023/04/26 15:17:28 UTC

[GitHub] [hbase] 2005hithlj opened a new pull request, #5203: HBASE-27516 Document the table based replication queue storage in ref guide

2005hithlj opened a new pull request, #5203:
URL: https://github.com/apache/hbase/pull/5203

   https://issues.apache.org/jira/browse/HBASE-27516


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1523642094

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |  11m 52s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |  12m 52s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux e2eb5704e152 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / b5535c98a3 |
   | Max. process+thread count | 29 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/1/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] 2005hithlj commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "2005hithlj (via GitHub)" <gi...@apache.org>.
2005hithlj commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1535640666

   @Apache9  sir. Could you take a look? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1536993088

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 57s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  2s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 59s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux f7128f90fe94 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 28 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/6/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache9 commented on a diff in pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #5203:
URL: https://github.com/apache/hbase/pull/5203#discussion_r1185751247


##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2454,6 +2450,12 @@ The `RS` Znode::
   The child znode name is the region server's hostname, client port, and start code.
   This list includes both live and dead region servers.
 
+[[hbase:replication]]
+hbase:replication::

Review Comment:
   Use "The `hbase:replication` Table"?



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2429,17 +2429,13 @@ This option was introduced in link:https://issues.apache.org/jira/browse/HBASE-1
 ==== Replication Internals
 
 Replication State Storage::
-  In HBASE-15867, we abstract two interfaces for storing replication state,
-`ReplicationPeerStorage` and `ReplicationQueueStorage`. The former one is for storing the
-replication peer related states, and the latter one is for storing the replication queue related
-states.
-  HBASE-15867 is only half done, as although we have abstract these two interfaces, we still only

Review Comment:
   The paragragh before here should be kept?



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2519,12 +2533,12 @@ The next time the cleaning process needs to look for a log, it starts by using i
 NOTE: WALs are saved when replication is enabled or disabled as long as peers exist.
 
 [[rs.failover.details]]
-==== Region Server Failover
+==== Region Server Failover(based on ZooKeeper)

Review Comment:
   Here we do not need to se two different sections. First, we could mention the 'setting a watcher' way, this is way in the old time.
   
   And starting from 2.5.0, the failover logic has been moved to SCP, where we add a `SERVER_CRASH_CLAIM_REPLICATION_QUEUES` step in SCP to claim the replication queues for a dead server.
   
   And starting from 3.0.0, where we changed the replication queue storage from zookeeper to table, the update to the replication queue storage is async, so we also need an extra step to add the missing replication queues before claiming.
   
   And on how to claim the replication queue, you can have two sections, to describe the layout and claiming way for zookeeper based implementation and table based implementation.



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2494,6 +2496,18 @@ If the log is in the queue, the path will be updated in memory.
 If the log is currently being replicated, the change will be done atomically so that the reader doesn't attempt to open the file when has already been moved.
 Because moving a file is a NameNode operation , if the reader is currently reading the log, it won't generate any exception.
 
+==== Keeping Track of Logs(based on hbase table)
+
+After 3.0.0, for table based implementation, we have server name in row key, which means we will have lots of rows for a given peer.
+
+For a normal replication queue, where the WAL files belong to it is still alive, all the WAL files are kept in memory, so we do not need to get the WAL files from replication queue storage.

Review Comment:
   "the region server is still alive"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1535107220

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m  0s |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  3s |  https://github.com/apache/hbase/pull/5203 does not apply to HBASE-27109/table_based_rqs. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/3/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1536491180

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 25s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   | +1 :green_heart: |  spotless  |   0m 41s |  branch has no errors when running spotless:check.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  spotless  |   0m 35s |  patch has no errors when running spotless:check.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m  9s |  The patch does not generate ASF License warnings.  |
   |  |   |   2m 59s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/5/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests | dupname asflicense spotless |
   | uname | Linux 0f03b6ea6554 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 43 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/5/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1535107142

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m  0s |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  6s |  https://github.com/apache/hbase/pull/5203 does not apply to HBASE-27109/table_based_rqs. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/3/console |
   | versions | git=2.25.1 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] 2005hithlj commented on a diff in pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "2005hithlj (via GitHub)" <gi...@apache.org>.
2005hithlj commented on code in PR #5203:
URL: https://github.com/apache/hbase/pull/5203#discussion_r1186684616


##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2610,6 +2636,32 @@ The new layout will be:
       1.1.1.2,60020.1312  (Contains a position)
 ----
 
+==== The replication queue claiming(based on hbase table)
+
+Given a master cluster with 3 region servers replicating to a single slave with id `2`, the following info represents what the storage layout of queue in the hbase:replication at some point in time.
+Row key is <PeerId>-<ServerName>[/<SourceServerName>], and value is WAL && Offset.
+
+----
+
+  <PeerId>-<ServerName>[/<SourceServerName>]                        WAL && Offset
+  2-1.1.1.1,60020,123456780                            1.1.1.1,60020.1234  (Contains a position)
+  2-1.1.1.2,60020,123456790                            1.1.1.2,60020.1214  (Contains a position)
+  2-1.1.1.3,60020,123456630                            1.1.1.3,60020.1280  (Contains a position)
+----
+
+Assume that 1.1.1.2 failed.
+The survivors will claim queue of that, and, arbitrarily, 1.1.1.3 wins.
+It will claim all the queue of 1.1.1.2, including removing the row of a replication queue, and inserting a new row(where we change the server name to the region server which claims the queue).
+Finally ,the layout will look like the following:

Review Comment:
   > The position of the ',' is incorrect?
   
   OK  sir , I have revised.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1537125076

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 59s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  2s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   2m  1s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux 3938f785b85e 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 31 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/7/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1537125236

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 26s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   | +1 :green_heart: |  spotless  |   0m 35s |  branch has no errors when running spotless:check.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  spotless  |   0m 36s |  patch has no errors when running spotless:check.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m  8s |  The patch does not generate ASF License warnings.  |
   |  |   |   2m 51s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/7/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests | dupname asflicense spotless |
   | uname | Linux cc33b6567d4f 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 43 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/7/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1525638504

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 28s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   | +1 :green_heart: |  spotless  |   0m 49s |  branch has no errors when running spotless:check.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  spotless  |   0m 40s |  patch has no errors when running spotless:check.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 12s |  The patch does not generate ASF License warnings.  |
   |  |   |   3m 18s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/2/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests | dupname asflicense spotless |
   | uname | Linux fa66fefa6c04 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / b1a7069fd9 |
   | Max. process+thread count | 43 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/2/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1523623420

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 20s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 27s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux 3d0519aed80e 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / b5535c98a3 |
   | Max. process+thread count | 37 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/1/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1535646262

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 25s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   | +1 :green_heart: |  spotless  |   0m 50s |  branch has no errors when running spotless:check.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  spotless  |   0m 41s |  patch has no errors when running spotless:check.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 11s |  The patch does not generate ASF License warnings.  |
   |  |   |   3m 18s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/4/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests | dupname asflicense spotless |
   | uname | Linux 37cd7847c84f 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 44 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/4/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1535645686

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 53s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 57s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux 9a4f8c678f84 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 28 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/4/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache9 merged pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 merged PR #5203:
URL: https://github.com/apache/hbase/pull/5203


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1535645533

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 26s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  2s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 35s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux 552f1c6e7f83 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 39 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/4/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1536994935

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 26s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   | +1 :green_heart: |  spotless  |   0m 43s |  branch has no errors when running spotless:check.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  spotless  |   0m 41s |  patch has no errors when running spotless:check.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m  8s |  The patch does not generate ASF License warnings.  |
   |  |   |   3m  6s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/6/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests | dupname asflicense spotless |
   | uname | Linux 5e2d206af3b0 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 44 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/6/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache9 commented on a diff in pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #5203:
URL: https://github.com/apache/hbase/pull/5203#discussion_r1179173386


##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2433,26 +2433,22 @@ Replication State Storage::
 `ReplicationPeerStorage` and `ReplicationQueueStorage`. The former one is for storing the
 replication peer related states, and the latter one is for storing the replication queue related
 states.
-  HBASE-15867 is only half done, as although we have abstract these two interfaces, we still only
-have zookeeper based implementations.
+  And in HBASE-27109, we have implemented the `ReplicationQueueStorage` interface to store the replication queue in the hbase:replication table.
 
 Replication State in ZooKeeper::
   By default, the state is contained in the base node _/hbase/replication_.
-  Usually this nodes contains two child nodes, the `peers` znode is for storing replication peer
-state, and the `rs` znodes is for storing replication queue state.
+  After 3.0.0, it only contains one child node, but before 3.0.0, we still use zk to store queue data.
 
 The `Peers` Znode::
   The `peers` znode is stored in _/hbase/replication/peers_ by default.
   It consists of a list of all peer replication clusters, along with the status of each of them.
   The value of each peer is its cluster key, which is provided in the HBase Shell.
   The cluster key contains a list of ZooKeeper nodes in the cluster's quorum, the client port for the ZooKeeper quorum, and the base znode for HBase in HDFS on that cluster.
 
-The `RS` Znode::

Review Comment:
   We'd better keep this unchanged, as it describes what we have before 3.0.0. And we can introduce a new section to describe the hbase:replication table storage.



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2433,26 +2433,22 @@ Replication State Storage::
 `ReplicationPeerStorage` and `ReplicationQueueStorage`. The former one is for storing the
 replication peer related states, and the latter one is for storing the replication queue related
 states.
-  HBASE-15867 is only half done, as although we have abstract these two interfaces, we still only
-have zookeeper based implementations.
+  And in HBASE-27109, we have implemented the `ReplicationQueueStorage` interface to store the replication queue in the hbase:replication table.
 
 Replication State in ZooKeeper::
   By default, the state is contained in the base node _/hbase/replication_.
-  Usually this nodes contains two child nodes, the `peers` znode is for storing replication peer
-state, and the `rs` znodes is for storing replication queue state.
+  After 3.0.0, it only contains one child node, but before 3.0.0, we still use zk to store queue data.

Review Comment:
   "Usually this nodes contains two child nodes, the `peers` znode is for storing replication peer state, and the `rs` znodes is for storing replication queue state. And if you choose the file system based replication peer storage, you will not see the `peers` znode. And starting from 3.0.0, we have moved the replication queue state to hbase:replication table, so you will not see the `rs` znode."



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2433,26 +2433,22 @@ Replication State Storage::
 `ReplicationPeerStorage` and `ReplicationQueueStorage`. The former one is for storing the
 replication peer related states, and the latter one is for storing the replication queue related
 states.
-  HBASE-15867 is only half done, as although we have abstract these two interfaces, we still only
-have zookeeper based implementations.
+  And in HBASE-27109, we have implemented the `ReplicationQueueStorage` interface to store the replication queue in the hbase:replication table.

Review Comment:
   And in HBASE-27110, we have implemented a file system based replication peer storage, to store replication peer state on file system. Of course you can still use the zookeeper based replication peer storage.
   
   And in HBASE-27109, we have changed the replication queue storage from zookeeper based to hbase table based. See the below 'Replication Queue State in hbase:replication table' section for more details.



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2475,14 +2471,14 @@ When nodes are removed from the slave cluster, or if nodes go down or come back
 
 ==== Keeping Track of Logs
 
-Each master cluster region server has its own znode in the replication znodes hierarchy.
-It contains one znode per peer cluster (if 5 slave clusters, 5 znodes are created), and each of these contain a queue of WALs to process.
+Before 3.0.0, for zookeeper based implementation, it is like a tree, we have a znode for a peer cluster, but under the znode we have lots of files.

Review Comment:
   I think here we'd better make two different sections to describe the logic before and after 3.0.0. As on zookeeper, we store all WAL files on it and for table based solution, we only store an offset.



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2475,14 +2471,14 @@ When nodes are removed from the slave cluster, or if nodes go down or come back
 
 ==== Keeping Track of Logs
 
-Each master cluster region server has its own znode in the replication znodes hierarchy.
-It contains one znode per peer cluster (if 5 slave clusters, 5 znodes are created), and each of these contain a queue of WALs to process.
+Before 3.0.0, for zookeeper based implementation, it is like a tree, we have a znode for a peer cluster, but under the znode we have lots of files.
+But after 3.0.0, for table based implementation, we have server name in row key, which means we will have lots of rows for a given peer.
 Each of these queues will track the WALs created by that region server, but they can differ in size.
 For example, if one slave cluster becomes unavailable for some time, the WALs should not be deleted, so they need to stay in the queue while the others are processed.
 See <<rs.failover.details,rs.failover.details>> for an example.
 
 When a source is instantiated, it contains the current WAL that the region server is writing to.
-During log rolling, the new file is added to the queue of each slave cluster's znode just before it is made available.
+During log rolling, the new file is added to the queue of each slave cluster's record just before it is made available.

Review Comment:
   This is different for table based replication queue storage, and it is the key point here. For zookeeper, it is an external system so there is no problem to let log rolling depend on it, but if we want to store the state in a hbase table, we can not let log rolling depend on it as it will introduce dead lock...
   We will only write to hbase:replication when want to record an offset after replicating something.



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2521,93 +2517,36 @@ NOTE: WALs are saved when replication is enabled or disabled as long as peers ex
 [[rs.failover.details]]
 ==== Region Server Failover
 
-When no region servers are failing, keeping track of the logs in ZooKeeper adds no value.
-Unfortunately, region servers do fail, and since ZooKeeper is highly available, it is useful for managing the transfer of the queues in the event of a failure.
-
-Each of the master cluster region servers keeps a watcher on every other region server, in order to be notified when one dies (just as the master does). When a failure happens, they all race to create a znode called `lock` inside the dead region server's znode that contains its queues.

Review Comment:
   I think we still need to keep this, as it is the logic for some hbase releases. Let me check the version where we start to use SCP to claim replication queue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1525635669

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 21s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 23s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux ed39ece159a4 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / b1a7069fd9 |
   | Max. process+thread count | 37 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/2/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache9 commented on a diff in pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #5203:
URL: https://github.com/apache/hbase/pull/5203#discussion_r1178051368


##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2521,93 +2517,38 @@ NOTE: WALs are saved when replication is enabled or disabled as long as peers ex
 [[rs.failover.details]]
 ==== Region Server Failover
 
-When no region servers are failing, keeping track of the logs in ZooKeeper adds no value.
-Unfortunately, region servers do fail, and since ZooKeeper is highly available, it is useful for managing the transfer of the queues in the event of a failure.
+When no region servers are failing, keeping track of the logs in hbase:replication table adds no value.
+However, in case of region server failure, we will manage the transfer of the queues based on hbase:replication.
 
-Each of the master cluster region servers keeps a watcher on every other region server, in order to be notified when one dies (just as the master does). When a failure happens, they all race to create a znode called `lock` inside the dead region server's znode that contains its queues.
-The region server that creates it successfully then transfers all the queues to its own znode, one at a time since ZooKeeper does not support renaming queues.
-After queues are all transferred, they are deleted from the old location.
-The znodes that were recovered are renamed with the ID of the slave cluster appended with the name of the dead server.
+When a region server fails, the HMaster of master cluster will trigger the SCP, and all replication queues on the failed region server will be claimed in the SCP.

Review Comment:
   OK, I think the current ref guide also need to be updated...
   
   I'm not sure which is the accruate version, but starting from a minor release for 2.x, we do not watch on the region server's znode any more, we use SCP to schedule claim queue operations...



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2521,93 +2517,38 @@ NOTE: WALs are saved when replication is enabled or disabled as long as peers ex
 [[rs.failover.details]]
 ==== Region Server Failover
 
-When no region servers are failing, keeping track of the logs in ZooKeeper adds no value.
-Unfortunately, region servers do fail, and since ZooKeeper is highly available, it is useful for managing the transfer of the queues in the event of a failure.
+When no region servers are failing, keeping track of the logs in hbase:replication table adds no value.
+However, in case of region server failure, we will manage the transfer of the queues based on hbase:replication.
 
-Each of the master cluster region servers keeps a watcher on every other region server, in order to be notified when one dies (just as the master does). When a failure happens, they all race to create a znode called `lock` inside the dead region server's znode that contains its queues.
-The region server that creates it successfully then transfers all the queues to its own znode, one at a time since ZooKeeper does not support renaming queues.
-After queues are all transferred, they are deleted from the old location.
-The znodes that were recovered are renamed with the ID of the slave cluster appended with the name of the dead server.
+When a region server fails, the HMaster of master cluster will trigger the SCP, and all replication queues on the failed region server will be claimed in the SCP.
+The claim queue operation is just to remove the row of a replication queue, and insert a new row, where we change the server name to the region server which claims the queue.

Review Comment:
   We need to mention that we use multi row mutate endpoint here, so the data for a single peer must be in the same region.



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2475,14 +2471,14 @@ When nodes are removed from the slave cluster, or if nodes go down or come back
 
 ==== Keeping Track of Logs
 
-Each master cluster region server has its own znode in the replication znodes hierarchy.
-It contains one znode per peer cluster (if 5 slave clusters, 5 znodes are created), and each of these contain a queue of WALs to process.
+Each master cluster region server has its queue state in the hbase:replication table.
+It contains one row per peer cluster (if 5 slave clusters, 5 rows are created), and each of these contain a queue of WALs to process.

Review Comment:
   Here things are a bit different. For zookeeper, it is like a tree, we have a znode for a peer cluster, but under the znode we have lots of files.
   But for table based implementation, we have server name in row key, which means we will have lots of rows for a given peer...



##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2433,26 +2433,22 @@ Replication State Storage::
 `ReplicationPeerStorage` and `ReplicationQueueStorage`. The former one is for storing the
 replication peer related states, and the latter one is for storing the replication queue related
 states.
-  HBASE-15867 is only half done, as although we have abstract these two interfaces, we still only
-have zookeeper based implementations.
+  And in HBASE-27109, we have implemented the `ReplicationQueueStorage` interface to store the replication queue in the hbase:replication table.
 
 Replication State in ZooKeeper::
   By default, the state is contained in the base node _/hbase/replication_.
-  Usually this nodes contains two child nodes, the `peers` znode is for storing replication peer
-state, and the `rs` znodes is for storing replication queue state.
+  Currently, this nodes contains only one child node, namely `peers` znode, which is used for storing replication peer state.

Review Comment:
   As now we only have one ref guide on master branch, we should include the doducmentation for all branches. So here I think we should mention that, after 3.0.0, it only contains one child node, but before 3.0.0, we still use zk to store queue data.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1536490348

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 58s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   2m  8s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux 8015d58a4466 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 28 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/5/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1536992000

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 20s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 20s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux fab387de22ed 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 36 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/6/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1537124987

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 28s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 31s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux ab42aee63ff2 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 37 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/7/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1523626663

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 28s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   | +1 :green_heart: |  spotless  |   0m 50s |  branch has no errors when running spotless:check.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  spotless  |   0m 35s |  patch has no errors when running spotless:check.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 11s |  The patch does not generate ASF License warnings.  |
   |  |   |   3m 17s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/1/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests | dupname asflicense spotless |
   | uname | Linux fcd1cb661364 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / b5535c98a3 |
   | Max. process+thread count | 44 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/1/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1525636684

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 55s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   2m  3s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux db91ecbf6949 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / b1a7069fd9 |
   | Max. process+thread count | 33 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/2/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1535106448

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m  0s |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  4s |  https://github.com/apache/hbase/pull/5203 does not apply to HBASE-27109/table_based_rqs. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/3/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache-HBase commented on pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #5203:
URL: https://github.com/apache/hbase/pull/5203#issuecomment-1536489910

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 22s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  2s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-27109/table_based_rqs Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   ||| _ Other Tests _ |
   |  |   |   1m 28s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/5203 |
   | Optional Tests |  |
   | uname | Linux 83233b2fae41 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-27109/table_based_rqs / 772acaaeea |
   | Max. process+thread count | 39 (vs. ulimit of 30000) |
   | modules | C: . U: . |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5203/5/console |
   | versions | git=2.34.1 maven=3.8.6 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hbase] Apache9 commented on a diff in pull request #5203: HBASE-27516 Document the table based replication queue storage in ref guide

Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #5203:
URL: https://github.com/apache/hbase/pull/5203#discussion_r1186682029


##########
src/main/asciidoc/_chapters/ops_mgt.adoc:
##########
@@ -2610,6 +2636,32 @@ The new layout will be:
       1.1.1.2,60020.1312  (Contains a position)
 ----
 
+==== The replication queue claiming(based on hbase table)
+
+Given a master cluster with 3 region servers replicating to a single slave with id `2`, the following info represents what the storage layout of queue in the hbase:replication at some point in time.
+Row key is <PeerId>-<ServerName>[/<SourceServerName>], and value is WAL && Offset.
+
+----
+
+  <PeerId>-<ServerName>[/<SourceServerName>]                        WAL && Offset
+  2-1.1.1.1,60020,123456780                            1.1.1.1,60020.1234  (Contains a position)
+  2-1.1.1.2,60020,123456790                            1.1.1.2,60020.1214  (Contains a position)
+  2-1.1.1.3,60020,123456630                            1.1.1.3,60020.1280  (Contains a position)
+----
+
+Assume that 1.1.1.2 failed.
+The survivors will claim queue of that, and, arbitrarily, 1.1.1.3 wins.
+It will claim all the queue of 1.1.1.2, including removing the row of a replication queue, and inserting a new row(where we change the server name to the region server which claims the queue).
+Finally ,the layout will look like the following:

Review Comment:
   The position of the ',' is incorrect?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org