You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/06/09 22:59:41 UTC
[GitHub] [hbase] ndimiduk opened a new pull request #1880: HBASE-24144 Update docs from master
ndimiduk opened a new pull request #1880:
URL: https://github.com/apache/hbase/pull/1880
Bring back documentation from master branch (9ef17c2784), using
```
$ git checkout master -- src/main/asciidoc/
$ git checkout master -- src/site/asciidoc/
```
Followed up with a commit to revert changes for synchronous replication, which is a feature only on master at this time. What other docs changes need dropped from this commit? I fear some content cannot be simply dropped because the language won't make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#discussion_r438429724
##########
File path: src/main/asciidoc/_chapters/architecture.adoc
##########
@@ -260,6 +260,73 @@ For region name, we only accept `byte[]` as the parameter type and it may be a f
Information on non-Java clients and custom protocols is covered in <<external_apis>>
+[[client.masterregistry]]
+=== Master registry (new as of release 3.0.0)
+
+Client internally works with a _connection registry_ to fetch the metadata needed by connections.
+This connection registry implementation is responsible for fetching the following metadata.
+
+* Active master address
+* Current meta region(s) locations
+* Cluster ID (unique to this cluster)
+
+This information is needed as a part of various client operations like connection set up, scans,
+gets etc. Up until releases 2.x.y, the default connection registry is based on ZooKeeper as the
+source of truth and the the clients fetched the metadata from zookeeper znodes. As of release 3.0.0,
+the default implementation for connection registry has been switched to a master based
+implementation. With this change, the clients now fetch the required metadata from master RPC end
+points directly. This change was done for the following reasons.
+
+* Reduce load on ZooKeeper since that is critical for cluster operation.
+* Holistic client timeout and retry configurations since the new registry brings all the client
+operations under HBase rpc framework.
+* Remove the ZooKeeper client dependency on HBase client library.
+
+This means that
+
+* At least a single active or stand by master is needed for cluster connection setup. Refer to
+<<master.runtime>> for more details.
+* Master can be in a critical path of read/write operations, especially if the client metadata cache
+is empty or stale.
+* There is higher connection load on the masters that before since the clients talk directly to
+HMasters instead of ZooKeeper ensemble`
+
+To reduce hot-spotting on a single master, all the masters (active & stand-by) expose the needed
+service to fetch the connection metadata. This lets the client connect to any master (not just active).
+
+==== RPC hedging
Review comment:
Okay, thank you. I started to do some word-smithing... my hope is that we have language in the docs on master that is version agnostic, covering all active release lines, so that it's simple as possible for release managers to copy them back.
You're the expert here, I'll let you handle it. Shout when you're ready for a review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-641626709
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 9s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 2m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux 3093744415de 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / b67f896954 |
| Max. process+thread count | 49 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-641627049
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 20s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 3m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux 48d35a65e3e1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / b67f896954 |
| Max. process+thread count | 47 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-642254366
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 40s | branch-2 passed |
| +1 :green_heart: | mvnsite | 16m 39s | branch-2 passed |
| +0 :ok: | refguide | 6m 46s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 15s | the patch passed |
| +1 :green_heart: | mvnsite | 12m 8s | the patch passed |
| -0 :warning: | whitespace | 0m 1s | The patch has 51 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +0 :ok: | refguide | 7m 10s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 52m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | dupname asflicense refguide mvnsite |
| uname | Linux ae0b5dfcdb45 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / ef984cb18e |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/artifact/yetus-general-check/output/branch-site/book.html |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/artifact/yetus-general-check/output/whitespace-eol.txt |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/artifact/yetus-general-check/output/patch-site/book.html |
| Max. process+thread count | 89 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-642277500
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 31s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 2m 58s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux e62937b9747d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / ef984cb18e |
| Max. process+thread count | 59 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-644332479
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 42s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 3m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux 4c14b9da37bc 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 035ea516e9 |
| Max. process+thread count | 41 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#discussion_r438402629
##########
File path: src/main/asciidoc/_chapters/architecture.adoc
##########
@@ -260,6 +260,73 @@ For region name, we only accept `byte[]` as the parameter type and it may be a f
Information on non-Java clients and custom protocols is covered in <<external_apis>>
+[[client.masterregistry]]
+=== Master registry (new as of release 3.0.0)
+
+Client internally works with a _connection registry_ to fetch the metadata needed by connections.
+This connection registry implementation is responsible for fetching the following metadata.
+
+* Active master address
+* Current meta region(s) locations
+* Cluster ID (unique to this cluster)
+
+This information is needed as a part of various client operations like connection set up, scans,
+gets etc. Up until releases 2.x.y, the default connection registry is based on ZooKeeper as the
+source of truth and the the clients fetched the metadata from zookeeper znodes. As of release 3.0.0,
+the default implementation for connection registry has been switched to a master based
+implementation. With this change, the clients now fetch the required metadata from master RPC end
+points directly. This change was done for the following reasons.
+
+* Reduce load on ZooKeeper since that is critical for cluster operation.
+* Holistic client timeout and retry configurations since the new registry brings all the client
+operations under HBase rpc framework.
+* Remove the ZooKeeper client dependency on HBase client library.
+
+This means that
+
+* At least a single active or stand by master is needed for cluster connection setup. Refer to
+<<master.runtime>> for more details.
+* Master can be in a critical path of read/write operations, especially if the client metadata cache
+is empty or stale.
+* There is higher connection load on the masters that before since the clients talk directly to
+HMasters instead of ZooKeeper ensemble`
+
+To reduce hot-spotting on a single master, all the masters (active & stand-by) expose the needed
+service to fetch the connection metadata. This lets the client connect to any master (not just active).
+
+==== RPC hedging
Review comment:
Not really, the functionality was moved from rpc layer to the master registry, so from a user perspective it didn't change much. But certain configs changed, will cover as a part of https://issues.apache.org/jira/browse/HBASE-24535
##########
File path: src/main/asciidoc/_chapters/architecture.adoc
##########
@@ -260,6 +260,73 @@ For region name, we only accept `byte[]` as the parameter type and it may be a f
Information on non-Java clients and custom protocols is covered in <<external_apis>>
+[[client.masterregistry]]
+=== Master registry (new as of release 3.0.0)
Review comment:
Will do as a part of https://issues.apache.org/jira/browse/HBASE-24535. Thanks for pointing it out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-644332537
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 43s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 3m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux ecf7c9589aca 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 035ea516e9 |
| Max. process+thread count | 47 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-642298013
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 59s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 38s | branch-2 passed |
| +1 :green_heart: | mvnsite | 16m 43s | branch-2 passed |
| +0 :ok: | refguide | 6m 52s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 11s | the patch passed |
| +1 :green_heart: | mvnsite | 12m 13s | the patch passed |
| -0 :warning: | whitespace | 0m 0s | The patch has 51 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +0 :ok: | refguide | 7m 2s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 55m 45s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | dupname asflicense refguide mvnsite |
| uname | Linux f9088f2775c8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / ef984cb18e |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/artifact/yetus-general-check/output/branch-site/book.html |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/artifact/yetus-general-check/output/whitespace-eol.txt |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/artifact/yetus-general-check/output/patch-site/book.html |
| Max. process+thread count | 89 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
ndimiduk merged pull request #1880:
URL: https://github.com/apache/hbase/pull/1880
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-644356618
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 29s | branch-2 passed |
| +1 :green_heart: | mvnsite | 16m 48s | branch-2 passed |
| +0 :ok: | refguide | 6m 40s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 7s | the patch passed |
| +1 :green_heart: | mvnsite | 12m 3s | the patch passed |
| -0 :warning: | whitespace | 0m 0s | The patch has 51 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +0 :ok: | refguide | 7m 0s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 51m 49s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | dupname asflicense refguide mvnsite |
| uname | Linux 0bdf544bb157 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 035ea516e9 |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/artifact/yetus-general-check/output/branch-site/book.html |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/artifact/yetus-general-check/output/whitespace-eol.txt |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/artifact/yetus-general-check/output/patch-site/book.html |
| Max. process+thread count | 89 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-641977824
Wow, good one to keep docs in sync. Looks like many commits related to docs were done only on master branch. Maybe backporting is not being followed religiously.
For this PR, I believe high level we can focus on these new docs. Snapshot scanner, hbtop, backup restore should be included. I am not sure about spark related doc though.
```
include::_chapters/snapshot_scanner.adoc[]
include::_chapters/backup_restore.adoc[]
include::_chapters/spark.adoc[]
include::_chapters/hbtop.adoc[]
```
Let me skim through this few more times. However, I might need some help since I am not sure if we have some feature/fixes only present in 3.0.0 so far (like Sync Replication) and not planned for backports on 2.x.
Btw I believe @busbey created one Jira last year to document feature(Jira) to relevant releases mapping. That one would be really helpful in such scenarios.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-642277039
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 2m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux 3a33f5f50d37 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / ef984cb18e |
| Max. process+thread count | 51 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-642230368
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 38s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 3m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux 2d57f5e401bd 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / ef984cb18e |
| Max. process+thread count | 49 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#discussion_r438346674
##########
File path: src/main/asciidoc/_chapters/cp.adoc
##########
@@ -139,7 +139,7 @@ Referential Integrity::
Secondary Indexes::
You can use a coprocessor to maintain secondary indexes. For more information, see
- link:https://wiki.apache.org/hadoop/Hbase/SecondaryIndexing[SecondaryIndexing].
+ link:https://cwiki.apache.org/confluence/display/HADOOP2/Hbase+SecondaryIndexing[SecondaryIndexing].
Review comment:
Yikes! https://issues.apache.org/jira/browse/HBASE-24533
##########
File path: src/main/asciidoc/_chapters/ops_mgt.adoc
##########
@@ -3400,20 +3549,10 @@ Here is example using a few of the rsgroup commands. To add a group, do as foll
.RegionServer Groups must be Enabled
[NOTE]
====
-If you have not enabled the rsgroup Coprocessor Endpoint in the master and
Review comment:
Looks like I need to revert HBASE-23890 from this patch.
##########
File path: src/main/asciidoc/_chapters/architecture.adoc
##########
@@ -260,6 +260,73 @@ For region name, we only accept `byte[]` as the parameter type and it may be a f
Information on non-Java clients and custom protocols is covered in <<external_apis>>
+[[client.masterregistry]]
+=== Master registry (new as of release 3.0.0)
Review comment:
@bharathv is there some tweak we can make to this language so that it's applicable to all the back ported branches as well? Backports have happened, so we can now be more precise about what the state is on each branch. Just a small change here saying something to the effect of "this is the new default in 3.0, but it available as an optional feature in versions 1.x+ and 2.y+" I see down in `configuration.adoc` it's very clear about what the state is on 3.x vs 2.3+.
Let's get it as a separate ticket, I don't want to also make doc changes in this back-port ticket.
##########
File path: src/main/asciidoc/_chapters/faq.adoc
##########
@@ -33,7 +33,7 @@ When should I use HBase?::
See <<arch.overview>> in the Architecture chapter.
Are there other HBase FAQs?::
- See the FAQ that is up on the wiki, link:https://wiki.apache.org/hadoop/Hbase/FAQ[HBase Wiki FAQ].
+ See the FAQ that is up on the wiki, link:https://cwiki.apache.org/confluence/display/HADOOP2/Hbase+FAQ[HBase Wiki FAQ].
Review comment:
Circular references are fun! https://issues.apache.org/jira/browse/HBASE-24534
##########
File path: src/main/asciidoc/_chapters/architecture.adoc
##########
@@ -260,6 +260,73 @@ For region name, we only accept `byte[]` as the parameter type and it may be a f
Information on non-Java clients and custom protocols is covered in <<external_apis>>
+[[client.masterregistry]]
+=== Master registry (new as of release 3.0.0)
+
+Client internally works with a _connection registry_ to fetch the metadata needed by connections.
+This connection registry implementation is responsible for fetching the following metadata.
+
+* Active master address
+* Current meta region(s) locations
+* Cluster ID (unique to this cluster)
+
+This information is needed as a part of various client operations like connection set up, scans,
+gets etc. Up until releases 2.x.y, the default connection registry is based on ZooKeeper as the
+source of truth and the the clients fetched the metadata from zookeeper znodes. As of release 3.0.0,
+the default implementation for connection registry has been switched to a master based
+implementation. With this change, the clients now fetch the required metadata from master RPC end
+points directly. This change was done for the following reasons.
+
+* Reduce load on ZooKeeper since that is critical for cluster operation.
+* Holistic client timeout and retry configurations since the new registry brings all the client
+operations under HBase rpc framework.
+* Remove the ZooKeeper client dependency on HBase client library.
+
+This means that
+
+* At least a single active or stand by master is needed for cluster connection setup. Refer to
+<<master.runtime>> for more details.
+* Master can be in a critical path of read/write operations, especially if the client metadata cache
+is empty or stale.
+* There is higher connection load on the masters that before since the clients talk directly to
+HMasters instead of ZooKeeper ensemble`
+
+To reduce hot-spotting on a single master, all the masters (active & stand-by) expose the needed
+service to fetch the connection metadata. This lets the client connect to any master (not just active).
+
+==== RPC hedging
Review comment:
@bharathv @Apache9 what was the conclusion on RPC hedging? This was reverted, wasn't it? Need to update the docs here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-641641239
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 20s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 41s | branch-2 passed |
| +1 :green_heart: | mvnsite | 16m 53s | branch-2 passed |
| +0 :ok: | refguide | 6m 53s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 13s | the patch passed |
| +1 :green_heart: | mvnsite | 12m 18s | the patch passed |
| -0 :warning: | whitespace | 0m 0s | The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +0 :ok: | refguide | 7m 7s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 54m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | dupname asflicense refguide mvnsite |
| uname | Linux 3d5884f49987 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / b67f896954 |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/artifact/yetus-general-check/output/branch-site/book.html |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/artifact/yetus-general-check/output/whitespace-eol.txt |
| refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/artifact/yetus-general-check/output/patch-site/book.html |
| Max. process+thread count | 89 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-644328501
rebased onto latest master to get HBASE-24535.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#issuecomment-642230138
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 2m 40s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1880 |
| Optional Tests | |
| uname | Linux 7bef5c7a43f9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / ef984cb18e |
| Max. process+thread count | 42 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1880/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #1880: HBASE-24144 Update docs from master
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #1880:
URL: https://github.com/apache/hbase/pull/1880#discussion_r438540619
##########
File path: src/main/asciidoc/_chapters/architecture.adoc
##########
@@ -260,6 +260,73 @@ For region name, we only accept `byte[]` as the parameter type and it may be a f
Information on non-Java clients and custom protocols is covered in <<external_apis>>
+[[client.masterregistry]]
+=== Master registry (new as of release 3.0.0)
+
+Client internally works with a _connection registry_ to fetch the metadata needed by connections.
+This connection registry implementation is responsible for fetching the following metadata.
+
+* Active master address
+* Current meta region(s) locations
+* Cluster ID (unique to this cluster)
+
+This information is needed as a part of various client operations like connection set up, scans,
+gets etc. Up until releases 2.x.y, the default connection registry is based on ZooKeeper as the
+source of truth and the the clients fetched the metadata from zookeeper znodes. As of release 3.0.0,
+the default implementation for connection registry has been switched to a master based
+implementation. With this change, the clients now fetch the required metadata from master RPC end
+points directly. This change was done for the following reasons.
+
+* Reduce load on ZooKeeper since that is critical for cluster operation.
+* Holistic client timeout and retry configurations since the new registry brings all the client
+operations under HBase rpc framework.
+* Remove the ZooKeeper client dependency on HBase client library.
+
+This means that
+
+* At least a single active or stand by master is needed for cluster connection setup. Refer to
+<<master.runtime>> for more details.
+* Master can be in a critical path of read/write operations, especially if the client metadata cache
+is empty or stale.
+* There is higher connection load on the masters that before since the clients talk directly to
+HMasters instead of ZooKeeper ensemble`
+
+To reduce hot-spotting on a single master, all the masters (active & stand-by) expose the needed
+service to fetch the connection metadata. This lets the client connect to any master (not just active).
+
+==== RPC hedging
Review comment:
Here you go #1888
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org