You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by ta...@apache.org on 2026/07/10 18:13:21 UTC
(hbase) branch master updated: HBASE-30167: Document the Read Replica feature (#8459)
This is an automated email from the ASF dual-hosted git repository.
taklwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new f9a41c73ae3 HBASE-30167: Document the Read Replica feature (#8459)
f9a41c73ae3 is described below
commit f9a41c73ae3f1ba4b794f52ae63adb43523cd480
Author: Kota-SH <sh...@gmail.com>
AuthorDate: Fri Jul 10 11:13:15 2026 -0700
HBASE-30167: Document the Read Replica feature (#8459)
Signed-off-by: Andor Molnár <an...@apache.org>
Signed-off-by: Peng Lu <lu...@apache.org>
Signed-off-by: Tak Lon (Stephen) Wu <ta...@apache.org>
---
.../pages/_docs/docs/_mdx/(multi-page)/meta.json | 1 +
.../_mdx/(multi-page)/read-replica-cluster.mdx | 205 +++++++++++++++++++++
2 files changed, 206 insertions(+)
diff --git a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/meta.json b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/meta.json
index 0c07b72bc26..9fc144368b2 100644
--- a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/meta.json
+++ b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/meta.json
@@ -36,6 +36,7 @@
"case-studies",
"---Operations & Development---",
"operational-management",
+ "read-replica-cluster",
"building-and-developing",
"unit-testing",
"---Internals---",
diff --git a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx
new file mode 100644
index 00000000000..fa93aa31b40
--- /dev/null
+++ b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx
@@ -0,0 +1,205 @@
+---
+title: "Read Replica Cluster"
+description: "Running a secondary HBase cluster in read-only mode against shared cloud storage to scale read workloads."
+---
+
+## Background [#read-replica-cluster-background]
+
+A _Read Replica Cluster_ is an entire HBase cluster running in global read-only mode against the same shared
+storage (`hbase.rootdir`) as an active read-write cluster. Both clusters list the same HFiles in the same
+HDFS / cloud-object-store location; no data is copied. Reads can be served from either cluster, letting the
+read workload be fanned out across multiple clusters without doubling storage cost.
+
+Typical use cases:
+
+- Fan out heavy scan / analytical workloads off the primary cluster.
+- Add cross-availability-zone read capacity backed by a single shared bucket.
+- Stand up an isolated cluster for read-mostly experiments without copying data.
+
+<Callout type="info">
+ **Eventual consistency.** A replica only sees data once (a) the active cluster has flushed the data to
+ HFiles in shared storage, and (b) the replica has been told to re-read shared storage via the
+ `refresh_meta` and `refresh_hfiles` commands. MemStore data on the active cluster is invisible to the
+ replica until flushed.
+</Callout>
+
+The parent design lives on [HBASE-29081](https://issues.apache.org/jira/browse/HBASE-29081).
+
+## Design
+
+The feature has three parts.
+
+### Custom `hbase:meta` per cluster
+
+Every cluster sharing a `hbase.rootdir` needs its own `hbase:meta` and its own master local region
+directory, because region assignments and master-local state are node-scoped and cannot be shared. Other
+system tables (`hbase:acl`, `hbase:replication`) _are_ safe to share because their contents are storage-wide
+and the replica never writes to them.
+
+The configuration key `hbase.meta.table.suffix` selects a per-cluster suffix; the meta table becomes
+`hbase:meta_<suffix>` and the master's local store directory becomes `MasterData_<suffix>`. Each cluster
+sharing the same `hbase.rootdir` must be configured with a distinct suffix so its `hbase:meta` and
+`MasterData` directory do not collide with any other cluster's. The suffix must match `[a-zA-Z0-9]+`.
+
+### Global read-only mode
+
+`hbase.global.readonly.enabled=true` puts a cluster into read-only mode. Five coprocessor controllers under
+`org.apache.hadoop.hbase.security.access` intercept every user-table mutation path and throw
+`WriteAttemptedOnReadOnlyClusterException` (a `DoNotRetryIOException`) with the message
+`Operation not allowed in Read-Only Mode`:
+
+| Class | Coprocessor host | Blocks |
+| -------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------- |
+| `MasterReadOnlyController` | Master | DDL, snapshots, splits, merges, namespace ops, ACL/quota ops, replication-peer ops |
+| `RegionReadOnlyController` | Region | put, delete, batchMutate, checkAnd\*, append, increment, flush, compaction, WAL append, commit/replay |
+| `RegionServerReadOnlyController` | RegionServer | WAL roll, replication sink mutations, log replay |
+| `BulkLoadReadOnlyController` | Region | bulk-load prepare/cleanup |
+| `EndpointReadOnlyController` | Region | all coprocessor endpoint invocations |
+
+Operators do not load these classes manually. `CoprocessorConfigurationUtil.syncReadOnlyConfigurations`
+adds them to `hbase.coprocessor.master.classes`, `hbase.coprocessor.regionserver.classes`, and
+`hbase.coprocessor.region.classes` at startup and on every dynamic
+`ConfigurationManager.notifyAllObservers` event — so the flag can be flipped at runtime with
+`update_all_config` (see Case 3).
+
+### Preventing Multiple Active Clusters (active.cluster.suffix.id)
+
+Two clusters writing to the same `hbase.rootdir` would corrupt shared storage. To enforce a single writer, an
+active master creates a protobuf-serialized sentinel at `<hbase.rootdir>/active.cluster.suffix.id` recording
+its cluster ID and meta suffix. `MasterFileSystem.negotiateActiveClusterSuffixFile` runs at master startup:
+
+- An **active** cluster (`hbase.global.readonly.enabled=false`) creates the file if absent, or verifies its
+ contents match its own identity. If the file belongs to another cluster, startup aborts with an
+ `IOException`.
+- A **replica** cluster (`hbase.global.readonly.enabled=true`) does not read or write the file; it logs
+ `[Read-replica feature] Replica cluster is being started in Read Only Mode` and continues.
+
+`AbstractReadOnlyController.manageActiveClusterIdFile` handles the dynamic toggle: switching to read-only
+deletes the file if this cluster owns it, and switching back to read-write creates the file if absent.
+
+## Configuration
+
+On every node of the **read replica cluster**, add the following to `hbase-site.xml`:
+
+```xml
+<property>
+ <name>hbase.global.readonly.enabled</name>
+ <value>true</value>
+ <description>
+ Put this cluster into global read-only mode. All user-table writes, flushes,
+ compactions, splits, and merges are blocked. The five ReadOnly coprocessor
+ controllers are loaded automatically.
+ </description>
+</property>
+<property>
+ <name>hbase.meta.table.suffix</name>
+ <value>replica1</value>
+ <description>
+ Optional. If set, the meta table is named hbase:meta_<suffix> and the
+ master's local store directory is MasterData_<suffix>. Value must match
+ [a-zA-Z0-9]+. Each cluster sharing the same hbase.rootdir MUST be
+ configured with a distinct suffix so its hbase:meta and MasterData
+ directory do not collide with any other cluster's.
+ </description>
+</property>
+```
+
+The **active cluster** uses the same `hbase.rootdir` but its own `hbase.meta.table.suffix` (distinct from
+every replica's suffix), and leaves `hbase.global.readonly.enabled` unset or `false`.
+
+`hbase.global.readonly.enabled` is a dynamic configuration — a config-change event reloads the read-only
+coprocessors without restarting the process. All nodes must agree on the value; operators are responsible for
+keeping every `hbase-site.xml` in sync before issuing `update_all_config`.
+
+## Operation and maintenance
+
+### Case 1. Bring up a new read replica cluster
+
+1. Provision the replica cluster on hardware that can reach the active cluster's `hbase.rootdir` (typically
+ the same HDFS or object store).
+2. Set `hbase.global.readonly.enabled=true` in the replica's `hbase-site.xml`. And set up
+ `hbase.meta.table.suffix`, to distinguish the replica cluster's meta table on the shared storage.
+3. Start the cluster and verify if the Master log shows
+ `[Read-replica feature] Replica cluster is being started in Read Only Mode`.
+4. From the replica shell, run `refresh_meta` and then `refresh_hfiles` to materialize the active cluster's
+ current state on the replica.
+
+### Case 2. Routine sync after writes on the active cluster
+
+```ruby
+# On the active cluster
+hbase> flush 'my_namespace:my_table'
+```
+
+```ruby
+# On the read replica cluster
+hbase> refresh_meta
+hbase> refresh_hfiles 'TABLE_NAME' => 'my_namespace:my_table'
+```
+
+Always run `refresh_meta` first, then `refresh_hfiles`. `refresh_hfiles` only refreshes regions that are open
+on the replica, so newly discovered regions must be in meta (and assigned) before their HFiles can be picked
+up. `refresh_hfiles` supports three scopes:
+
+```ruby
+hbase> refresh_hfiles # all user tables
+hbase> refresh_hfiles 'TABLE_NAME' => 'ns:table' # one table
+hbase> refresh_hfiles 'NAMESPACE' => 'ns' # one namespace
+```
+
+Passing both `TABLE_NAME` and `NAMESPACE` to `refresh_hfiles` is rejected. Both commands return a procedure
+ID that can be tracked through the master UI or `Admin.getProcedures()`.
+
+If the replica's block cache holds stale entries for a table that has just been refreshed, evict them with
+the pre-existing `clear_block_cache 'my_namespace:my_table'` shell command.
+
+`Admin` and `AsyncAdmin` expose the same operations programmatically:
+
+```java
+long pid;
+pid = admin.refreshMeta();
+pid = admin.refreshHFiles(); // all user tables
+pid = admin.refreshHFiles(TableName.valueOf("ns:table")); // one table
+pid = admin.refreshHFiles("ns"); // one namespace
+```
+
+### Case 3. Dynamically toggle read-only mode
+
+`hbase.global.readonly.enabled` can be changed without a restart. Edit `hbase-site.xml` then
+trigger a configuration refresh (for example, `update_all_config` from the shell). `ConfigurationManager`
+notifies its observers, which load or unload the read-only coprocessors and call
+`AbstractReadOnlyController.manageActiveClusterIdFile`:
+
+- **false → true (becoming a replica):** the `active.cluster.suffix.id` file is deleted only if its contents
+ match this cluster; if another cluster owns the file, it is left in place.
+- **true → false (becoming active):** the file is recreated with this cluster's identity, unless it already
+ exists.
+
+In-flight batch operations are not interrupted; write operations submitted _after_ the toggle throw
+`WriteAttemptedOnReadOnlyClusterException`. The caller is responsible for handling and (if desired) resubmitting the failed
+mutations.
+
+### Case 4. Promote a replica when the active cluster is lost
+
+1. Confirm the original active cluster is fully down.
+2. If a stale `active.cluster.suffix.id` from the previous active is still present, remove it manually
+ (e.g. `hdfs dfs -rm <hbase.rootdir>/active.cluster.suffix.id`, or the equivalent CLI for your object
+ store). The new active master will refuse to start while a foreign sentinel file is in place.
+3. Set `hbase.global.readonly.enabled=false` on the replica and apply the change (dynamic update or
+ restart). The master writes a fresh sentinel file with this cluster's identity.
+
+## Configurations and Commands
+
+### New configs
+
+| Config | Default | Description |
+| ------------------------------- | ------- | ---------------------------------------------------------------------------------------------- |
+| `hbase.meta.table.suffix` | `""` | Adds a suffix to the meta table name. `value='test'` produces the table name `hbase:meta_test`. |
+| `hbase.global.readonly.enabled` | `false` | Puts the entire cluster into read-only mode. |
+
+### New commands
+
+| Command | Usage | Description |
+| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
+| `refresh_hfiles` | `refresh_hfiles`<br />`refresh_hfiles 'TABLE_NAME' => 'tablename'`<br />`refresh_hfiles 'TABLE_NAME' => 'namespace:test_table'`<br />`refresh_hfiles 'NAMESPACE' => 'namespace'` | Refreshes HFiles from disk. Used to pick up new edits on the read replica. |
+| `refresh_meta` | `refresh_meta` | Syncs the meta table with the backing storage. Used to pick up new tables and regions. |