You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2018/06/13 05:24:27 UTC
hbase git commit: HBASE-20725 Copy down docs from master branch
(absent chapters on features not in 2.0.x)
Repository: hbase
Updated Branches:
refs/heads/branch-2.0 abe5fd071 -> dd99e2880
HBASE-20725 Copy down docs from master branch (absent chapters on features not in 2.0.x)
Copied over from master minus backup, spark. Added qualifier on
serial replication section.
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/dd99e288
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/dd99e288
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/dd99e288
Branch: refs/heads/branch-2.0
Commit: dd99e288061ab2a61d2a56246670a7989cb7f656
Parents: abe5fd0
Author: Michael Stack <st...@apache.org>
Authored: Tue Jun 12 22:23:06 2018 -0700
Committer: Michael Stack <st...@apache.org>
Committed: Tue Jun 12 22:23:16 2018 -0700
----------------------------------------------------------------------
src/main/asciidoc/_chapters/compression.adoc | 2 +-
src/main/asciidoc/_chapters/configuration.adoc | 26 +++----
src/main/asciidoc/_chapters/developer.adoc | 44 +++++++++++
src/main/asciidoc/_chapters/external_apis.adoc | 8 +-
.../asciidoc/_chapters/inmemory_compaction.adoc | 11 +--
src/main/asciidoc/_chapters/ops_mgt.adoc | 78 +++++++++++++++++++-
src/main/asciidoc/_chapters/security.adoc | 1 +
src/main/asciidoc/_chapters/upgrading.adoc | 12 ++-
8 files changed, 155 insertions(+), 27 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/compression.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/compression.adoc b/src/main/asciidoc/_chapters/compression.adoc
index b21f106..b2ff5ce 100644
--- a/src/main/asciidoc/_chapters/compression.adoc
+++ b/src/main/asciidoc/_chapters/compression.adoc
@@ -377,7 +377,7 @@ Options:
LZ4]
-data_block_encoding <arg> Encoding algorithm (e.g. prefix compression) to
use for data blocks in the test column family, one
- of [NONE, PREFIX, DIFF, FAST_DIFF, PREFIX_TREE].
+ of [NONE, PREFIX, DIFF, FAST_DIFF, ROW_INDEX_V1].
-encryption <arg> Enables transparent encryption on the test table,
one of [AES]
-generator <arg> The class which generates load for the tool. Any
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/configuration.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc
index 1be9e13..31250ff 100644
--- a/src/main/asciidoc/_chapters/configuration.adoc
+++ b/src/main/asciidoc/_chapters/configuration.adoc
@@ -217,20 +217,20 @@ Use the following legend to interpret this table:
* "X" = not supported
* "NT" = Not tested
-[cols="1,1,1,1", options="header"]
+[cols="1,1,1,1,1,1", options="header"]
|===
-| | HBase-1.2.x | HBase-1.3.x | HBase-2.0.x
-|Hadoop-2.4.x | S | S | X
-|Hadoop-2.5.x | S | S | X
-|Hadoop-2.6.0 | X | X | X
-|Hadoop-2.6.1+ | S | S | S
-|Hadoop-2.7.0 | X | X | X
-|Hadoop-2.7.1+ | S | S | S
-|Hadoop-2.8.[0-1] | X | X | X
-|Hadoop-2.8.2 | NT | NT | NT
-|Hadoop-2.8.3+ | NT | NT | S
-|Hadoop-2.9.0 | X | X | X
-|Hadoop-3.0.0 | NT | NT | NT
+| | HBase-1.2.x | HBase-1.3.x | HBase-1.5.x | HBase-2.0.x | HBase-2.1.x
+|Hadoop-2.4.x | S | S | X | X | X
+|Hadoop-2.5.x | S | S | X | X | X
+|Hadoop-2.6.0 | X | X | X | X | X
+|Hadoop-2.6.1+ | S | S | X | S | X
+|Hadoop-2.7.0 | X | X | X | X | X
+|Hadoop-2.7.1+ | S | S | S | S | S
+|Hadoop-2.8.[0-1] | X | X | X | X | X
+|Hadoop-2.8.2 | NT | NT | NT | NT | NT
+|Hadoop-2.8.3+ | NT | NT | NT | S | S
+|Hadoop-2.9.0 | X | X | X | X | X
+|Hadoop-3.0.0 | NT | NT | NT | NT | NT
|===
.Hadoop Pre-2.6.1 and JDK 1.8 Kerberos
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/developer.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc
index 957f68b..6d0a7d1 100644
--- a/src/main/asciidoc/_chapters/developer.adoc
+++ b/src/main/asciidoc/_chapters/developer.adoc
@@ -870,6 +870,50 @@ See link:http://search-hadoop.com/m/DHED4dhFaU[HBase, mail # dev - On
recent discussion clarifying ASF release policy].
for how we arrived at this process.
+[[hbase.release.announcement]]
+== Announcing Releases
+
+Once an RC has passed successfully and the needed artifacts have been staged for disribution, you'll need to let everyone know about our shiny new release. It's not a requirement, but to make things easier for release managers we have a template you can start with. Be sure you replace \_version_ and other markers with the relevant version numbers. You should manually verify all links before sending.
+
+[source,email]
+----
+The HBase team is happy to announce the immediate availability of HBase _version_.
+
+Apache HBase™ is an open-source, distributed, versioned, non-relational database.
+Apache HBase gives you low latency random access to billions of rows with
+millions of columns atop non-specialized hardware. To learn more about HBase,
+see https://hbase.apache.org/.
+
+HBase _version_ is the _nth_ minor release in the HBase _major_.x line, which aims to
+improve the stability and reliability of HBase. This release includes roughly
+XXX resolved issues not covered by previous _major_.x releases.
+
+Notable new features include:
+- List text descriptions of features that fit on one line
+- Including if JDK or Hadoop support versions changes
+- If the "stable" pointer changes, call that out
+- For those with obvious JIRA IDs, include them (HBASE-YYYYY)
+
+The full list of issues can be found in the included CHANGES.md and RELEASENOTES.md,
+or via our issue tracker:
+
+ https://s.apache.org/hbase-_version_-jira
+
+To download please follow the links and instructions on our website:
+
+ https://hbase.apache.org/downloads.html
+
+
+Question, comments, and problems are always welcome at: dev@hbase.apache.org.
+
+Thanks to all who contributed and made this release possible.
+
+Cheers,
+The HBase Dev Team
+----
+
+You should sent this message to the following lists: dev@hbase.apache.org, user@hbase.apache.org, announce@apache.org. If you'd like a spot check before sending, feel free to ask via jira or the dev list.
+
[[documentation]]
== Generating the HBase Reference Guide
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/external_apis.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/external_apis.adoc b/src/main/asciidoc/_chapters/external_apis.adoc
index ffb6ee6..c1299a6 100644
--- a/src/main/asciidoc/_chapters/external_apis.adoc
+++ b/src/main/asciidoc/_chapters/external_apis.adoc
@@ -186,20 +186,20 @@ creation or mutation, and `DELETE` for deletion.
|/_table_/schema
|POST
-|Create a new table, or replace an existing table's schema
+|Update an existing table with the provided schema fragment
|curl -vi -X POST \
-H "Accept: text/xml" \
-H "Content-Type: text/xml" \
- -d '<?xml version="1.0" encoding="UTF-8"?><TableSchema name="users"><ColumnSchema name="cf" /></TableSchema>' \
+ -d '<?xml version="1.0" encoding="UTF-8"?><TableSchema name="users"><ColumnSchema name="cf" KEEP_DELETED_CELLS="true" /></TableSchema>' \
"http://example.com:8000/users/schema"
|/_table_/schema
|PUT
-|Update an existing table with the provided schema fragment
+|Create a new table, or replace an existing table's schema
|curl -vi -X PUT \
-H "Accept: text/xml" \
-H "Content-Type: text/xml" \
- -d '<?xml version="1.0" encoding="UTF-8"?><TableSchema name="users"><ColumnSchema name="cf" KEEP_DELETED_CELLS="true" /></TableSchema>' \
+ -d '<?xml version="1.0" encoding="UTF-8"?><TableSchema name="users"><ColumnSchema name="cf" /></TableSchema>' \
"http://example.com:8000/users/schema"
|/_table_/schema
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/inmemory_compaction.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/inmemory_compaction.adoc b/src/main/asciidoc/_chapters/inmemory_compaction.adoc
index 3fd2d8b..f64bc7a 100644
--- a/src/main/asciidoc/_chapters/inmemory_compaction.adoc
+++ b/src/main/asciidoc/_chapters/inmemory_compaction.adoc
@@ -54,7 +54,7 @@ To enable in-memory compactions, set the _IN_MEMORY_COMPACTION_ attribute
on per column family where you want the behavior. The _IN_MEMORY_COMPACTION_
attribute can have one of four values.
- * _NONE_: No in-memory compaction. The default.
+ * _NONE_: No in-memory compaction.
* _BASIC_: Basic policy enables flushing and keeps a pipeline of flushes until we trip the pipeline maximum threshold and then we flush to disk. No in-memory compaction but can help throughput as data is moved from the profligate, native ConcurrentSkipListMap data-type to more compact (and efficient) data types.
* _EAGER_: This is _BASIC_ policy plus in-memory compaction of flushes (much like the on-disk compactions done to hfiles); on compaction we apply on-disk rules eliminating versions, duplicates, ttl'd cells, etc.
* _ADAPTIVE_: Adaptive compaction adapts to the workload. It applies either index compaction or data compaction based on the ratio of duplicate cells in the data. Experimental.
@@ -84,12 +84,13 @@ Note how the IN_MEMORY_COMPACTION attribute shows as part of the _METADATA_ map.
There is also a global configuration, _hbase.hregion.compacting.memstore.type_ which you can set in your _hbase-site.xml_ file. Use it to set the
default on creation of a new table (On creation of a column family Store, we look first to the column family configuration looking for the
-_IN_MEMORY_COMPACTION_ setting, and if none, we then consult the _hbase.hregion.compacting.memstore.type_ value using its content).
+_IN_MEMORY_COMPACTION_ setting, and if none, we then consult the _hbase.hregion.compacting.memstore.type_ value using its content; default is
+_BASIC_).
-By default, new hbase system tables will have _NONE_ in-memory compaction set. To specify otherwise,
-on new table-creation, set _hbase.hregion.compacting.memstore.type_ to _BASIC_ (Note, setting this value
+By default, new hbase system tables will have _BASIC_ in-memory compaction set. To specify otherwise,
+on new table-creation, set _hbase.hregion.compacting.memstore.type_ to _NONE_ (Note, setting this value
post-creation of system tables will not have a retroactive effect; you will have to alter your tables
-to set the in-memory attribute).
+to set the in-memory attribute to _NONE_).
When an in-memory flush happens is calculated by dividing the configured region flush size (Set in the table descriptor
or read from _hbase.hregion.memstore.flush.size_) by the number of column families and then multiplying by
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/ops_mgt.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/ops_mgt.adoc b/src/main/asciidoc/_chapters/ops_mgt.adoc
index f4502cb..99a08a1 100644
--- a/src/main/asciidoc/_chapters/ops_mgt.adoc
+++ b/src/main/asciidoc/_chapters/ops_mgt.adoc
@@ -1496,9 +1496,12 @@ If a slave cluster does run out of room, or is inaccessible for other reasons, i
.Consistency Across Replicated Clusters
[WARNING]
====
-How your application builds on top of the HBase API matters when replication is in play. HBase's replication system provides at-least-once delivery of client edits for an enabled column family to each configured destination cluster. In the event of failure to reach a given destination, the replication system will retry sending edits in a way that might repeat a given message. Further more, there is not a guaranteed order of delivery for client edits. In the event of a RegionServer failing, recovery of the replication queue happens independent of recovery of the individual regions that server was previously handling. This means that it is possible for the not-yet-replicated edits to be serviced by a RegionServer that is currently slower to replicate than the one that handles edits from after the failure.
+How your application builds on top of the HBase API matters when replication is in play. HBase's replication system provides at-least-once delivery of client edits for an enabled column family to each configured destination cluster. In the event of failure to reach a given destination, the replication system will retry sending edits in a way that might repeat a given message. HBase provides two ways of replication, one is the original replication and the other is serial replication. In the previous way of replication, there is not a guaranteed order of delivery for client edits. In the event of a RegionServer failing, recovery of the replication queue happens independent of recovery of the individual regions that server was previously handling. This means that it is possible for the not-yet-replicated edits to be serviced by a RegionServer that is currently slower to replicate than the one that handles edits from after the failure.
The combination of these two properties (at-least-once delivery and the lack of message ordering) means that some destination clusters may end up in a different state if your application makes use of operations that are not idempotent, e.g. Increments.
+
+To solve the problem, HBase now supports serial replication, which sends edits to destination cluster as the order of requests from client. See <<Serial Replication,Serial Replication>>.
+
====
.Terminology Changes
@@ -1539,11 +1542,14 @@ Instead of SQL statements, entire WALEdits (consisting of multiple cell inserts
LOG.info("Replicating "+clusterId + " -> " + peerClusterId);
----
+.Serial Replication Configuration
+See <<Serial Replication,Serial Replication>>
+
.Cluster Management Commands
add_peer <ID> <CLUSTER_KEY>::
Adds a replication relationship between two clusters. +
* ID -- a unique string, which must not contain a hyphen.
- * CLUSTER_KEY: composed using the following template, with appropriate place-holders: `hbase.zookeeper.quorum:hbase.zookeeper.property.clientPort:zookeeper.znode.parent`
+ * CLUSTER_KEY: composed using the following template, with appropriate place-holders: `hbase.zookeeper.quorum:hbase.zookeeper.property.clientPort:zookeeper.znode.parent`. This value can be found on the Master UI info page.
* STATE(optional): ENABLED or DISABLED, default value is ENABLED
list_peers:: list all replication relationships known by this cluster
enable_peer <ID>::
@@ -1560,6 +1566,32 @@ enable_table_replication <TABLE_NAME>::
disable_table_replication <TABLE_NAME>::
Disable the table replication switch for all its column families.
+=== Serial Replication
+
+Note: this feature is introduced in HBase 2.1 and is not available in 2.0.x, your current version.
+
+.Function of serial replication
+
+Serial replication supports to push logs to the destination cluster in the same order as logs reach to the source cluster.
+
+.Why need serial replication?
+In replication of HBase, we push mutations to destination cluster by reading WAL in each region server. We have a queue for WAL files so we can read them in order of creation time. However, when region-move or RS failure occurs in source cluster, the hlog entries that are not pushed before region-move or RS-failure will be pushed by original RS(for region move) or another RS which takes over the remained hlog of dead RS(for RS failure), and the new entries for the same region(s) will be pushed by the RS which now serves the region(s), but they push the hlog entries of a same region concurrently without coordination.
+
+This treatment can possibly lead to data inconsistency between source and destination clusters:
+
+1. there are put and then delete written to source cluster.
+
+2. due to region-move / RS-failure, they are pushed by different replication-source threads to peer cluster.
+
+3. if delete is pushed to peer cluster before put, and flush and major-compact occurs in peer cluster before put is pushed to peer cluster, the delete is collected and the put remains in peer cluster, but in source cluster the put is masked by the delete, hence data inconsistency between source and destination clusters.
+
+
+.Serial replication configuration
+
+. Set the serial flag to true for a repliation peer. You can either set it to true when creating a replication peer, or change it to true later.
+
+The serial replication feature had been done firstly in link:https://issues.apache.org/jira/browse/HBASE-9465[HBASE-9465] and then reverted and redone in link:https://issues.apache.org/jira/browse/HBASE-20046[HBASE-20046]. You can find more details in these issues.
+
=== Verifying Replicated Data
The `VerifyReplication` MapReduce job, which is included in HBase, performs a systematic comparison of replicated data between two different clusters. Run the VerifyReplication job on the master cluster, supplying it with the peer ID and table name to use for validation. You can limit the verification further by specifying a time range or specific families. The job's short name is `verifyrep`. To run the job, use a command like the following:
@@ -2797,6 +2829,48 @@ Viewing the Master log will give you insight on rsgroup operation.
If it appears stuck, restart the Master process.
+=== Remove RegionServer Grouping
+Removing RegionServer Grouping feature from a cluster on which it was enabled involves
+more steps in addition to removing the relevant properties from `hbase-site.xml`. This is
+to clean the RegionServer grouping related meta data so that if the feature is re-enabled
+in the future, the old meta data will not affect the functioning of the cluster.
+
+- Move all tables in non-default rsgroups to `default` regionserver group
+[source,bash]
+----
+#Reassigning table t1 from non default group - hbase shell
+hbase(main):005:0> move_tables_rsgroup 'default',['t1']
+----
+- Move all regionservers in non-default rsgroups to `default` regionserver group
+[source, bash]
+----
+#Reassigning all the servers in the non-default rsgroup to default - hbase shell
+hbase(main):008:0> move_servers_rsgroup 'default',['rs1.xxx.com:16206','rs2.xxx.com:16202','rs3.xxx.com:16204']
+----
+- Remove all non-default rsgroups. `default` rsgroup created implicitly doesn't have to be removed
+[source,bash]
+----
+#removing non default rsgroup - hbase shell
+hbase(main):009:0> remove_rsgroup 'group2'
+----
+- Remove the changes made in `hbase-site.xml` and restart the cluster
+- Drop the table `hbase:rsgroup` from `hbase`
+[source, bash]
+----
+#Through hbase shell drop table hbase:rsgroup
+hbase(main):001:0> disable 'hbase:rsgroup'
+0 row(s) in 2.6270 seconds
+
+hbase(main):002:0> drop 'hbase:rsgroup'
+0 row(s) in 1.2730 seconds
+----
+- Remove znode `rsgroup` from the cluster ZooKeeper using zkCli.sh
+[source, bash]
+----
+#From ZK remove the node /hbase/rsgroup through zkCli.sh
+rmr /hbase/rsgroup
+----
+
=== ACL
To enable ACL, add the following to your hbase-site.xml and restart your Master:
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/security.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc
index c007053..dae6c53 100644
--- a/src/main/asciidoc/_chapters/security.adoc
+++ b/src/main/asciidoc/_chapters/security.adoc
@@ -662,6 +662,7 @@ You also need to enable the DataBlockEncoder for the column family, for encoding
You can enable compression of each tag in the WAL, if WAL compression is also enabled, by setting the value of `hbase.regionserver.wal.tags.enablecompression` to `true` in _hbase-site.xml_.
Tag compression uses dictionary encoding.
+Coprocessors that run server-side on RegionServers can perform get and set operations on cell Tags. Tags are stripped out at the RPC layer before the read response is sent back, so clients do not see these tags.
Tag compression is not supported when using WAL encryption.
[[hbase.accesscontrol.configuration]]
http://git-wip-us.apache.org/repos/asf/hbase/blob/dd99e288/src/main/asciidoc/_chapters/upgrading.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/upgrading.adoc b/src/main/asciidoc/_chapters/upgrading.adoc
index dea72ab..a126422 100644
--- a/src/main/asciidoc/_chapters/upgrading.adoc
+++ b/src/main/asciidoc/_chapters/upgrading.adoc
@@ -86,8 +86,16 @@ for warning about incompatible changes). All effort will be made to provide a de
* Example: Old compiled Coprocessor, Filter, or Plugin code will work unchanged with the new jars.
.Dependency Compatibility
-* An upgrade of HBase will not require an incompatible upgrade of a dependent project, including the Java runtime.
-* Example: An upgrade of Hadoop will not invalidate any of the compatibilities guarantees we made.
+* An upgrade of HBase will not require an incompatible upgrade of a dependent project, except for Apache Hadoop.
+* An upgrade of HBase will not require an incompatible upgrade of the Java runtime.
+* Example: Upgrading HBase to a version that supports _Dependency Compatibility_ won't require that you upgrade your Apache ZooKeeper service.
+* Example: If your current version of HBase supported running on JDK 8, then an upgrade to a version that supports _Dependency Compatibility_ will also run on JDK 8.
+
+.Hadoop Versions
+[TIP]
+====
+Previously, we tried to maintain dependency compatibility for the underly Hadoop service but over the last few years this has proven untenable. While the HBase project attempts to maintain support for older versions of Hadoop, we drop the "supported" designator for minor versions that fail to continue to see releases. Additionally, the Hadoop project has its own set of compatibility guidelines, which means in some cases having to update to a newer supported minor release might break some of our compatibility promises.
+====
.Operational Compatibility
* Metric changes