You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by nd...@apache.org on 2016/01/17 01:07:15 UTC
[1/4] hbase git commit: updating docs from master
Repository: hbase
Updated Branches:
refs/heads/branch-1.1 5a1cfc1e9 -> c07ddc6db
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/rpc.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/rpc.adoc b/src/main/asciidoc/_chapters/rpc.adoc
index c319f39..1d363eb 100644
--- a/src/main/asciidoc/_chapters/rpc.adoc
+++ b/src/main/asciidoc/_chapters/rpc.adoc
@@ -47,7 +47,7 @@ For more background on how we arrived at this spec., see link:https://docs.googl
. A wire-format we can evolve
-. A format that does not require our rewriting server core or radically changing its current architecture (for later).
+. A format that does not require our rewriting server core or radically changing its current architecture (for later).
=== TODO
@@ -58,7 +58,7 @@ For more background on how we arrived at this spec., see link:https://docs.googl
. Diagram on how it works
. A grammar that succinctly describes the wire-format.
Currently we have these words and the content of the rpc protobuf idl but a grammar for the back and forth would help with groking rpc.
- Also, a little state machine on client/server interactions would help with understanding (and ensuring correct implementation).
+ Also, a little state machine on client/server interactions would help with understanding (and ensuring correct implementation).
=== RPC
@@ -79,7 +79,7 @@ link:https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=blob;f=hbase-protocol
Client initiates connection.
===== Client
-On connection setup, client sends a preamble followed by a connection header.
+On connection setup, client sends a preamble followed by a connection header.
.<preamble>
[source]
@@ -106,7 +106,7 @@ After client sends preamble and connection header, server does NOT respond if su
No response means server is READY to accept requests and to give out response.
If the version or authentication in the preamble is not agreeable or the server has trouble parsing the preamble, it will throw a org.apache.hadoop.hbase.ipc.FatalConnectionException explaining the error and will then disconnect.
If the client in the connection header -- i.e.
-the protobuf'd Message that comes after the connection preamble -- asks for for a Service the server does not support or a codec the server does not have, again we throw a FatalConnectionException with explanation.
+the protobuf'd Message that comes after the connection preamble -- asks for a Service the server does not support or a codec the server does not have, again we throw a FatalConnectionException with explanation.
==== Request
@@ -118,7 +118,7 @@ The header includes the method name and optionally, metadata on the optional Cel
The parameter type suits the method being invoked: i.e.
if we are doing a getRegionInfo request, the protobuf Message param will be an instance of GetRegionInfoRequest.
The response will be a GetRegionInfoResponse.
-The CellBlock is optionally used ferrying the bulk of the RPC data: i.e Cells/KeyValues.
+The CellBlock is optionally used ferrying the bulk of the RPC data: i.e. Cells/KeyValues.
===== Request Parts
@@ -182,7 +182,7 @@ Codecs will live on the server for all time so old clients can connect.
.Constraints
In some part, current wire-format -- i.e.
-all requests and responses preceeded by a length -- has been dictated by current server non-async architecture.
+all requests and responses preceded by a length -- has been dictated by current server non-async architecture.
.One fat pb request or header+param
We went with pb header followed by pb param making a request and a pb header followed by pb response for now.
@@ -191,7 +191,7 @@ Doing header+param rather than a single protobuf Message with both header and pa
. Is closer to what we currently have
. Having a single fat pb requires extra copying putting the already pb'd param into the body of the fat request pb (and same making result)
. We can decide whether to accept the request or not before we read the param; for example, the request might be low priority.
- As is, we read header+param in one go as server is currently implemented so this is a TODO.
+ As is, we read header+param in one go as server is currently implemented so this is a TODO.
The advantages are minor.
If later, fat request has clear advantage, can roll out a v2 later.
@@ -205,18 +205,18 @@ Codec must implement hbase's `Codec` Interface.
After connection setup, all passed cellblocks will be sent with this codec.
The server will return cellblocks using this same codec as long as the codec is on the servers' CLASSPATH (else you will get `UnsupportedCellCodecException`).
-To change the default codec, set `hbase.client.default.rpc.codec`.
+To change the default codec, set `hbase.client.default.rpc.codec`.
To disable cellblocks completely and to go pure protobuf, set the default to the empty String and do not specify a codec in your Configuration.
So, set `hbase.client.default.rpc.codec` to the empty string and do not set `hbase.client.rpc.codec`.
This will cause the client to connect to the server with no codec specified.
If a server sees no codec, it will return all responses in pure protobuf.
-Running pure protobuf all the time will be slower than running with cellblocks.
+Running pure protobuf all the time will be slower than running with cellblocks.
.Compression
-Uses hadoops compression codecs.
+Uses hadoop's compression codecs.
To enable compressing of passed CellBlocks, set `hbase.client.rpc.compressor` to the name of the Compressor to use.
-Compressor must implement Hadoops' CompressionCodec Interface.
+Compressor must implement Hadoop's CompressionCodec Interface.
After connection setup, all passed cellblocks will be sent compressed.
The server will return cellblocks compressed using this same compressor as long as the compressor is on its CLASSPATH (else you will get `UnsupportedCompressionCodecException`).
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/schema_design.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/schema_design.adoc b/src/main/asciidoc/_chapters/schema_design.adoc
index a212a5c..5cf8d12 100644
--- a/src/main/asciidoc/_chapters/schema_design.adoc
+++ b/src/main/asciidoc/_chapters/schema_design.adoc
@@ -53,7 +53,7 @@ Tables must be disabled when making ColumnFamily modifications, for example:
Configuration config = HBaseConfiguration.create();
Admin admin = new Admin(conf);
-String table = "myTable";
+TableName table = TableName.valueOf("myTable");
admin.disableTable(table);
@@ -76,6 +76,50 @@ When changes are made to either Tables or ColumnFamilies (e.g. region size, bloc
See <<store,store>> for more information on StoreFiles.
+[[table_schema_rules_of_thumb]]
+== Table Schema Rules Of Thumb
+
+There are many different data sets, with different access patterns and service-level
+expectations. Therefore, these rules of thumb are only an overview. Read the rest
+of this chapter to get more details after you have gone through this list.
+
+* Aim to have regions sized between 10 and 50 GB.
+* Aim to have cells no larger than 10 MB, or 50 MB if you use <<mob>>. Otherwise,
+consider storing your cell data in HDFS and store a pointer to the data in HBase.
+* A typical schema has between 1 and 3 column families per table. HBase tables should
+not be designed to mimic RDBMS tables.
+* Around 50-100 regions is a good number for a table with 1 or 2 column families.
+Remember that a region is a contiguous segment of a column family.
+* Keep your column family names as short as possible. The column family names are
+stored for every value (ignoring prefix encoding). They should not be self-documenting
+and descriptive like in a typical RDBMS.
+* If you are storing time-based machine data or logging information, and the row key
+is based on device ID or service ID plus time, you can end up with a pattern where
+older data regions never have additional writes beyond a certain age. In this type
+of situation, you end up with a small number of active regions and a large number
+of older regions which have no new writes. For these situations, you can tolerate
+a larger number of regions because your resource consumption is driven by the active
+regions only.
+* If only one column family is busy with writes, only that column family accomulates
+memory. Be aware of write patterns when allocating resources.
+
+[[regionserver_sizing_rules_of_thumb]]
+= RegionServer Sizing Rules of Thumb
+
+Lars Hofhansl wrote a great
+link:http://hadoop-hbase.blogspot.com/2013/01/hbase-region-server-memory-sizing.html[blog post]
+about RegionServer memory sizing. The upshot is that you probably need more memory
+than you think you need. He goes into the impact of region size, memstore size, HDFS
+replication factor, and other things to check.
+
+[quote, Lars Hofhansl, http://hadoop-hbase.blogspot.com/2013/01/hbase-region-server-memory-sizing.html]
+____
+Personally I would place the maximum disk space per machine that can be served
+exclusively with HBase around 6T, unless you have a very read-heavy workload.
+In that case the Java heap should be 32GB (20G regions, 128M memstores, the rest
+defaults).
+____
+
[[number.of.cfs]]
== On the number of column families
@@ -187,7 +231,7 @@ See this comic by IKai Lan on why monotonically increasing row keys are problema
The pile-up on a single region brought on by monotonically increasing keys can be mitigated by randomizing the input records to not be in sorted order, but in general it's best to avoid using a timestamp or a sequence (e.g. 1, 2, 3) as the row-key.
If you do need to upload time series data into HBase, you should study link:http://opentsdb.net/[OpenTSDB] as a successful example.
-It has a page describing the link: http://opentsdb.net/schema.html[schema] it uses in HBase.
+It has a page describing the link:http://opentsdb.net/schema.html[schema] it uses in HBase.
The key format in OpenTSDB is effectively [metric_type][event_timestamp], which would appear at first glance to contradict the previous advice about not using a timestamp as the key.
However, the difference is that the timestamp is not in the _lead_ position of the key, and the design assumption is that there are dozens or hundreds (or more) of different metric types.
Thus, even with a continual stream of input data with a mix of metric types, the Puts are distributed across various points of regions in the table.
@@ -339,8 +383,8 @@ As an example of why this is important, consider the example of using displayabl
The problem is that all the data is going to pile up in the first 2 regions and the last region thus creating a "lumpy" (and possibly "hot") region problem.
To understand why, refer to an link:http://www.asciitable.com[ASCII Table].
-'0' is byte 48, and 'f' is byte 102, but there is a huge gap in byte values (bytes 58 to 96) that will _never appear in this keyspace_ because the only values are [0-9] and [a-f]. Thus, the middle regions regions will never be used.
-To make pre-spliting work with this example keyspace, a custom definition of splits (i.e., and not relying on the built-in split method) is required.
+'0' is byte 48, and 'f' is byte 102, but there is a huge gap in byte values (bytes 58 to 96) that will _never appear in this keyspace_ because the only values are [0-9] and [a-f]. Thus, the middle regions will never be used.
+To make pre-splitting work with this example keyspace, a custom definition of splits (i.e., and not relying on the built-in split method) is required.
Lesson #1: Pre-splitting tables is generally a best practice, but you need to pre-split them in such a way that all the regions are accessible in the keyspace.
While this example demonstrated the problem with a hex-key keyspace, the same problem can happen with _any_ keyspace.
@@ -406,7 +450,7 @@ The minimum number of row versions parameter is used together with the time-to-l
HBase supports a "bytes-in/bytes-out" interface via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html[Put] and link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html[Result], so anything that can be converted to an array of bytes can be stored as a value.
Input could be strings, numbers, complex objects, or even images as long as they can rendered as bytes.
-There are practical limits to the size of values (e.g., storing 10-50MB objects in HBase would probably be too much to ask); search the mailling list for conversations on this topic.
+There are practical limits to the size of values (e.g., storing 10-50MB objects in HBase would probably be too much to ask); search the mailing list for conversations on this topic.
All rows in HBase conform to the <<datamodel>>, and that includes versioning.
Take that into consideration when making your design, as well as block size for the ColumnFamily.
@@ -514,7 +558,7 @@ ROW COLUMN+CELL
Notice how delete cells are let go.
-Now lets run the same test only with `KEEP_DELETED_CELLS` set on the table (you can do table or per-column-family):
+Now let's run the same test only with `KEEP_DELETED_CELLS` set on the table (you can do table or per-column-family):
[source]
----
@@ -605,7 +649,7 @@ However, don't try a full-scan on a large table like this from an application (i
[[secondary.indexes.periodic]]
=== Periodic-Update Secondary Index
-A secondary index could be created in an other table which is periodically updated via a MapReduce job.
+A secondary index could be created in another table which is periodically updated via a MapReduce job.
The job could be executed intra-day, but depending on load-strategy it could still potentially be out of sync with the main data table.
See <<mapreduce.example.readwrite,mapreduce.example.readwrite>> for more information.
@@ -632,8 +676,13 @@ For more information, see <<coprocessors,coprocessors>>
== Constraints
HBase currently supports 'constraints' in traditional (SQL) database parlance.
-The advised usage for Constraints is in enforcing business rules for attributes in the table (e.g. make sure values are in the range 1-10). Constraints could also be used to enforce referential integrity, but this is strongly discouraged as it will dramatically decrease the write throughput of the tables where integrity checking is enabled.
-Extensive documentation on using Constraints can be found at: link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/constraint[Constraint] since version 0.94.
+The advised usage for Constraints is in enforcing business rules for attributes
+in the table (e.g. make sure values are in the range 1-10). Constraints could
+also be used to enforce referential integrity, but this is strongly discouraged
+as it will dramatically decrease the write throughput of the tables where integrity
+checking is enabled. Extensive documentation on using Constraints can be found at
+link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/constraint/Constraint.html[Constraint]
+since version 0.94.
[[schema.casestudies]]
== Schema Design Case Studies
@@ -733,10 +782,12 @@ Composite Rowkey With Numeric Substitution:
For this approach another lookup table would be needed in addition to LOG_DATA, called LOG_TYPES.
The rowkey of LOG_TYPES would be:
-* [type] (e.g., byte indicating hostname vs. event-type)
-* [bytes] variable length bytes for raw hostname or event-type.
+* `[type]` (e.g., byte indicating hostname vs. event-type)
+* `[bytes]` variable length bytes for raw hostname or event-type.
-A column for this rowkey could be a long with an assigned number, which could be obtained by using an link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#incrementColumnValue%28byte[],%20byte[],%20byte[],%20long%29[HBase counter].
+A column for this rowkey could be a long with an assigned number, which could be obtained
+by using an
++++<a href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#incrementColumnValue%28byte[],%20byte[],%20byte[],%20long%29">HBase counter</a>+++.
So the resulting composite rowkey would be:
@@ -751,7 +802,9 @@ In either the Hash or Numeric substitution approach, the raw values for hostname
This effectively is the OpenTSDB approach.
What OpenTSDB does is re-write data and pack rows into columns for certain time-periods.
-For a detailed explanation, see: link:http://opentsdb.net/schema.html, and link:http://www.cloudera.com/content/cloudera/en/resources/library/hbasecon/video-hbasecon-2012-lessons-learned-from-opentsdb.html[Lessons Learned from OpenTSDB] from HBaseCon2012.
+For a detailed explanation, see: http://opentsdb.net/schema.html, and
++++<a href="http://www.cloudera.com/content/cloudera/en/resources/library/hbasecon/video-hbasecon-2012-lessons-learned-from-opentsdb.html">Lessons Learned from OpenTSDB</a>+++
+from HBaseCon2012.
But this is how the general concept works: data is ingested, for example, in this manner...
@@ -796,7 +849,7 @@ Assuming that the combination of customer number and sales order uniquely identi
[customer number][order number]
----
-for a ORDER table.
+for an ORDER table.
However, there are more design decisions to make: are the _raw_ values the best choices for rowkeys?
The same design questions in the Log Data use-case confront us here.
@@ -854,14 +907,14 @@ The ORDER table's rowkey was described above: <<schema.casestudies.custorder,sch
The SHIPPING_LOCATION's composite rowkey would be something like this:
-* [order-rowkey]
-* [shipping location number] (e.g., 1st location, 2nd, etc.)
+* `[order-rowkey]`
+* `[shipping location number]` (e.g., 1st location, 2nd, etc.)
The LINE_ITEM table's composite rowkey would be something like this:
-* [order-rowkey]
-* [shipping location number] (e.g., 1st location, 2nd, etc.)
-* [line item number] (e.g., 1st lineitem, 2nd, etc.)
+* `[order-rowkey]`
+* `[shipping location number]` (e.g., 1st location, 2nd, etc.)
+* `[line item number]` (e.g., 1st lineitem, 2nd, etc.)
Such a normalized model is likely to be the approach with an RDBMS, but that's not your only option with HBase.
The cons of such an approach is that to retrieve information about any Order, you will need:
@@ -879,21 +932,21 @@ With this approach, there would exist a single table ORDER that would contain
The Order rowkey was described above: <<schema.casestudies.custorder,schema.casestudies.custorder>>
-* [order-rowkey]
-* [ORDER record type]
+* `[order-rowkey]`
+* `[ORDER record type]`
The ShippingLocation composite rowkey would be something like this:
-* [order-rowkey]
-* [SHIPPING record type]
-* [shipping location number] (e.g., 1st location, 2nd, etc.)
+* `[order-rowkey]`
+* `[SHIPPING record type]`
+* `[shipping location number]` (e.g., 1st location, 2nd, etc.)
The LineItem composite rowkey would be something like this:
-* [order-rowkey]
-* [LINE record type]
-* [shipping location number] (e.g., 1st location, 2nd, etc.)
-* [line item number] (e.g., 1st lineitem, 2nd, etc.)
+* `[order-rowkey]`
+* `[LINE record type]`
+* `[shipping location number]` (e.g., 1st location, 2nd, etc.)
+* `[line item number]` (e.g., 1st lineitem, 2nd, etc.)
[[schema.casestudies.custorder.obj.denorm]]
===== Denormalized
@@ -902,9 +955,9 @@ A variant of the Single Table With Record Types approach is to denormalize and f
The LineItem composite rowkey would be something like this:
-* [order-rowkey]
-* [LINE record type]
-* [line item number] (e.g., 1st lineitem, 2nd, etc., care must be taken that there are unique across the entire order)
+* `[order-rowkey]`
+* `[LINE record type]`
+* `[line item number]` (e.g., 1st lineitem, 2nd, etc., care must be taken that there are unique across the entire order)
and the LineItem columns would be something like this:
@@ -927,9 +980,9 @@ For example, the ORDER table's rowkey was described above: <<schema.casestudies.
There are many options here: JSON, XML, Java Serialization, Avro, Hadoop Writables, etc.
All of them are variants of the same approach: encode the object graph to a byte-array.
-Care should be taken with this approach to ensure backward compatibilty in case the object model changes such that older persisted structures can still be read back out of HBase.
+Care should be taken with this approach to ensure backward compatibility in case the object model changes such that older persisted structures can still be read back out of HBase.
-Pros are being able to manage complex object graphs with minimal I/O (e.g., a single HBase Get per Order in this example), but the cons include the aforementioned warning about backward compatiblity of serialization, language dependencies of serialization (e.g., Java Serialization only works with Java clients), the fact that you have to deserialize the entire object to get any piece of information inside the BLOB, and the difficulty in getting frameworks like Hive to work with custom objects like this.
+Pros are being able to manage complex object graphs with minimal I/O (e.g., a single HBase Get per Order in this example), but the cons include the aforementioned warning about backward compatibility of serialization, language dependencies of serialization (e.g., Java Serialization only works with Java clients), the fact that you have to deserialize the entire object to get any piece of information inside the BLOB, and the difficulty in getting frameworks like Hive to work with custom objects like this.
[[schema.smackdown]]
=== Case Study - "Tall/Wide/Middle" Schema Design Smackdown
@@ -941,7 +994,7 @@ These are general guidelines and not laws - each application must consider its o
==== Rows vs. Versions
A common question is whether one should prefer rows or HBase's built-in-versioning.
-The context is typically where there are "a lot" of versions of a row to be retained (e.g., where it is significantly above the HBase default of 1 max versions). The rows-approach would require storing a timestamp in some portion of the rowkey so that they would not overwite with each successive update.
+The context is typically where there are "a lot" of versions of a row to be retained (e.g., where it is significantly above the HBase default of 1 max versions). The rows-approach would require storing a timestamp in some portion of the rowkey so that they would not overwrite with each successive update.
Preference: Rows (generally speaking).
@@ -1040,14 +1093,14 @@ The tl;dr version is that you should probably go with one row per user+value, an
Your two options mirror a common question people have when designing HBase schemas: should I go "tall" or "wide"? Your first schema is "tall": each row represents one value for one user, and so there are many rows in the table for each user; the row key is user + valueid, and there would be (presumably) a single column qualifier that means "the value". This is great if you want to scan over rows in sorted order by row key (thus my question above, about whether these ids are sorted correctly). You can start a scan at any user+valueid, read the next 30, and be done.
What you're giving up is the ability to have transactional guarantees around all the rows for one user, but it doesn't sound like you need that.
-Doing it this way is generally recommended (see here link:http://hbase.apache.org/book.html#schema.smackdown).
+Doing it this way is generally recommended (see here http://hbase.apache.org/book.html#schema.smackdown).
Your second option is "wide": you store a bunch of values in one row, using different qualifiers (where the qualifier is the valueid). The simple way to do that would be to just store ALL values for one user in a single row.
I'm guessing you jumped to the "paginated" version because you're assuming that storing millions of columns in a single row would be bad for performance, which may or may not be true; as long as you're not trying to do too much in a single request, or do things like scanning over and returning all of the cells in the row, it shouldn't be fundamentally worse.
The client has methods that allow you to get specific slices of columns.
Note that neither case fundamentally uses more disk space than the other; you're just "shifting" part of the identifying information for a value either to the left (into the row key, in option one) or to the right (into the column qualifiers in option 2). Under the covers, every key/value still stores the whole row key, and column family name.
-(If this is a bit confusing, take an hour and watch Lars George's excellent video about understanding HBase schema design: link:http://www.youtube.com/watch?v=_HLoH_PgrLk).
+(If this is a bit confusing, take an hour and watch Lars George's excellent video about understanding HBase schema design: http://www.youtube.com/watch?v=_HLoH_PgrLk).
A manually paginated version has lots more complexities, as you note, like having to keep track of how many things are in each page, re-shuffling if new values are inserted, etc.
That seems significantly more complex.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/security.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc
index fb2a6b0..c346435 100644
--- a/src/main/asciidoc/_chapters/security.adoc
+++ b/src/main/asciidoc/_chapters/security.adoc
@@ -236,39 +236,41 @@ To enable it, do the following.
<<security.gateway.thrift>> describes how to configure the Thrift gateway to authenticate to HBase on the client's behalf, and to access HBase using a proxy user. The limitation of this approach is that after the client is initialized with a particular set of credentials, it cannot change these credentials during the session. The `doAs` feature provides a flexible way to impersonate multiple principals using the same client. This feature was implemented in link:https://issues.apache.org/jira/browse/HBASE-12640[HBASE-12640] for Thrift 1, but is currently not available for Thrift 2.
-*To allow proxy users*, add the following to the _hbase-site.xml_ file for every HBase node:
+*To enable the `doAs` feature*, add the following to the _hbase-site.xml_ file for every Thrift gateway:
[source,xml]
----
<property>
- <name>hadoop.security.authorization</name>
+ <name>hbase.regionserver.thrift.http</name>
<value>true</value>
</property>
<property>
- <name>hadoop.proxyuser.$USER.groups</name>
- <value>$GROUPS</value>
-</property>
-<property>
- <name>hadoop.proxyuser.$USER.hosts</name>
- <value>$GROUPS</value>
+ <name>hbase.thrift.support.proxyuser</name>
+ <value>true/value>
</property>
----
-*To enable the `doAs` feature*, add the following to the _hbase-site.xml_ file for every Thrift gateway:
+*To allow proxy users* when using `doAs` impersonation, add the following to the _hbase-site.xml_ file for every HBase node:
[source,xml]
----
<property>
- <name>hbase.regionserver.thrift.http</name>
+ <name>hadoop.security.authorization</name>
<value>true</value>
</property>
<property>
- <name>hbase.thrift.support.proxyuser</name>
- <value>true/value>
+ <name>hadoop.proxyuser.$USER.groups</name>
+ <value>$GROUPS</value>
+</property>
+<property>
+ <name>hadoop.proxyuser.$USER.hosts</name>
+ <value>$GROUPS</value>
</property>
----
-Take a look at the link:https://github.com/apache/hbase/blob/master/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java[demo client] to get an overall idea of how to use this feature in your client.
+Take a look at the
+link:https://github.com/apache/hbase/blob/master/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java[demo client]
+to get an overall idea of how to use this feature in your client.
=== Client-side Configuration for Secure Operation - REST Gateway
@@ -306,6 +308,10 @@ To enable REST gateway Kerberos authentication for client access, add the follow
[source,xml]
----
<property>
+ <name>hbase.rest.support.proxyuser</name>
+ <value>true</value>
+</property>
+<property>
<name>hbase.rest.authentication.type</name>
<value>kerberos</value>
</property>
@@ -331,7 +337,7 @@ To enable REST gateway Kerberos authentication for client access, add the follow
Substitute the keytab for HTTP for _$KEYTAB_.
HBase REST gateway supports different 'hbase.rest.authentication.type': simple, kerberos.
-You can also implement a custom authentication by implemening Hadoop AuthenticationHandler, then specify the full class name as 'hbase.rest.authentication.type' value.
+You can also implement a custom authentication by implementing Hadoop AuthenticationHandler, then specify the full class name as 'hbase.rest.authentication.type' value.
For more information, refer to link:http://hadoop.apache.org/docs/stable/hadoop-auth/index.html[SPNEGO HTTP authentication].
[[security.rest.gateway]]
@@ -343,7 +349,7 @@ To the HBase server, all requests are from the REST gateway user.
The actual users are unknown.
You can turn on the impersonation support.
With impersonation, the REST gateway user is a proxy user.
-The HBase server knows the acutal/real user of each request.
+The HBase server knows the actual/real user of each request.
So it can apply proper authorizations.
To turn on REST gateway impersonation, we need to configure HBase servers (masters and region servers) to allow proxy users; configure REST gateway to enable impersonation.
@@ -1117,7 +1123,7 @@ NOTE: Visibility labels are not currently applied for superusers.
| Interpretation
| fulltime
-| Allow accesss to users associated with the fulltime label.
+| Allow access to users associated with the fulltime label.
| !public
| Allow access to users not associated with the public label.
@@ -1332,11 +1338,21 @@ static Table createTableAndWriteDataWithLabels(TableName tableName, String... la
----
====
-<<reading_cells_with_labels>>
+[[reading_cells_with_labels]]
==== Reading Cells with Labels
-When you issue a Scan or Get, HBase uses your default set of authorizations to filter out cells that you do not have access to. A superuser can set the default set of authorizations for a given user by using the `set_auths` HBase Shell command or the link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.html#setAuths(org.apache.hadoop.hbase.client.Connection,%20java.lang.String[],%20java.lang.String)[VisibilityClient.setAuths()] method.
-You can specify a different authorization during the Scan or Get, by passing the AUTHORIZATIONS option in HBase Shell, or the link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setAuthorizations%28org.apache.hadoop.hbase.security.visibility.Authorizations%29[setAuthorizations()] method if you use the API. This authorization will be combined with your default set as an additional filter. It will further filter your results, rather than giving you additional authorization.
+When you issue a Scan or Get, HBase uses your default set of authorizations to
+filter out cells that you do not have access to. A superuser can set the default
+set of authorizations for a given user by using the `set_auths` HBase Shell command
+or the
+link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.html#setAuths(org.apache.hadoop.hbase.client.Connection,%20java.lang.String\[\],%20java.lang.String)[VisibilityClient.setAuths()] method.
+
+You can specify a different authorization during the Scan or Get, by passing the
+AUTHORIZATIONS option in HBase Shell, or the
+link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setAuthorizations%28org.apache.hadoop.hbase.security.visibility.Authorizations%29[setAuthorizations()]
+method if you use the API. This authorization will be combined with your default
+set as an additional filter. It will further filter your results, rather than
+giving you additional authorization.
.HBase Shell
====
@@ -1582,8 +1598,10 @@ Rotate the Master Key::
=== Secure Bulk Load
Bulk loading in secure mode is a bit more involved than normal setup, since the client has to transfer the ownership of the files generated from the MapReduce job to HBase.
-Secure bulk loading is implemented by a coprocessor, named link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.html
-[SecureBulkLoadEndpoint], which uses a staging directory configured by the configuration property `hbase.bulkload.staging.dir`, which defaults to _/tmp/hbase-staging/_.
+Secure bulk loading is implemented by a coprocessor, named
+link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.html[SecureBulkLoadEndpoint],
+which uses a staging directory configured by the configuration property `hbase.bulkload.staging.dir`, which defaults to
+_/tmp/hbase-staging/_.
.Secure Bulk Load Algorithm
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/shell.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/shell.adoc b/src/main/asciidoc/_chapters/shell.adoc
index 237089e..a4237fd 100644
--- a/src/main/asciidoc/_chapters/shell.adoc
+++ b/src/main/asciidoc/_chapters/shell.adoc
@@ -76,7 +76,7 @@ NOTE: Spawning HBase Shell commands in this way is slow, so keep that in mind wh
.Passing Commands to the HBase Shell
====
-You can pass commands to the HBase Shell in non-interactive mode (see <<hbasee.shell.noninteractive,hbasee.shell.noninteractive>>) using the `echo` command and the `|` (pipe) operator.
+You can pass commands to the HBase Shell in non-interactive mode (see <<hbase.shell.noninteractive,hbase.shell.noninteractive>>) using the `echo` command and the `|` (pipe) operator.
Be sure to escape characters in the HBase commands which would otherwise be interpreted by the shell.
Some debug-level output has been truncated from the example below.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/spark.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/spark.adoc b/src/main/asciidoc/_chapters/spark.adoc
new file mode 100644
index 0000000..37503e9
--- /dev/null
+++ b/src/main/asciidoc/_chapters/spark.adoc
@@ -0,0 +1,451 @@
+////
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ . . http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+////
+
+[[spark]]
+= HBase and Spark
+:doctype: book
+:numbered:
+:toc: left
+:icons: font
+:experimental:
+
+link:http://spark.apache.org/[Apache Spark] is a software framework that is used
+to process data in memory in a distributed manner, and is replacing MapReduce in
+many use cases.
+
+Spark itself is out of scope of this document, please refer to the Spark site for
+more information on the Spark project and subprojects. This document will focus
+on 4 main interaction points between Spark and HBase. Those interaction points are:
+
+Basic Spark::
+ The ability to have an HBase Connection at any point in your Spark DAG.
+Spark Streaming::
+ The ability to have an HBase Connection at any point in your Spark Streaming
+ application.
+Spark Bulk Load::
+ The ability to write directly to HBase HFiles for bulk insertion into HBase
+SparkSQL/DataFrames::
+ The ability to write SparkSQL that draws on tables that are represented in HBase.
+
+The following sections will walk through examples of all these interaction points.
+
+== Basic Spark
+
+This section discusses Spark HBase integration at the lowest and simplest levels.
+All the other interaction points are built upon the concepts that will be described
+here.
+
+At the root of all Spark and HBase integration is the HBaseContext. The HBaseContext
+takes in HBase configurations and pushes them to the Spark executors. This allows
+us to have an HBase Connection per Spark Executor in a static location.
+
+For reference, Spark Executors can be on the same nodes as the Region Servers or
+on different nodes there is no dependence of co-location. Think of every Spark
+Executor as a multi-threaded client application. This allows any Spark Tasks
+running on the executors to access the shared Connection object.
+
+.HBaseContext Usage Example
+====
+
+This example shows how HBaseContext can be used to do a `foreachPartition` on a RDD
+in Scala:
+
+[source, scala]
+----
+val sc = new SparkContext("local", "test")
+val config = new HBaseConfiguration()
+
+...
+
+val hbaseContext = new HBaseContext(sc, config)
+
+rdd.hbaseForeachPartition(hbaseContext, (it, conn) => {
+ val bufferedMutator = conn.getBufferedMutator(TableName.valueOf("t1"))
+ it.foreach((putRecord) => {
+. val put = new Put(putRecord._1)
+. putRecord._2.foreach((putValue) => put.addColumn(putValue._1, putValue._2, putValue._3))
+. bufferedMutator.mutate(put)
+ })
+ bufferedMutator.flush()
+ bufferedMutator.close()
+})
+----
+
+Here is the same example implemented in Java:
+
+[source, java]
+----
+JavaSparkContext jsc = new JavaSparkContext(sparkConf);
+
+try {
+ List<byte[]> list = new ArrayList<>();
+ list.add(Bytes.toBytes("1"));
+ ...
+ list.add(Bytes.toBytes("5"));
+
+ JavaRDD<byte[]> rdd = jsc.parallelize(list);
+ Configuration conf = HBaseConfiguration.create();
+
+ JavaHBaseContext hbaseContext = new JavaHBaseContext(jsc, conf);
+
+ hbaseContext.foreachPartition(rdd,
+ new VoidFunction<Tuple2<Iterator<byte[]>, Connection>>() {
+ public void call(Tuple2<Iterator<byte[]>, Connection> t)
+ throws Exception {
+ Table table = t._2().getTable(TableName.valueOf(tableName));
+ BufferedMutator mutator = t._2().getBufferedMutator(TableName.valueOf(tableName));
+ while (t._1().hasNext()) {
+ byte[] b = t._1().next();
+ Result r = table.get(new Get(b));
+ if (r.getExists()) {
+ mutator.mutate(new Put(b));
+ }
+ }
+
+ mutator.flush();
+ mutator.close();
+ table.close();
+ }
+ });
+} finally {
+ jsc.stop();
+}
+----
+====
+
+All functionality between Spark and HBase will be supported both in Scala and in
+Java, with the exception of SparkSQL which will support any language that is
+supported by Spark. For the remaining of this documentation we will focus on
+Scala examples for now.
+
+The examples above illustrate how to do a foreachPartition with a connection. A
+number of other Spark base functions are supported out of the box:
+
+// tag::spark_base_functions[]
+`bulkPut`:: For massively parallel sending of puts to HBase
+`bulkDelete`:: For massively parallel sending of deletes to HBase
+`bulkGet`:: For massively parallel sending of gets to HBase to create a new RDD
+`mapPartition`:: To do a Spark Map function with a Connection object to allow full
+access to HBase
+`hBaseRDD`:: To simplify a distributed scan to create a RDD
+// end::spark_base_functions[]
+
+For examples of all these functionalities, see the HBase-Spark Module.
+
+== Spark Streaming
+http://spark.apache.org/streaming/[Spark Streaming] is a micro batching stream
+processing framework built on top of Spark. HBase and Spark Streaming make great
+companions in that HBase can help serve the following benefits alongside Spark
+Streaming.
+
+* A place to grab reference data or profile data on the fly
+* A place to store counts or aggregates in a way that supports Spark Streaming
+promise of _only once processing_.
+
+The HBase-Spark module’s integration points with Spark Streaming are similar to
+its normal Spark integration points, in that the following commands are possible
+straight off a Spark Streaming DStream.
+
+include::spark.adoc[tags=spark_base_functions]
+
+.`bulkPut` Example with DStreams
+====
+
+Below is an example of bulkPut with DStreams. It is very close in feel to the RDD
+bulk put.
+
+[source, scala]
+----
+val sc = new SparkContext("local", "test")
+val config = new HBaseConfiguration()
+
+val hbaseContext = new HBaseContext(sc, config)
+val ssc = new StreamingContext(sc, Milliseconds(200))
+
+val rdd1 = ...
+val rdd2 = ...
+
+val queue = mutable.Queue[RDD[(Array[Byte], Array[(Array[Byte],
+ Array[Byte], Array[Byte])])]]()
+
+queue += rdd1
+queue += rdd2
+
+val dStream = ssc.queueStream(queue)
+
+dStream.hbaseBulkPut(
+ hbaseContext,
+ TableName.valueOf(tableName),
+ (putRecord) => {
+ val put = new Put(putRecord._1)
+ putRecord._2.foreach((putValue) => put.addColumn(putValue._1, putValue._2, putValue._3))
+ put
+ })
+----
+
+There are three inputs to the `hbaseBulkPut` function.
+. The hbaseContext that carries the configuration boardcast information link us
+to the HBase Connections in the executors
+. The table name of the table we are putting data into
+. A function that will convert a record in the DStream into an HBase Put object.
+====
+
+== Bulk Load
+
+Spark bulk load follows very closely to the MapReduce implementation of bulk
+load. In short, a partitioner partitions based on region splits and
+the row keys are sent to the reducers in order, so that HFiles can be written
+out. In Spark terms, the bulk load will be focused around a
+`repartitionAndSortWithinPartitions` followed by a `foreachPartition`.
+
+The only major difference with the Spark implementation compared to the
+MapReduce implementation is that the column qualifier is included in the shuffle
+ordering process. This was done because the MapReduce bulk load implementation
+would have memory issues with loading rows with a large numbers of columns, as a
+result of the sorting of those columns being done in the memory of the reducer JVM.
+Instead, that ordering is done in the Spark Shuffle, so there should no longer
+be a limit to the number of columns in a row for bulk loading.
+
+.Bulk Loading Example
+====
+
+The following example shows bulk loading with Spark.
+
+[source, scala]
+----
+val sc = new SparkContext("local", "test")
+val config = new HBaseConfiguration()
+
+val hbaseContext = new HBaseContext(sc, config)
+
+val stagingFolder = ...
+
+rdd.hbaseBulkLoad(TableName.valueOf(tableName),
+ t => {
+ val rowKey = t._1
+ val family:Array[Byte] = t._2(0)._1
+ val qualifier = t._2(0)._2
+ val value = t._2(0)._3
+
+ val keyFamilyQualifier= new KeyFamilyQualifier(rowKey, family, qualifier)
+
+ Seq((keyFamilyQualifier, value)).iterator
+ },
+ stagingFolder.getPath)
+
+val load = new LoadIncrementalHFiles(config)
+load.doBulkLoad(new Path(stagingFolder.getPath),
+ conn.getAdmin, table, conn.getRegionLocator(TableName.valueOf(tableName)))
+----
+====
+
+The `hbaseBulkLoad` function takes three required parameters:
+
+. The table name of the table we intend to bulk load too
+
+. A function that will convert a record in the RDD to a tuple key value par. With
+the tuple key being a KeyFamilyQualifer object and the value being the cell value.
+The KeyFamilyQualifer object will hold the RowKey, Column Family, and Column Qualifier.
+The shuffle will partition on the RowKey but will sort by all three values.
+
+. The temporary path for the HFile to be written out too
+
+Following the Spark bulk load command, use the HBase's LoadIncrementalHFiles object
+to load the newly created HFiles into HBase.
+
+.Additional Parameters for Bulk Loading with Spark
+
+You can set the following attributes with additional parameter options on hbaseBulkLoad.
+
+* Max file size of the HFiles
+* A flag to exclude HFiles from compactions
+* Column Family settings for compression, bloomType, blockSize, and dataBlockEncoding
+
+.Using Additional Parameters
+====
+
+[source, scala]
+----
+val sc = new SparkContext("local", "test")
+val config = new HBaseConfiguration()
+
+val hbaseContext = new HBaseContext(sc, config)
+
+val stagingFolder = ...
+
+val familyHBaseWriterOptions = new java.util.HashMap[Array[Byte], FamilyHFileWriteOptions]
+val f1Options = new FamilyHFileWriteOptions("GZ", "ROW", 128, "PREFIX")
+
+familyHBaseWriterOptions.put(Bytes.toBytes("columnFamily1"), f1Options)
+
+rdd.hbaseBulkLoad(TableName.valueOf(tableName),
+ t => {
+ val rowKey = t._1
+ val family:Array[Byte] = t._2(0)._1
+ val qualifier = t._2(0)._2
+ val value = t._2(0)._3
+
+ val keyFamilyQualifier= new KeyFamilyQualifier(rowKey, family, qualifier)
+
+ Seq((keyFamilyQualifier, value)).iterator
+ },
+ stagingFolder.getPath,
+ familyHBaseWriterOptions,
+ compactionExclude = false,
+ HConstants.DEFAULT_MAX_FILE_SIZE)
+
+val load = new LoadIncrementalHFiles(config)
+load.doBulkLoad(new Path(stagingFolder.getPath),
+ conn.getAdmin, table, conn.getRegionLocator(TableName.valueOf(tableName)))
+----
+====
+
+== SparkSQL/DataFrames
+
+http://spark.apache.org/sql/[SparkSQL] is a subproject of Spark that supports
+SQL that will compute down to a Spark DAG. In addition,SparkSQL is a heavy user
+of DataFrames. DataFrames are like RDDs with schema information.
+
+The HBase-Spark module includes support for Spark SQL and DataFrames, which allows
+you to write SparkSQL directly on HBase tables. In addition the HBase-Spark
+will push down query filtering logic to HBase.
+
+=== Predicate Push Down
+
+There are two examples of predicate push down in the HBase-Spark implementation.
+The first example shows the push down of filtering logic on the RowKey. HBase-Spark
+will reduce the filters on RowKeys down to a set of Get and/or Scan commands.
+
+NOTE: The Scans are distributed scans, rather than a single client scan operation.
+
+If the query looks something like the following, the logic will push down and get
+the rows through 3 Gets and 0 Scans. We can do gets because all the operations
+are `equal` operations.
+
+[source,sql]
+----
+SELECT
+ KEY_FIELD,
+ B_FIELD,
+ A_FIELD
+FROM hbaseTmp
+WHERE (KEY_FIELD = 'get1' or KEY_FIELD = 'get2' or KEY_FIELD = 'get3')
+----
+
+Now let's look at an example where we will end up doing two scans on HBase.
+
+[source, sql]
+----
+SELECT
+ KEY_FIELD,
+ B_FIELD,
+ A_FIELD
+FROM hbaseTmp
+WHERE KEY_FIELD < 'get2' or KEY_FIELD > 'get3'
+----
+
+In this example we will get 0 Gets and 2 Scans. One scan will load everything
+from the first row in the table until “get2” and the second scan will get
+everything from “get3” until the last row in the table.
+
+The next query is a good example of having a good deal of range checks. However
+the ranges overlap. To the code will be smart enough to get the following data
+in a single scan that encompasses all the data asked by the query.
+
+[source, sql]
+----
+SELECT
+ KEY_FIELD,
+ B_FIELD,
+ A_FIELD
+FROM hbaseTmp
+WHERE
+ (KEY_FIELD >= 'get1' and KEY_FIELD <= 'get3') or
+ (KEY_FIELD > 'get3' and KEY_FIELD <= 'get5')
+----
+
+The second example of push down functionality offered by the HBase-Spark module
+is the ability to push down filter logic for column and cell fields. Just like
+the RowKey logic, all query logic will be consolidated into the minimum number
+of range checks and equal checks by sending a Filter object along with the Scan
+with information about consolidated push down predicates
+
+.SparkSQL Code Example
+====
+This example shows how we can interact with HBase with SQL.
+
+[source, scala]
+----
+val sc = new SparkContext("local", "test")
+val config = new HBaseConfiguration()
+
+new HBaseContext(sc, TEST_UTIL.getConfiguration)
+val sqlContext = new SQLContext(sc)
+
+df = sqlContext.load("org.apache.hadoop.hbase.spark",
+ Map("hbase.columns.mapping" ->
+ "KEY_FIELD STRING :key, A_FIELD STRING c:a, B_FIELD STRING c:b",
+ "hbase.table" -> "t1"))
+
+df.registerTempTable("hbaseTmp")
+
+val results = sqlContext.sql("SELECT KEY_FIELD, B_FIELD FROM hbaseTmp " +
+ "WHERE " +
+ "(KEY_FIELD = 'get1' and B_FIELD < '3') or " +
+ "(KEY_FIELD >= 'get3' and B_FIELD = '8')").take(5)
+----
+
+There are three major parts of this example that deserve explaining.
+
+The sqlContext.load function::
+ In the sqlContext.load function we see two
+ parameters. The first of these parameters is pointing Spark to the HBase
+ DefaultSource class that will act as the interface between SparkSQL and HBase.
+
+A map of key value pairs::
+ In this example we have two keys in our map, `hbase.columns.mapping` and
+ `hbase.table`. The `hbase.table` directs SparkSQL to use the given HBase table.
+ The `hbase.columns.mapping` key give us the logic to translate HBase columns to
+ SparkSQL columns.
++
+The `hbase.columns.mapping` is a string that follows the following format
++
+[source, scala]
+----
+(SparkSQL.ColumnName) (SparkSQL.ColumnType) (HBase.ColumnFamily):(HBase.Qualifier)
+----
++
+In the example below we see the definition of three fields. Because KEY_FIELD has
+no ColumnFamily, it is the RowKey.
++
+----
+KEY_FIELD STRING :key, A_FIELD STRING c:a, B_FIELD STRING c:b
+----
+
+The registerTempTable function::
+ This is a SparkSQL function that allows us now to be free of Scala when accessing
+ our HBase table directly with SQL with the table name of "hbaseTmp".
+
+The last major point to note in the example is the `sqlContext.sql` function, which
+allows the user to ask their questions in SQL which will be pushed down to the
+DefaultSource code in the HBase-Spark module. The result of this command will be
+a DataFrame with the Schema of KEY_FIELD and B_FIELD.
+====
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/tracing.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/tracing.adoc b/src/main/asciidoc/_chapters/tracing.adoc
index 9b3711e..0cddd8a 100644
--- a/src/main/asciidoc/_chapters/tracing.adoc
+++ b/src/main/asciidoc/_chapters/tracing.adoc
@@ -31,12 +31,12 @@
:experimental:
link:https://issues.apache.org/jira/browse/HBASE-6449[HBASE-6449] added support for tracing requests through HBase, using the open source tracing library, link:http://htrace.incubator.apache.org/[HTrace].
-Setting up tracing is quite simple, however it currently requires some very minor changes to your client code (it would not be very difficult to remove this requirement).
+Setting up tracing is quite simple, however it currently requires some very minor changes to your client code (it would not be very difficult to remove this requirement).
[[tracing.spanreceivers]]
=== SpanReceivers
-The tracing system works by collecting information in structures called 'Spans'. It is up to you to choose how you want to receive this information by implementing the `SpanReceiver` interface, which defines one method:
+The tracing system works by collecting information in structures called 'Spans'. It is up to you to choose how you want to receive this information by implementing the `SpanReceiver` interface, which defines one method:
[source]
----
@@ -45,12 +45,12 @@ public void receiveSpan(Span span);
----
This method serves as a callback whenever a span is completed.
-HTrace allows you to use as many SpanReceivers as you want so you can easily send trace information to multiple destinations.
+HTrace allows you to use as many SpanReceivers as you want so you can easily send trace information to multiple destinations.
-Configure what SpanReceivers you'd like to us by putting a comma separated list of the fully-qualified class name of classes implementing `SpanReceiver` in _hbase-site.xml_ property: `hbase.trace.spanreceiver.classes`.
+Configure what SpanReceivers you'd like to us by putting a comma separated list of the fully-qualified class name of classes implementing `SpanReceiver` in _hbase-site.xml_ property: `hbase.trace.spanreceiver.classes`.
HTrace includes a `LocalFileSpanReceiver` that writes all span information to local files in a JSON-based format.
-The `LocalFileSpanReceiver` looks in _hbase-site.xml_ for a `hbase.local-file-span-receiver.path` property with a value describing the name of the file to which nodes should write their span information.
+The `LocalFileSpanReceiver` looks in _hbase-site.xml_ for a `hbase.local-file-span-receiver.path` property with a value describing the name of the file to which nodes should write their span information.
[source]
----
@@ -65,7 +65,7 @@ The `LocalFileSpanReceiver` looks in _hbase-site.xml_ for a `hbase.local-fi
</property>
----
-HTrace also provides `ZipkinSpanReceiver` which converts spans to link:http://github.com/twitter/zipkin[Zipkin] span format and send them to Zipkin server. In order to use this span receiver, you need to install the jar of htrace-zipkin to your HBase's classpath on all of the nodes in your cluster.
+HTrace also provides `ZipkinSpanReceiver` which converts spans to link:http://github.com/twitter/zipkin[Zipkin] span format and send them to Zipkin server. In order to use this span receiver, you need to install the jar of htrace-zipkin to your HBase's classpath on all of the nodes in your cluster.
_htrace-zipkin_ is published to the link:http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.htrace%22%20AND%20a%3A%22htrace-zipkin%22[Maven central repository]. You could get the latest version from there or just build it locally (see the link:http://htrace.incubator.apache.org/[HTrace] homepage for information on how to do this) and then copy it out to all nodes.
@@ -77,11 +77,11 @@ _htrace-zipkin_ is published to the link:http://search.maven.org/#search%7Cgav%7
<property>
<name>hbase.trace.spanreceiver.classes</name>
<value>org.apache.htrace.impl.ZipkinSpanReceiver</value>
-</property>
+</property>
<property>
<name>hbase.htrace.zipkin.collector-hostname</name>
<value>localhost</value>
-</property>
+</property>
<property>
<name>hbase.htrace.zipkin.collector-port</name>
<value>9410</value>
@@ -93,7 +93,7 @@ If you do not want to use the included span receivers, you are encouraged to wri
[[tracing.client.modifications]]
== Client Modifications
-In order to turn on tracing in your client code, you must initialize the module sending spans to receiver once per client process.
+In order to turn on tracing in your client code, you must initialize the module sending spans to receiver once per client process.
[source,java]
----
@@ -107,7 +107,7 @@ private SpanReceiverHost spanReceiverHost;
----
Then you simply start tracing span before requests you think are interesting, and close it when the request is done.
-For example, if you wanted to trace all of your get operations, you change this:
+For example, if you wanted to trace all of your get operations, you change this:
[source,java]
----
@@ -118,7 +118,7 @@ Get get = new Get(Bytes.toBytes("r1"));
Result res = table.get(get);
----
-into:
+into:
[source,java]
----
@@ -133,7 +133,7 @@ try {
}
----
-If you wanted to trace half of your 'get' operations, you would pass in:
+If you wanted to trace half of your 'get' operations, you would pass in:
[source,java]
----
@@ -142,12 +142,12 @@ new ProbabilitySampler(0.5)
----
in lieu of `Sampler.ALWAYS` to `Trace.startSpan()`.
-See the HTrace _README_ for more information on Samplers.
+See the HTrace _README_ for more information on Samplers.
[[tracing.client.shell]]
== Tracing from HBase Shell
-You can use `trace` command for tracing requests from HBase Shell. `trace 'start'` command turns on tracing and `trace 'stop'` command turns off tracing.
+You can use `trace` command for tracing requests from HBase Shell. `trace 'start'` command turns on tracing and `trace 'stop'` command turns off tracing.
[source]
----
@@ -158,7 +158,7 @@ hbase(main):003:0> trace 'stop'
----
`trace 'start'` and `trace 'stop'` always returns boolean value representing if or not there is ongoing tracing.
-As a result, `trace 'stop'` returns false on success. `trace 'status'` just returns if or not tracing is turned on.
+As a result, `trace 'stop'` returns false on success. `trace 'status'` just returns if or not tracing is turned on.
[source]
----
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/troubleshooting.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/troubleshooting.adoc b/src/main/asciidoc/_chapters/troubleshooting.adoc
index 8ae61b4..e372760 100644
--- a/src/main/asciidoc/_chapters/troubleshooting.adoc
+++ b/src/main/asciidoc/_chapters/troubleshooting.adoc
@@ -89,11 +89,11 @@ Additionally, each DataNode server will also have a TaskTracker/NodeManager log
[[rpc.logging]]
==== Enabling RPC-level logging
-Enabling the RPC-level logging on a RegionServer can often given insight on timings at the server.
+Enabling the RPC-level logging on a RegionServer can often give insight on timings at the server.
Once enabled, the amount of log spewed is voluminous.
It is not recommended that you leave this logging on for more than short bursts of time.
To enable RPC-level logging, browse to the RegionServer UI and click on _Log Level_.
-Set the log level to `DEBUG` for the package `org.apache.hadoop.ipc` (Thats right, for `hadoop.ipc`, NOT, `hbase.ipc`). Then tail the RegionServers log.
+Set the log level to `DEBUG` for the package `org.apache.hadoop.ipc` (That's right, for `hadoop.ipc`, NOT, `hbase.ipc`). Then tail the RegionServers log.
Analyze.
To disable, set the logging level back to `INFO` level.
@@ -185,7 +185,7 @@ The key points here is to keep all these pauses low.
CMS pauses are always low, but if your ParNew starts growing, you can see minor GC pauses approach 100ms, exceed 100ms and hit as high at 400ms.
This can be due to the size of the ParNew, which should be relatively small.
-If your ParNew is very large after running HBase for a while, in one example a ParNew was about 150MB, then you might have to constrain the size of ParNew (The larger it is, the longer the collections take but if its too small, objects are promoted to old gen too quickly). In the below we constrain new gen size to 64m.
+If your ParNew is very large after running HBase for a while, in one example a ParNew was about 150MB, then you might have to constrain the size of ParNew (The larger it is, the longer the collections take but if it's too small, objects are promoted to old gen too quickly). In the below we constrain new gen size to 64m.
Add the below line in _hbase-env.sh_:
[source,bourne]
@@ -443,7 +443,7 @@ java.lang.Thread.State: WAITING (on object monitor)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)
----
-A handler thread that's waiting for stuff to do (like put, delete, scan, etc):
+A handler thread that's waiting for stuff to do (like put, delete, scan, etc.):
[source]
----
@@ -849,7 +849,7 @@ are snapshots and WALs.
Snapshots::
When you create a snapshot, HBase retains everything it needs to recreate the table's
- state at that time of tne snapshot. This includes deleted cells or expired versions.
+ state at that time of the snapshot. This includes deleted cells or expired versions.
For this reason, your snapshot usage pattern should be well-planned, and you should
prune snapshots that you no longer need. Snapshots are stored in `/hbase/.snapshots`,
and archives needed to restore snapshots are stored in
@@ -1070,7 +1070,7 @@ However, if the NotServingRegionException is logged ERROR, then the client ran o
Fix your DNS.
In versions of Apache HBase before 0.92.x, reverse DNS needs to give same answer as forward lookup.
-See link:https://issues.apache.org/jira/browse/HBASE-3431[HBASE 3431 RegionServer is not using the name given it by the master; double entry in master listing of servers] for gorey details.
+See link:https://issues.apache.org/jira/browse/HBASE-3431[HBASE 3431 RegionServer is not using the name given it by the master; double entry in master listing of servers] for gory details.
[[brand.new.compressor]]
==== Logs flooded with '2011-01-10 12:40:48,407 INFO org.apache.hadoop.io.compress.CodecPool: Gotbrand-new compressor' messages
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/unit_testing.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/unit_testing.adoc b/src/main/asciidoc/_chapters/unit_testing.adoc
index 3f70001..6f13864 100644
--- a/src/main/asciidoc/_chapters/unit_testing.adoc
+++ b/src/main/asciidoc/_chapters/unit_testing.adoc
@@ -47,7 +47,7 @@ public class MyHBaseDAO {
Put put = createPut(obj);
table.put(put);
}
-
+
private static Put createPut(HBaseTestObj obj) {
Put put = new Put(Bytes.toBytes(obj.getRowKey()));
put.add(Bytes.toBytes("CF"), Bytes.toBytes("CQ-1"),
@@ -96,13 +96,13 @@ public class TestMyHbaseDAOData {
These tests ensure that your `createPut` method creates, populates, and returns a `Put` object with expected values.
Of course, JUnit can do much more than this.
-For an introduction to JUnit, see link:https://github.com/junit-team/junit/wiki/Getting-started.
+For an introduction to JUnit, see https://github.com/junit-team/junit/wiki/Getting-started.
== Mockito
Mockito is a mocking framework.
It goes further than JUnit by allowing you to test the interactions between objects without having to replicate the entire environment.
-You can read more about Mockito at its project site, link:https://code.google.com/p/mockito/.
+You can read more about Mockito at its project site, https://code.google.com/p/mockito/.
You can use Mockito to do unit testing on smaller units.
For instance, you can mock a `org.apache.hadoop.hbase.Server` instance or a `org.apache.hadoop.hbase.master.MasterServices` interface reference rather than a full-blown `org.apache.hadoop.hbase.master.HMaster`.
@@ -133,7 +133,7 @@ public class TestMyHBaseDAO{
Configuration config = HBaseConfiguration.create();
@Mock
Connection connection = ConnectionFactory.createConnection(config);
- @Mock
+ @Mock
private Table table;
@Captor
private ArgumentCaptor putCaptor;
@@ -150,7 +150,7 @@ public class TestMyHBaseDAO{
MyHBaseDAO.insertRecord(table, obj);
verify(table).put(putCaptor.capture());
Put put = putCaptor.getValue();
-
+
assertEquals(Bytes.toString(put.getRow()), obj.getRowKey());
assert(put.has(Bytes.toBytes("CF"), Bytes.toBytes("CQ-1")));
assert(put.has(Bytes.toBytes("CF"), Bytes.toBytes("CQ-2")));
@@ -182,7 +182,7 @@ public class MyReducer extends TableReducer<Text, Text, ImmutableBytesWritable>
public static final byte[] CF = "CF".getBytes();
public static final byte[] QUALIFIER = "CQ-1".getBytes();
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
- //bunch of processing to extract data to be inserted, in our case, lets say we are simply
+ //bunch of processing to extract data to be inserted, in our case, let's say we are simply
//appending all the records we receive from the mapper for this particular
//key and insert one record into HBase
StringBuffer data = new StringBuffer();
@@ -197,7 +197,7 @@ public class MyReducer extends TableReducer<Text, Text, ImmutableBytesWritable>
}
----
-To test this code, the first step is to add a dependency to MRUnit to your Maven POM file.
+To test this code, the first step is to add a dependency to MRUnit to your Maven POM file.
[source,xml]
----
@@ -225,16 +225,16 @@ public class MyReducerTest {
MyReducer reducer = new MyReducer();
reduceDriver = ReduceDriver.newReduceDriver(reducer);
}
-
+
@Test
public void testHBaseInsert() throws IOException {
- String strKey = "RowKey-1", strValue = "DATA", strValue1 = "DATA1",
+ String strKey = "RowKey-1", strValue = "DATA", strValue1 = "DATA1",
strValue2 = "DATA2";
List<Text> list = new ArrayList<Text>();
list.add(new Text(strValue));
list.add(new Text(strValue1));
list.add(new Text(strValue2));
- //since in our case all that the reducer is doing is appending the records that the mapper
+ //since in our case all that the reducer is doing is appending the records that the mapper
//sends it, we should get the following back
String expectedOutput = strValue + strValue1 + strValue2;
//Setup Input, mimic what mapper would have passed
@@ -242,10 +242,10 @@ strValue2 = "DATA2";
reduceDriver.withInput(new Text(strKey), list);
//run the reducer and get its output
List<Pair<ImmutableBytesWritable, Writable>> result = reduceDriver.run();
-
+
//extract key from result and verify
assertEquals(Bytes.toString(result.get(0).getFirst().get()), strKey);
-
+
//extract value for CF/QUALIFIER and verify
Put a = (Put)result.get(0).getSecond();
String c = Bytes.toString(a.get(CF, QUALIFIER).get(0).getValue());
@@ -259,7 +259,7 @@ Your MRUnit test verifies that the output is as expected, the Put that is insert
MRUnit includes a MapperDriver to test mapping jobs, and you can use MRUnit to test other operations, including reading from HBase, processing data, or writing to HDFS,
-== Integration Testing with a HBase Mini-Cluster
+== Integration Testing with an HBase Mini-Cluster
HBase ships with HBaseTestingUtility, which makes it easy to write integration tests using a [firstterm]_mini-cluster_.
The first step is to add some dependencies to your Maven POM file.
@@ -283,7 +283,7 @@ Check the versions to be sure they are appropriate.
<type>test-jar</type>
<scope>test</scope>
</dependency>
-
+
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
@@ -309,7 +309,7 @@ public class MyHBaseIntegrationTest {
private static HBaseTestingUtility utility;
byte[] CF = "CF".getBytes();
byte[] QUALIFIER = "CQ-1".getBytes();
-
+
@Before
public void setup() throws Exception {
utility = new HBaseTestingUtility();
@@ -343,7 +343,7 @@ This code creates an HBase mini-cluster and starts it.
Next, it creates a table called `MyTest` with one column family, `CF`.
A record is inserted, a Get is performed from the same table, and the insertion is verified.
-NOTE: Starting the mini-cluster takes about 20-30 seconds, but that should be appropriate for integration testing.
+NOTE: Starting the mini-cluster takes about 20-30 seconds, but that should be appropriate for integration testing.
To use an HBase mini-cluster on Microsoft Windows, you need to use a Cygwin environment.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/upgrading.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/upgrading.adoc b/src/main/asciidoc/_chapters/upgrading.adoc
index 13c3c0e..6327c5a 100644
--- a/src/main/asciidoc/_chapters/upgrading.adoc
+++ b/src/main/asciidoc/_chapters/upgrading.adoc
@@ -132,7 +132,7 @@ HBase Client API::
[[hbase.limitetprivate.api]]
HBase LimitedPrivate API::
- LimitedPrivate annotation comes with a set of target consumers for the interfaces. Those consumers are coprocessors, phoenix, replication endpoint implemnetations or similar. At this point, HBase only guarantees source and binary compatibility for these interfaces between patch versions.
+ LimitedPrivate annotation comes with a set of target consumers for the interfaces. Those consumers are coprocessors, phoenix, replication endpoint implementations or similar. At this point, HBase only guarantees source and binary compatibility for these interfaces between patch versions.
[[hbase.private.api]]
HBase Private API::
@@ -158,7 +158,7 @@ When we say two HBase versions are compatible, we mean that the versions are wir
A rolling upgrade is the process by which you update the servers in your cluster a server at a time. You can rolling upgrade across HBase versions if they are binary or wire compatible. See <<hbase.rolling.restart>> for more on what this means. Coarsely, a rolling upgrade is a graceful stop each server, update the software, and then restart. You do this for each server in the cluster. Usually you upgrade the Master first and then the RegionServers. See <<rolling>> for tools that can help use the rolling upgrade process.
-For example, in the below, HBase was symlinked to the actual HBase install. On upgrade, before running a rolling restart over the cluser, we changed the symlink to point at the new HBase software version and then ran
+For example, in the below, HBase was symlinked to the actual HBase install. On upgrade, before running a rolling restart over the cluster, we changed the symlink to point at the new HBase software version and then ran
[source,bash]
----
@@ -200,7 +200,7 @@ ports.
[[upgrade1.0.hbase.bucketcache.percentage.in.combinedcache]]
.hbase.bucketcache.percentage.in.combinedcache configuration has been REMOVED
-You may have made use of this configuration if you are using BucketCache. If NOT using BucketCache, this change does not effect you. Its removal means that your L1 LruBlockCache is now sized using `hfile.block.cache.size` -- i.e. the way you would size the on-heap L1 LruBlockCache if you were NOT doing BucketCache -- and the BucketCache size is not whatever the setting for `hbase.bucketcache.size` is. You may need to adjust configs to get the LruBlockCache and BucketCache sizes set to what they were in 0.98.x and previous. If you did not set this config., its default value was 0.9. If you do nothing, your BucketCache will increase in size by 10%. Your L1 LruBlockCache will become `hfile.block.cache.size` times your java heap size (`hfile.block.cache.size` is a float between 0.0 and 1.0). To read more, see link:https://issues.apache.org/jira/browse/HBASE-11520[HBASE-11520 Simplify offheap cache config by removing the confusing "hbase.bucketcache.percentage.in.combinedcache"].
+You may have made use of this configuration if you are using BucketCache. If NOT using BucketCache, this change does not affect you. Its removal means that your L1 LruBlockCache is now sized using `hfile.block.cache.size` -- i.e. the way you would size the on-heap L1 LruBlockCache if you were NOT doing BucketCache -- and the BucketCache size is not whatever the setting for `hbase.bucketcache.size` is. You may need to adjust configs to get the LruBlockCache and BucketCache sizes set to what they were in 0.98.x and previous. If you did not set this config., its default value was 0.9. If you do nothing, your BucketCache will increase in size by 10%. Your L1 LruBlockCache will become `hfile.block.cache.size` times your java heap size (`hfile.block.cache.size` is a float between 0.0 and 1.0). To read more, see link:https://issues.apache.org/jira/browse/HBASE-11520[HBASE-11520 Simplify offheap cache config by removing the confusing "hbase.bucketcache.percentage.in.combinedcache"].
[[hbase-12068]]
.If you have your own customer filters.
@@ -392,7 +392,7 @@ The migration is a one-time event. However, every time your cluster starts, `MET
[[upgrade0.94]]
=== Upgrading from 0.92.x to 0.94.x
-We used to think that 0.92 and 0.94 were interface compatible and that you can do a rolling upgrade between these versions but then we figured that link:https://issues.apache.org/jira/browse/HBASE-5357[HBASE-5357 Use builder pattern in HColumnDescriptor] changed method signatures so rather than return `void` they instead return `HColumnDescriptor`. This will throw`java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V` so 0.92 and 0.94 are NOT compatible. You cannot do a rolling upgrade between them.
+We used to think that 0.92 and 0.94 were interface compatible and that you can do a rolling upgrade between these versions but then we figured that link:https://issues.apache.org/jira/browse/HBASE-5357[HBASE-5357 Use builder pattern in HColumnDescriptor] changed method signatures so rather than return `void` they instead return `HColumnDescriptor`. This will throw `java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V` so 0.92 and 0.94 are NOT compatible. You cannot do a rolling upgrade between them.
[[upgrade0.92]]
=== Upgrading from 0.90.x to 0.92.x
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/zookeeper.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/zookeeper.adoc b/src/main/asciidoc/_chapters/zookeeper.adoc
index 3266964..2319360 100644
--- a/src/main/asciidoc/_chapters/zookeeper.adoc
+++ b/src/main/asciidoc/_chapters/zookeeper.adoc
@@ -45,7 +45,7 @@ HBase does not ship with a _zoo.cfg_ so you will need to browse the _conf_ direc
You must at least list the ensemble servers in _hbase-site.xml_ using the `hbase.zookeeper.quorum` property.
This property defaults to a single ensemble member at `localhost` which is not suitable for a fully distributed HBase.
-(It binds to the local machine only and remote clients will not be able to connect).
+(It binds to the local machine only and remote clients will not be able to connect).
.How many ZooKeepers should I run?
[NOTE]
@@ -54,7 +54,7 @@ You can run a ZooKeeper ensemble that comprises 1 node only but in production it
Also, run an odd number of machines.
In ZooKeeper, an even number of peers is supported, but it is normally not used because an even sized ensemble requires, proportionally, more peers to form a quorum than an odd sized ensemble requires.
For example, an ensemble with 4 peers requires 3 to form a quorum, while an ensemble with 5 also requires 3 to form a quorum.
-Thus, an ensemble of 5 allows 2 peers to fail, and thus is more fault tolerant than the ensemble of 4, which allows only 1 down peer.
+Thus, an ensemble of 5 allows 2 peers to fail, and thus is more fault tolerant than the ensemble of 4, which allows only 1 down peer.
Give each ZooKeeper server around 1GB of RAM, and if possible, its own dedicated disk (A dedicated disk is the best thing you can do to ensure a performant ZooKeeper ensemble). For very heavily loaded clusters, run ZooKeeper servers on separate machines from RegionServers (DataNodes and TaskTrackers).
====
@@ -97,12 +97,12 @@ In the example below we have ZooKeeper persist to _/user/local/zookeeper_.
</configuration>
----
-.What verion of ZooKeeper should I use?
+.What version of ZooKeeper should I use?
[CAUTION]
====
The newer version, the better.
For example, some folks have been bitten by link:https://issues.apache.org/jira/browse/ZOOKEEPER-1277[ZOOKEEPER-1277].
-If running zookeeper 3.5+, you can ask hbase to make use of the new multi operation by enabling <<hbase.zookeeper.usemulti,hbase.zookeeper.useMulti>>" in your _hbase-site.xml_.
+If running zookeeper 3.5+, you can ask hbase to make use of the new multi operation by enabling <<hbase.zookeeper.usemulti,hbase.zookeeper.useMulti>>" in your _hbase-site.xml_.
====
.ZooKeeper Maintenance
@@ -140,7 +140,7 @@ Just make sure to set `HBASE_MANAGES_ZK` to `false` if you want it to stay
For more information about running a distinct ZooKeeper cluster, see the ZooKeeper link:http://hadoop.apache.org/zookeeper/docs/current/zookeeperStarted.html[Getting
Started Guide].
Additionally, see the link:http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7[ZooKeeper Wiki] or the link:http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup[ZooKeeper
- documentation] for more information on ZooKeeper sizing.
+ documentation] for more information on ZooKeeper sizing.
[[zk.sasl.auth]]
== SASL Authentication with ZooKeeper
@@ -148,24 +148,24 @@ Additionally, see the link:http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7[ZooKee
Newer releases of Apache HBase (>= 0.92) will support connecting to a ZooKeeper Quorum that supports SASL authentication (which is available in Zookeeper versions 3.4.0 or later).
This describes how to set up HBase to mutually authenticate with a ZooKeeper Quorum.
-ZooKeeper/HBase mutual authentication (link:https://issues.apache.org/jira/browse/HBASE-2418[HBASE-2418]) is required as part of a complete secure HBase configuration (link:https://issues.apache.org/jira/browse/HBASE-3025[HBASE-3025]). For simplicity of explication, this section ignores additional configuration required (Secure HDFS and Coprocessor configuration). It's recommended to begin with an HBase-managed Zookeeper configuration (as opposed to a standalone Zookeeper quorum) for ease of learning.
+ZooKeeper/HBase mutual authentication (link:https://issues.apache.org/jira/browse/HBASE-2418[HBASE-2418]) is required as part of a complete secure HBase configuration (link:https://issues.apache.org/jira/browse/HBASE-3025[HBASE-3025]). For simplicity of explication, this section ignores additional configuration required (Secure HDFS and Coprocessor configuration). It's recommended to begin with an HBase-managed Zookeeper configuration (as opposed to a standalone Zookeeper quorum) for ease of learning.
=== Operating System Prerequisites
You need to have a working Kerberos KDC setup.
For each `$HOST` that will run a ZooKeeper server, you should have a principle `zookeeper/$HOST`.
For each such host, add a service key (using the `kadmin` or `kadmin.local` tool's `ktadd` command) for `zookeeper/$HOST` and copy this file to `$HOST`, and make it readable only to the user that will run zookeeper on `$HOST`.
-Note the location of this file, which we will use below as _$PATH_TO_ZOOKEEPER_KEYTAB_.
+Note the location of this file, which we will use below as _$PATH_TO_ZOOKEEPER_KEYTAB_.
Similarly, for each `$HOST` that will run an HBase server (master or regionserver), you should have a principle: `hbase/$HOST`.
For each host, add a keytab file called _hbase.keytab_ containing a service key for `hbase/$HOST`, copy this file to `$HOST`, and make it readable only to the user that will run an HBase service on `$HOST`.
-Note the location of this file, which we will use below as _$PATH_TO_HBASE_KEYTAB_.
+Note the location of this file, which we will use below as _$PATH_TO_HBASE_KEYTAB_.
Each user who will be an HBase client should also be given a Kerberos principal.
This principal should usually have a password assigned to it (as opposed to, as with the HBase servers, a keytab file) which only this user knows.
The client's principal's `maxrenewlife` should be set so that it can be renewed enough so that the user can complete their HBase client processes.
For example, if a user runs a long-running HBase client process that takes at most 3 days, we might create this user's principal within `kadmin` with: `addprinc -maxrenewlife 3days`.
-The Zookeeper client and server libraries manage their own ticket refreshment by running threads that wake up periodically to do the refreshment.
+The Zookeeper client and server libraries manage their own ticket refreshment by running threads that wake up periodically to do the refreshment.
On each host that will run an HBase client (e.g. `hbase shell`), add the following file to the HBase home directory's _conf_ directory:
@@ -210,7 +210,7 @@ where the _$PATH_TO_HBASE_KEYTAB_ and _$PATH_TO_ZOOKEEPER_KEYTAB_ files are what
The `Server` section will be used by the Zookeeper quorum server, while the `Client` section will be used by the HBase master and regionservers.
The path to this file should be substituted for the text _$HBASE_SERVER_CONF_ in the _hbase-env.sh_ listing below.
-The path to this file should be substituted for the text _$CLIENT_CONF_ in the _hbase-env.sh_ listing below.
+The path to this file should be substituted for the text _$CLIENT_CONF_ in the _hbase-env.sh_ listing below.
Modify your _hbase-env.sh_ to include the following:
@@ -257,7 +257,7 @@ Modify your _hbase-site.xml_ on each node that will run zookeeper, master or reg
where `$ZK_NODES` is the comma-separated list of hostnames of the Zookeeper Quorum hosts.
-Start your hbase cluster by running one or more of the following set of commands on the appropriate hosts:
+Start your hbase cluster by running one or more of the following set of commands on the appropriate hosts:
----
@@ -344,7 +344,7 @@ Server {
----
where `$HOST` is the hostname of each Quorum host.
-We will refer to the full pathname of this file as _$ZK_SERVER_CONF_ below.
+We will refer to the full pathname of this file as _$ZK_SERVER_CONF_ below.
Start your Zookeepers on each Zookeeper Quorum host with:
@@ -354,7 +354,7 @@ Start your Zookeepers on each Zookeeper Quorum host with:
SERVER_JVMFLAGS="-Djava.security.auth.login.config=$ZK_SERVER_CONF" bin/zkServer start
----
-Start your HBase cluster by running one or more of the following set of commands on the appropriate nodes:
+Start your HBase cluster by running one or more of the following set of commands on the appropriate nodes:
----
@@ -415,7 +415,7 @@ mvn clean test -Dtest=TestZooKeeperACL
----
Then configure HBase as described above.
-Manually edit target/cached_classpath.txt (see below):
+Manually edit target/cached_classpath.txt (see below):
----
@@ -439,7 +439,7 @@ mv target/tmp.txt target/cached_classpath.txt
==== Set JAAS configuration programmatically
-This would avoid the need for a separate Hadoop jar that fixes link:https://issues.apache.org/jira/browse/HADOOP-7070[HADOOP-7070].
+This would avoid the need for a separate Hadoop jar that fixes link:https://issues.apache.org/jira/browse/HADOOP-7070[HADOOP-7070].
==== Elimination of `kerberos.removeHostFromPrincipal` and`kerberos.removeRealmFromPrincipal`
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/book.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/book.adoc b/src/main/asciidoc/book.adoc
index b2bd151..392ba2e 100644
--- a/src/main/asciidoc/book.adoc
+++ b/src/main/asciidoc/book.adoc
@@ -64,6 +64,7 @@ include::_chapters/architecture.adoc[]
include::_chapters/hbase_apis.adoc[]
include::_chapters/external_apis.adoc[]
include::_chapters/thrift_filter_language.adoc[]
+include::_chapters/spark.adoc[]
include::_chapters/cp.adoc[]
include::_chapters/performance.adoc[]
include::_chapters/troubleshooting.adoc[]
[2/4] hbase git commit: updating docs from master
Posted by nd...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/hbase-default.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/hbase-default.adoc b/src/main/asciidoc/_chapters/hbase-default.adoc
index 8df9b17..26929a3 100644
--- a/src/main/asciidoc/_chapters/hbase-default.adoc
+++ b/src/main/asciidoc/_chapters/hbase-default.adoc
@@ -46,7 +46,7 @@ Temporary directory on the local filesystem.
.Default
`${java.io.tmpdir}/hbase-${user.name}`
-
+
[[hbase.rootdir]]
*`hbase.rootdir`*::
+
@@ -64,7 +64,7 @@ The directory shared by region servers and into
.Default
`${hbase.tmp.dir}/hbase`
-
+
[[hbase.cluster.distributed]]
*`hbase.cluster.distributed`*::
+
@@ -77,7 +77,7 @@ The mode the cluster will be in. Possible values are
.Default
`false`
-
+
[[hbase.zookeeper.quorum]]
*`hbase.zookeeper.quorum`*::
+
@@ -97,7 +97,7 @@ Comma separated list of servers in the ZooKeeper ensemble
.Default
`localhost`
-
+
[[hbase.local.dir]]
*`hbase.local.dir`*::
+
@@ -108,7 +108,7 @@ Directory on the local filesystem to be used
.Default
`${hbase.tmp.dir}/local/`
-
+
[[hbase.master.info.port]]
*`hbase.master.info.port`*::
+
@@ -119,18 +119,18 @@ The port for the HBase Master web UI.
.Default
`16010`
-
+
[[hbase.master.info.bindAddress]]
*`hbase.master.info.bindAddress`*::
+
.Description
The bind address for the HBase Master web UI
-
+
+
.Default
`0.0.0.0`
-
+
[[hbase.master.logcleaner.plugins]]
*`hbase.master.logcleaner.plugins`*::
+
@@ -145,7 +145,7 @@ A comma-separated list of BaseLogCleanerDelegate invoked by
.Default
`org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner`
-
+
[[hbase.master.logcleaner.ttl]]
*`hbase.master.logcleaner.ttl`*::
+
@@ -156,7 +156,7 @@ Maximum time a WAL can stay in the .oldlogdir directory,
.Default
`600000`
-
+
[[hbase.master.hfilecleaner.plugins]]
*`hbase.master.hfilecleaner.plugins`*::
+
@@ -172,7 +172,7 @@ A comma-separated list of BaseHFileCleanerDelegate invoked by
.Default
`org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner`
-
+
[[hbase.master.catalog.timeout]]
*`hbase.master.catalog.timeout`*::
+
@@ -183,7 +183,7 @@ Timeout value for the Catalog Janitor from the master to
.Default
`600000`
-
+
[[hbase.master.infoserver.redirect]]
*`hbase.master.infoserver.redirect`*::
+
@@ -195,7 +195,7 @@ Whether or not the Master listens to the Master web
.Default
`true`
-
+
[[hbase.regionserver.port]]
*`hbase.regionserver.port`*::
+
@@ -205,7 +205,7 @@ The port the HBase RegionServer binds to.
.Default
`16020`
-
+
[[hbase.regionserver.info.port]]
*`hbase.regionserver.info.port`*::
+
@@ -216,7 +216,7 @@ The port for the HBase RegionServer web UI
.Default
`16030`
-
+
[[hbase.regionserver.info.bindAddress]]
*`hbase.regionserver.info.bindAddress`*::
+
@@ -226,7 +226,7 @@ The address for the HBase RegionServer web UI
.Default
`0.0.0.0`
-
+
[[hbase.regionserver.info.port.auto]]
*`hbase.regionserver.info.port.auto`*::
+
@@ -239,7 +239,7 @@ Whether or not the Master or RegionServer
.Default
`false`
-
+
[[hbase.regionserver.handler.count]]
*`hbase.regionserver.handler.count`*::
+
@@ -250,7 +250,7 @@ Count of RPC Listener instances spun up on RegionServers.
.Default
`30`
-
+
[[hbase.ipc.server.callqueue.handler.factor]]
*`hbase.ipc.server.callqueue.handler.factor`*::
+
@@ -262,7 +262,7 @@ Factor to determine the number of call queues.
.Default
`0.1`
-
+
[[hbase.ipc.server.callqueue.read.ratio]]
*`hbase.ipc.server.callqueue.read.ratio`*::
+
@@ -287,12 +287,12 @@ Split the call queues into read and write queues.
and 2 queues will contain only write requests.
a read.ratio of 1 means that: 9 queues will contain only read requests
and 1 queues will contain only write requests.
-
+
+
.Default
`0`
-
+
[[hbase.ipc.server.callqueue.scan.ratio]]
*`hbase.ipc.server.callqueue.scan.ratio`*::
+
@@ -313,12 +313,12 @@ Given the number of read call queues, calculated from the total number
and 4 queues will contain only short-read requests.
a scan.ratio of 0.8 means that: 6 queues will contain only long-read requests
and 2 queues will contain only short-read requests.
-
+
+
.Default
`0`
-
+
[[hbase.regionserver.msginterval]]
*`hbase.regionserver.msginterval`*::
+
@@ -329,7 +329,7 @@ Interval between messages from the RegionServer to Master
.Default
`3000`
-
+
[[hbase.regionserver.regionSplitLimit]]
*`hbase.regionserver.regionSplitLimit`*::
+
@@ -342,7 +342,7 @@ Limit for the number of regions after which no more region
.Default
`2147483647`
-
+
[[hbase.regionserver.logroll.period]]
*`hbase.regionserver.logroll.period`*::
+
@@ -353,7 +353,7 @@ Period at which we will roll the commit log regardless
.Default
`3600000`
-
+
[[hbase.regionserver.logroll.errors.tolerated]]
*`hbase.regionserver.logroll.errors.tolerated`*::
+
@@ -367,7 +367,7 @@ The number of consecutive WAL close errors we will allow
.Default
`2`
-
+
[[hbase.regionserver.hlog.reader.impl]]
*`hbase.regionserver.hlog.reader.impl`*::
+
@@ -377,7 +377,7 @@ The WAL file reader implementation.
.Default
`org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader`
-
+
[[hbase.regionserver.hlog.writer.impl]]
*`hbase.regionserver.hlog.writer.impl`*::
+
@@ -387,7 +387,7 @@ The WAL file writer implementation.
.Default
`org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter`
-
+
[[hbase.master.distributed.log.replay]]
*`hbase.master.distributed.log.replay`*::
+
@@ -397,13 +397,13 @@ Enable 'distributed log replay' as default engine splitting
back to the old mode 'distributed log splitter', set the value to
'false'. 'Disributed log replay' improves MTTR because it does not
write intermediate files. 'DLR' required that 'hfile.format.version'
- be set to version 3 or higher.
-
+ be set to version 3 or higher.
+
+
.Default
`true`
-
+
[[hbase.regionserver.global.memstore.size]]
*`hbase.regionserver.global.memstore.size`*::
+
@@ -416,20 +416,20 @@ Maximum size of all memstores in a region server before new
.Default
`0.4`
-
+
[[hbase.regionserver.global.memstore.size.lower.limit]]
*`hbase.regionserver.global.memstore.size.lower.limit`*::
+
.Description
Maximum size of all memstores in a region server before flushes are forced.
Defaults to 95% of hbase.regionserver.global.memstore.size.
- A 100% value for this value causes the minimum possible flushing to occur when updates are
+ A 100% value for this value causes the minimum possible flushing to occur when updates are
blocked due to memstore limiting.
+
.Default
`0.95`
-
+
[[hbase.regionserver.optionalcacheflushinterval]]
*`hbase.regionserver.optionalcacheflushinterval`*::
+
@@ -441,7 +441,7 @@ Maximum size of all memstores in a region server before flushes are forced.
.Default
`3600000`
-
+
[[hbase.regionserver.catalog.timeout]]
*`hbase.regionserver.catalog.timeout`*::
+
@@ -451,7 +451,7 @@ Timeout value for the Catalog Janitor from the regionserver to META.
.Default
`600000`
-
+
[[hbase.regionserver.dns.interface]]
*`hbase.regionserver.dns.interface`*::
+
@@ -462,7 +462,7 @@ The name of the Network Interface from which a region server
.Default
`default`
-
+
[[hbase.regionserver.dns.nameserver]]
*`hbase.regionserver.dns.nameserver`*::
+
@@ -474,7 +474,7 @@ The host name or IP address of the name server (DNS)
.Default
`default`
-
+
[[hbase.regionserver.region.split.policy]]
*`hbase.regionserver.region.split.policy`*::
+
@@ -483,12 +483,12 @@ The host name or IP address of the name server (DNS)
A split policy determines when a region should be split. The various other split policies that
are available currently are ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy,
DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc.
-
+
+
.Default
`org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy`
-
+
[[zookeeper.session.timeout]]
*`zookeeper.session.timeout`*::
+
@@ -497,17 +497,18 @@ ZooKeeper session timeout in milliseconds. It is used in two different ways.
First, this value is used in the ZK client that HBase uses to connect to the ensemble.
It is also used by HBase when it starts a ZK server and it is passed as the 'maxSessionTimeout'. See
http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions.
- For example, if a HBase region server connects to a ZK ensemble that's also managed by HBase, then the
+ For example, if an HBase region server connects to a ZK ensemble that's also managed
+ by HBase, then the
session timeout will be the one specified by this configuration. But, a region server that connects
to an ensemble managed with a different configuration will be subjected that ensemble's maxSessionTimeout. So,
even though HBase might propose using 90 seconds, the ensemble can have a max timeout lower than this and
it will take precedence. The current default that ZK ships with is 40 seconds, which is lower than HBase's.
-
+
+
.Default
`90000`
-
+
[[zookeeper.znode.parent]]
*`zookeeper.znode.parent`*::
+
@@ -520,7 +521,7 @@ Root ZNode for HBase in ZooKeeper. All of HBase's ZooKeeper
.Default
`/hbase`
-
+
[[zookeeper.znode.rootserver]]
*`zookeeper.znode.rootserver`*::
+
@@ -533,7 +534,7 @@ Path to ZNode holding root region location. This is written by
.Default
`root-region-server`
-
+
[[zookeeper.znode.acl.parent]]
*`zookeeper.znode.acl.parent`*::
+
@@ -543,7 +544,7 @@ Root ZNode for access control lists.
.Default
`acl`
-
+
[[hbase.zookeeper.dns.interface]]
*`hbase.zookeeper.dns.interface`*::
+
@@ -554,7 +555,7 @@ The name of the Network Interface from which a ZooKeeper server
.Default
`default`
-
+
[[hbase.zookeeper.dns.nameserver]]
*`hbase.zookeeper.dns.nameserver`*::
+
@@ -566,7 +567,7 @@ The host name or IP address of the name server (DNS)
.Default
`default`
-
+
[[hbase.zookeeper.peerport]]
*`hbase.zookeeper.peerport`*::
+
@@ -578,7 +579,7 @@ Port used by ZooKeeper peers to talk to each other.
.Default
`2888`
-
+
[[hbase.zookeeper.leaderport]]
*`hbase.zookeeper.leaderport`*::
+
@@ -590,7 +591,7 @@ Port used by ZooKeeper for leader election.
.Default
`3888`
-
+
[[hbase.zookeeper.useMulti]]
*`hbase.zookeeper.useMulti`*::
+
@@ -616,7 +617,7 @@ Property from ZooKeeper's config zoo.cfg.
.Default
`10`
-
+
[[hbase.zookeeper.property.syncLimit]]
*`hbase.zookeeper.property.syncLimit`*::
+
@@ -628,7 +629,7 @@ Property from ZooKeeper's config zoo.cfg.
.Default
`5`
-
+
[[hbase.zookeeper.property.dataDir]]
*`hbase.zookeeper.property.dataDir`*::
+
@@ -639,7 +640,7 @@ Property from ZooKeeper's config zoo.cfg.
.Default
`${hbase.tmp.dir}/zookeeper`
-
+
[[hbase.zookeeper.property.clientPort]]
*`hbase.zookeeper.property.clientPort`*::
+
@@ -650,7 +651,7 @@ Property from ZooKeeper's config zoo.cfg.
.Default
`2181`
-
+
[[hbase.zookeeper.property.maxClientCnxns]]
*`hbase.zookeeper.property.maxClientCnxns`*::
+
@@ -664,7 +665,7 @@ Property from ZooKeeper's config zoo.cfg.
.Default
`300`
-
+
[[hbase.client.write.buffer]]
*`hbase.client.write.buffer`*::
+
@@ -679,7 +680,7 @@ Default size of the HTable client write buffer in bytes.
.Default
`2097152`
-
+
[[hbase.client.pause]]
*`hbase.client.pause`*::
+
@@ -692,7 +693,7 @@ General client pause value. Used mostly as value to wait
.Default
`100`
-
+
[[hbase.client.retries.number]]
*`hbase.client.retries.number`*::
+
@@ -707,7 +708,7 @@ Maximum retries. Used as maximum for all retryable
.Default
`35`
-
+
[[hbase.client.max.total.tasks]]
*`hbase.client.max.total.tasks`*::
+
@@ -718,7 +719,7 @@ The maximum number of concurrent tasks a single HTable instance will
.Default
`100`
-
+
[[hbase.client.max.perserver.tasks]]
*`hbase.client.max.perserver.tasks`*::
+
@@ -729,7 +730,7 @@ The maximum number of concurrent tasks a single HTable instance will
.Default
`5`
-
+
[[hbase.client.max.perregion.tasks]]
*`hbase.client.max.perregion.tasks`*::
+
@@ -742,7 +743,7 @@ The maximum number of concurrent connections the client will
.Default
`1`
-
+
[[hbase.client.scanner.caching]]
*`hbase.client.scanner.caching`*::
+
@@ -757,7 +758,7 @@ Number of rows that will be fetched when calling next
.Default
`100`
-
+
[[hbase.client.keyvalue.maxsize]]
*`hbase.client.keyvalue.maxsize`*::
+
@@ -772,7 +773,7 @@ Specifies the combined maximum allowed size of a KeyValue
.Default
`10485760`
-
+
[[hbase.client.scanner.timeout.period]]
*`hbase.client.scanner.timeout.period`*::
+
@@ -782,7 +783,7 @@ Client scanner lease period in milliseconds.
.Default
`60000`
-
+
[[hbase.client.localityCheck.threadPoolSize]]
*`hbase.client.localityCheck.threadPoolSize`*::
+
@@ -792,7 +793,7 @@ Client scanner lease period in milliseconds.
.Default
`2`
-
+
[[hbase.bulkload.retries.number]]
*`hbase.bulkload.retries.number`*::
+
@@ -804,7 +805,7 @@ Maximum retries. This is maximum number of iterations
.Default
`10`
-
+
[[hbase.balancer.period
]]
*`hbase.balancer.period
@@ -816,7 +817,7 @@ Period at which the region balancer runs in the Master.
.Default
`300000`
-
+
[[hbase.regions.slop]]
*`hbase.regions.slop`*::
+
@@ -826,7 +827,7 @@ Rebalance if any regionserver has average + (average * slop) regions.
.Default
`0.2`
-
+
[[hbase.server.thread.wakefrequency]]
*`hbase.server.thread.wakefrequency`*::
+
@@ -837,20 +838,20 @@ Time to sleep in between searches for work (in milliseconds).
.Default
`10000`
-
+
[[hbase.server.versionfile.writeattempts]]
*`hbase.server.versionfile.writeattempts`*::
+
.Description
How many time to retry attempting to write a version file
- before just aborting. Each attempt is seperated by the
+ before just aborting. Each attempt is separated by the
hbase.server.thread.wakefrequency milliseconds.
+
.Default
`3`
-
+
[[hbase.hregion.memstore.flush.size]]
*`hbase.hregion.memstore.flush.size`*::
+
@@ -863,7 +864,7 @@ Time to sleep in between searches for work (in milliseconds).
.Default
`134217728`
-
+
[[hbase.hregion.percolumnfamilyflush.size.lower.bound]]
*`hbase.hregion.percolumnfamilyflush.size.lower.bound`*::
+
@@ -876,12 +877,12 @@ Time to sleep in between searches for work (in milliseconds).
memstore size more than this, all the memstores will be flushed
(just as usual). This value should be less than half of the total memstore
threshold (hbase.hregion.memstore.flush.size).
-
+
+
.Default
`16777216`
-
+
[[hbase.hregion.preclose.flush.size]]
*`hbase.hregion.preclose.flush.size`*::
+
@@ -900,7 +901,7 @@ Time to sleep in between searches for work (in milliseconds).
.Default
`5242880`
-
+
[[hbase.hregion.memstore.block.multiplier]]
*`hbase.hregion.memstore.block.multiplier`*::
+
@@ -916,7 +917,7 @@ Time to sleep in between searches for work (in milliseconds).
.Default
`4`
-
+
[[hbase.hregion.memstore.mslab.enabled]]
*`hbase.hregion.memstore.mslab.enabled`*::
+
@@ -930,19 +931,19 @@ Time to sleep in between searches for work (in milliseconds).
.Default
`true`
-
+
[[hbase.hregion.max.filesize]]
*`hbase.hregion.max.filesize`*::
+
.Description
- Maximum HFile size. If the sum of the sizes of a region's HFiles has grown to exceed this
+ Maximum HFile size. If the sum of the sizes of a region's HFiles has grown to exceed this
value, the region is split in two.
+
.Default
`10737418240`
-
+
[[hbase.hregion.majorcompaction]]
*`hbase.hregion.majorcompaction`*::
+
@@ -959,7 +960,7 @@ Time between major compactions, expressed in milliseconds. Set to 0 to disable
.Default
`604800000`
-
+
[[hbase.hregion.majorcompaction.jitter]]
*`hbase.hregion.majorcompaction.jitter`*::
+
@@ -972,32 +973,32 @@ A multiplier applied to hbase.hregion.majorcompaction to cause compaction to occ
.Default
`0.50`
-
+
[[hbase.hstore.compactionThreshold]]
*`hbase.hstore.compactionThreshold`*::
+
.Description
- If more than this number of StoreFiles exist in any one Store
- (one StoreFile is written per flush of MemStore), a compaction is run to rewrite all
+ If more than this number of StoreFiles exist in any one Store
+ (one StoreFile is written per flush of MemStore), a compaction is run to rewrite all
StoreFiles into a single StoreFile. Larger values delay compaction, but when compaction does
occur, it takes longer to complete.
+
.Default
`3`
-
+
[[hbase.hstore.flusher.count]]
*`hbase.hstore.flusher.count`*::
+
.Description
The number of flush threads. With fewer threads, the MemStore flushes will be
queued. With more threads, the flushes will be executed in parallel, increasing the load on
- HDFS, and potentially causing more compactions.
+ HDFS, and potentially causing more compactions.
+
.Default
`2`
-
+
[[hbase.hstore.blockingStoreFiles]]
*`hbase.hstore.blockingStoreFiles`*::
+
@@ -1009,40 +1010,40 @@ A multiplier applied to hbase.hregion.majorcompaction to cause compaction to occ
.Default
`10`
-
+
[[hbase.hstore.blockingWaitTime]]
*`hbase.hstore.blockingWaitTime`*::
+
.Description
The time for which a region will block updates after reaching the StoreFile limit
- defined by hbase.hstore.blockingStoreFiles. After this time has elapsed, the region will stop
+ defined by hbase.hstore.blockingStoreFiles. After this time has elapsed, the region will stop
blocking updates even if a compaction has not been completed.
+
.Default
`90000`
-
+
[[hbase.hstore.compaction.min]]
*`hbase.hstore.compaction.min`*::
+
.Description
-The minimum number of StoreFiles which must be eligible for compaction before
- compaction can run. The goal of tuning hbase.hstore.compaction.min is to avoid ending up with
- too many tiny StoreFiles to compact. Setting this value to 2 would cause a minor compaction
+The minimum number of StoreFiles which must be eligible for compaction before
+ compaction can run. The goal of tuning hbase.hstore.compaction.min is to avoid ending up with
+ too many tiny StoreFiles to compact. Setting this value to 2 would cause a minor compaction
each time you have two StoreFiles in a Store, and this is probably not appropriate. If you
- set this value too high, all the other values will need to be adjusted accordingly. For most
+ set this value too high, all the other values will need to be adjusted accordingly. For most
cases, the default value is appropriate. In previous versions of HBase, the parameter
hbase.hstore.compaction.min was named hbase.hstore.compactionThreshold.
+
.Default
`3`
-
+
[[hbase.hstore.compaction.max]]
*`hbase.hstore.compaction.max`*::
+
.Description
-The maximum number of StoreFiles which will be selected for a single minor
+The maximum number of StoreFiles which will be selected for a single minor
compaction, regardless of the number of eligible StoreFiles. Effectively, the value of
hbase.hstore.compaction.max controls the length of time it takes a single compaction to
complete. Setting it larger means that more StoreFiles are included in a compaction. For most
@@ -1051,88 +1052,88 @@ The maximum number of StoreFiles which will be selected for a single minor
.Default
`10`
-
+
[[hbase.hstore.compaction.min.size]]
*`hbase.hstore.compaction.min.size`*::
+
.Description
-A StoreFile smaller than this size will always be eligible for minor compaction.
- HFiles this size or larger are evaluated by hbase.hstore.compaction.ratio to determine if
- they are eligible. Because this limit represents the "automatic include"limit for all
- StoreFiles smaller than this value, this value may need to be reduced in write-heavy
- environments where many StoreFiles in the 1-2 MB range are being flushed, because every
+A StoreFile smaller than this size will always be eligible for minor compaction.
+ HFiles this size or larger are evaluated by hbase.hstore.compaction.ratio to determine if
+ they are eligible. Because this limit represents the "automatic include"limit for all
+ StoreFiles smaller than this value, this value may need to be reduced in write-heavy
+ environments where many StoreFiles in the 1-2 MB range are being flushed, because every
StoreFile will be targeted for compaction and the resulting StoreFiles may still be under the
minimum size and require further compaction. If this parameter is lowered, the ratio check is
- triggered more quickly. This addressed some issues seen in earlier versions of HBase but
- changing this parameter is no longer necessary in most situations. Default: 128 MB expressed
+ triggered more quickly. This addressed some issues seen in earlier versions of HBase but
+ changing this parameter is no longer necessary in most situations. Default: 128 MB expressed
in bytes.
+
.Default
`134217728`
-
+
[[hbase.hstore.compaction.max.size]]
*`hbase.hstore.compaction.max.size`*::
+
.Description
-A StoreFile larger than this size will be excluded from compaction. The effect of
- raising hbase.hstore.compaction.max.size is fewer, larger StoreFiles that do not get
+A StoreFile larger than this size will be excluded from compaction. The effect of
+ raising hbase.hstore.compaction.max.size is fewer, larger StoreFiles that do not get
compacted often. If you feel that compaction is happening too often without much benefit, you
can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes.
+
.Default
`9223372036854775807`
-
+
[[hbase.hstore.compaction.ratio]]
*`hbase.hstore.compaction.ratio`*::
+
.Description
-For minor compaction, this ratio is used to determine whether a given StoreFile
+For minor compaction, this ratio is used to determine whether a given StoreFile
which is larger than hbase.hstore.compaction.min.size is eligible for compaction. Its
effect is to limit compaction of large StoreFiles. The value of hbase.hstore.compaction.ratio
- is expressed as a floating-point decimal. A large ratio, such as 10, will produce a single
- giant StoreFile. Conversely, a low value, such as .25, will produce behavior similar to the
+ is expressed as a floating-point decimal. A large ratio, such as 10, will produce a single
+ giant StoreFile. Conversely, a low value, such as .25, will produce behavior similar to the
BigTable compaction algorithm, producing four StoreFiles. A moderate value of between 1.0 and
- 1.4 is recommended. When tuning this value, you are balancing write costs with read costs.
- Raising the value (to something like 1.4) will have more write costs, because you will
- compact larger StoreFiles. However, during reads, HBase will need to seek through fewer
- StoreFiles to accomplish the read. Consider this approach if you cannot take advantage of
- Bloom filters. Otherwise, you can lower this value to something like 1.0 to reduce the
- background cost of writes, and use Bloom filters to control the number of StoreFiles touched
+ 1.4 is recommended. When tuning this value, you are balancing write costs with read costs.
+ Raising the value (to something like 1.4) will have more write costs, because you will
+ compact larger StoreFiles. However, during reads, HBase will need to seek through fewer
+ StoreFiles to accomplish the read. Consider this approach if you cannot take advantage of
+ Bloom filters. Otherwise, you can lower this value to something like 1.0 to reduce the
+ background cost of writes, and use Bloom filters to control the number of StoreFiles touched
during reads. For most cases, the default value is appropriate.
+
.Default
`1.2F`
-
+
[[hbase.hstore.compaction.ratio.offpeak]]
*`hbase.hstore.compaction.ratio.offpeak`*::
+
.Description
Allows you to set a different (by default, more aggressive) ratio for determining
- whether larger StoreFiles are included in compactions during off-peak hours. Works in the
- same way as hbase.hstore.compaction.ratio. Only applies if hbase.offpeak.start.hour and
+ whether larger StoreFiles are included in compactions during off-peak hours. Works in the
+ same way as hbase.hstore.compaction.ratio. Only applies if hbase.offpeak.start.hour and
hbase.offpeak.end.hour are also enabled.
+
.Default
`5.0F`
-
+
[[hbase.hstore.time.to.purge.deletes]]
*`hbase.hstore.time.to.purge.deletes`*::
+
.Description
-The amount of time to delay purging of delete markers with future timestamps. If
- unset, or set to 0, all delete markers, including those with future timestamps, are purged
- during the next major compaction. Otherwise, a delete marker is kept until the major compaction
+The amount of time to delay purging of delete markers with future timestamps. If
+ unset, or set to 0, all delete markers, including those with future timestamps, are purged
+ during the next major compaction. Otherwise, a delete marker is kept until the major compaction
which occurs after the marker's timestamp plus the value of this setting, in milliseconds.
-
+
+
.Default
`0`
-
+
[[hbase.offpeak.start.hour]]
*`hbase.offpeak.start.hour`*::
+
@@ -1143,7 +1144,7 @@ The start of off-peak hours, expressed as an integer between 0 and 23, inclusive
.Default
`-1`
-
+
[[hbase.offpeak.end.hour]]
*`hbase.offpeak.end.hour`*::
+
@@ -1154,7 +1155,7 @@ The end of off-peak hours, expressed as an integer between 0 and 23, inclusive.
.Default
`-1`
-
+
[[hbase.regionserver.thread.compaction.throttle]]
*`hbase.regionserver.thread.compaction.throttle`*::
+
@@ -1170,19 +1171,19 @@ There are two different thread pools for compactions, one for large compactions
.Default
`2684354560`
-
+
[[hbase.hstore.compaction.kv.max]]
*`hbase.hstore.compaction.kv.max`*::
+
.Description
The maximum number of KeyValues to read and then write in a batch when flushing or
compacting. Set this lower if you have big KeyValues and problems with Out Of Memory
- Exceptions Set this higher if you have wide, small rows.
+ Exceptions Set this higher if you have wide, small rows.
+
.Default
`10`
-
+
[[hbase.storescanner.parallel.seek.enable]]
*`hbase.storescanner.parallel.seek.enable`*::
+
@@ -1194,7 +1195,7 @@ The maximum number of KeyValues to read and then write in a batch when flushing
.Default
`false`
-
+
[[hbase.storescanner.parallel.seek.threads]]
*`hbase.storescanner.parallel.seek.threads`*::
+
@@ -1205,7 +1206,7 @@ The maximum number of KeyValues to read and then write in a batch when flushing
.Default
`10`
-
+
[[hfile.block.cache.size]]
*`hfile.block.cache.size`*::
+
@@ -1218,7 +1219,7 @@ Percentage of maximum heap (-Xmx setting) to allocate to block cache
.Default
`0.4`
-
+
[[hfile.block.index.cacheonwrite]]
*`hfile.block.index.cacheonwrite`*::
+
@@ -1229,7 +1230,7 @@ This allows to put non-root multi-level index blocks into the block
.Default
`false`
-
+
[[hfile.index.block.max.size]]
*`hfile.index.block.max.size`*::
+
@@ -1241,31 +1242,33 @@ When the size of a leaf-level, intermediate-level, or root-level
.Default
`131072`
-
+
[[hbase.bucketcache.ioengine]]
*`hbase.bucketcache.ioengine`*::
+
.Description
-Where to store the contents of the bucketcache. One of: onheap,
- offheap, or file. If a file, set it to file:PATH_TO_FILE. See https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/io/hfile/CacheConfig.html for more information.
-
+Where to store the contents of the bucketcache. One of: onheap,
+ offheap, or file. If a file, set it to file:PATH_TO_FILE.
+ See https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/hfile/CacheConfig.html
+ for more information.
+
+
.Default
``
-
+
[[hbase.bucketcache.combinedcache.enabled]]
*`hbase.bucketcache.combinedcache.enabled`*::
+
.Description
-Whether or not the bucketcache is used in league with the LRU
- on-heap block cache. In this mode, indices and blooms are kept in the LRU
+Whether or not the bucketcache is used in league with the LRU
+ on-heap block cache. In this mode, indices and blooms are kept in the LRU
blockcache and the data blocks are kept in the bucketcache.
+
.Default
`true`
-
+
[[hbase.bucketcache.size]]
*`hbase.bucketcache.size`*::
+
@@ -1276,19 +1279,19 @@ Used along with bucket cache, this is a float that EITHER represents a percentag
.Default
`0` when specified as a float
-
+
[[hbase.bucketcache.sizes]]
*`hbase.bucketcache.sizes`*::
+
.Description
-A comma-separated list of sizes for buckets for the bucketcache
- if you use multiple sizes. Should be a list of block sizes in order from smallest
+A comma-separated list of sizes for buckets for the bucketcache
+ if you use multiple sizes. Should be a list of block sizes in order from smallest
to largest. The sizes you use will depend on your data access patterns.
+
.Default
``
-
+
[[hfile.format.version]]
*`hfile.format.version`*::
+
@@ -1296,13 +1299,13 @@ A comma-separated list of sizes for buckets for the bucketcache
The HFile format version to use for new files.
Version 3 adds support for tags in hfiles (See http://hbase.apache.org/book.html#hbase.tags).
Distributed Log Replay requires that tags are enabled. Also see the configuration
- 'hbase.replication.rpc.codec'.
-
+ 'hbase.replication.rpc.codec'.
+
+
.Default
`3`
-
+
[[hfile.block.bloom.cacheonwrite]]
*`hfile.block.bloom.cacheonwrite`*::
+
@@ -1312,7 +1315,7 @@ Enables cache-on-write for inline blocks of a compound Bloom filter.
.Default
`false`
-
+
[[io.storefile.bloom.block.size]]
*`io.storefile.bloom.block.size`*::
+
@@ -1325,7 +1328,7 @@ The size in bytes of a single block ("chunk") of a compound Bloom
.Default
`131072`
-
+
[[hbase.rs.cacheblocksonwrite]]
*`hbase.rs.cacheblocksonwrite`*::
+
@@ -1336,7 +1339,7 @@ Whether an HFile block should be added to the block cache when the
.Default
`false`
-
+
[[hbase.rpc.timeout]]
*`hbase.rpc.timeout`*::
+
@@ -1348,7 +1351,7 @@ This is for the RPC layer to define how long HBase client applications
.Default
`60000`
-
+
[[hbase.rpc.shortoperation.timeout]]
*`hbase.rpc.shortoperation.timeout`*::
+
@@ -1361,7 +1364,7 @@ This is another version of "hbase.rpc.timeout". For those RPC operation
.Default
`10000`
-
+
[[hbase.ipc.client.tcpnodelay]]
*`hbase.ipc.client.tcpnodelay`*::
+
@@ -1372,7 +1375,7 @@ Set no delay on rpc socket connections. See
.Default
`true`
-
+
[[hbase.master.keytab.file]]
*`hbase.master.keytab.file`*::
+
@@ -1383,7 +1386,7 @@ Full path to the kerberos keytab file to use for logging in
.Default
``
-
+
[[hbase.master.kerberos.principal]]
*`hbase.master.kerberos.principal`*::
+
@@ -1397,7 +1400,7 @@ Ex. "hbase/_HOST@EXAMPLE.COM". The kerberos principal name
.Default
``
-
+
[[hbase.regionserver.keytab.file]]
*`hbase.regionserver.keytab.file`*::
+
@@ -1408,7 +1411,7 @@ Full path to the kerberos keytab file to use for logging in
.Default
``
-
+
[[hbase.regionserver.kerberos.principal]]
*`hbase.regionserver.kerberos.principal`*::
+
@@ -1423,7 +1426,7 @@ Ex. "hbase/_HOST@EXAMPLE.COM". The kerberos principal name
.Default
``
-
+
[[hadoop.policy.file]]
*`hadoop.policy.file`*::
+
@@ -1435,7 +1438,7 @@ The policy configuration file used by RPC servers to make
.Default
`hbase-policy.xml`
-
+
[[hbase.superuser]]
*`hbase.superuser`*::
+
@@ -1447,7 +1450,7 @@ List of users or groups (comma-separated), who are allowed
.Default
``
-
+
[[hbase.auth.key.update.interval]]
*`hbase.auth.key.update.interval`*::
+
@@ -1458,7 +1461,7 @@ The update interval for master key for authentication tokens
.Default
`86400000`
-
+
[[hbase.auth.token.max.lifetime]]
*`hbase.auth.token.max.lifetime`*::
+
@@ -1469,7 +1472,7 @@ The maximum lifetime in milliseconds after which an
.Default
`604800000`
-
+
[[hbase.ipc.client.fallback-to-simple-auth-allowed]]
*`hbase.ipc.client.fallback-to-simple-auth-allowed`*::
+
@@ -1484,7 +1487,7 @@ When a client is configured to attempt a secure connection, but attempts to
.Default
`false`
-
+
[[hbase.display.keys]]
*`hbase.display.keys`*::
+
@@ -1496,7 +1499,7 @@ When this is set to true the webUI and such will display all start/end keys
.Default
`true`
-
+
[[hbase.coprocessor.region.classes]]
*`hbase.coprocessor.region.classes`*::
+
@@ -1510,7 +1513,7 @@ A comma-separated list of Coprocessors that are loaded by
.Default
``
-
+
[[hbase.rest.port]]
*`hbase.rest.port`*::
+
@@ -1520,7 +1523,7 @@ The port for the HBase REST server.
.Default
`8080`
-
+
[[hbase.rest.readonly]]
*`hbase.rest.readonly`*::
+
@@ -1532,7 +1535,7 @@ Defines the mode the REST server will be started in. Possible values are:
.Default
`false`
-
+
[[hbase.rest.threads.max]]
*`hbase.rest.threads.max`*::
+
@@ -1547,7 +1550,7 @@ The maximum number of threads of the REST server thread pool.
.Default
`100`
-
+
[[hbase.rest.threads.min]]
*`hbase.rest.threads.min`*::
+
@@ -1559,7 +1562,7 @@ The minimum number of threads of the REST server thread pool.
.Default
`2`
-
+
[[hbase.rest.support.proxyuser]]
*`hbase.rest.support.proxyuser`*::
+
@@ -1569,7 +1572,7 @@ Enables running the REST server to support proxy-user mode.
.Default
`false`
-
+
[[hbase.defaults.for.version.skip]]
*`hbase.defaults.for.version.skip`*::
+
@@ -1578,14 +1581,14 @@ Set to true to skip the 'hbase.defaults.for.version' check.
Setting this to true can be useful in contexts other than
the other side of a maven generation; i.e. running in an
ide. You'll want to set this boolean to true to avoid
- seeing the RuntimException complaint: "hbase-default.xml file
+ seeing the RuntimeException complaint: "hbase-default.xml file
seems to be for and old version of HBase (\${hbase.version}), this
version is X.X.X-SNAPSHOT"
+
.Default
`false`
-
+
[[hbase.coprocessor.master.classes]]
*`hbase.coprocessor.master.classes`*::
+
@@ -1600,7 +1603,7 @@ A comma-separated list of
.Default
``
-
+
[[hbase.coprocessor.abortonerror]]
*`hbase.coprocessor.abortonerror`*::
+
@@ -1615,7 +1618,7 @@ Set to true to cause the hosting server (master or regionserver)
.Default
`true`
-
+
[[hbase.online.schema.update.enable]]
*`hbase.online.schema.update.enable`*::
+
@@ -1625,7 +1628,7 @@ Set true to enable online schema changes.
.Default
`true`
-
+
[[hbase.table.lock.enable]]
*`hbase.table.lock.enable`*::
+
@@ -1637,7 +1640,7 @@ Set to true to enable locking the table in zookeeper for schema change operation
.Default
`true`
-
+
[[hbase.table.max.rowsize]]
*`hbase.table.max.rowsize`*::
+
@@ -1646,12 +1649,12 @@ Set to true to enable locking the table in zookeeper for schema change operation
Maximum size of single row in bytes (default is 1 Gb) for Get'ting
or Scan'ning without in-row scan flag set. If row size exceeds this limit
RowTooBigException is thrown to client.
-
+
+
.Default
`1073741824`
-
+
[[hbase.thrift.minWorkerThreads]]
*`hbase.thrift.minWorkerThreads`*::
+
@@ -1662,7 +1665,7 @@ The "core size" of the thread pool. New threads are created on every
.Default
`16`
-
+
[[hbase.thrift.maxWorkerThreads]]
*`hbase.thrift.maxWorkerThreads`*::
+
@@ -1674,7 +1677,7 @@ The maximum size of the thread pool. When the pending request queue
.Default
`1000`
-
+
[[hbase.thrift.maxQueuedRequests]]
*`hbase.thrift.maxQueuedRequests`*::
+
@@ -1687,7 +1690,7 @@ The maximum number of pending Thrift connections waiting in the queue. If
.Default
`1000`
-
+
[[hbase.thrift.htablepool.size.max]]
*`hbase.thrift.htablepool.size.max`*::
+
@@ -1696,12 +1699,12 @@ The upper bound for the table pool used in the Thrift gateways server.
Since this is per table name, we assume a single table and so with 1000 default
worker threads max this is set to a matching number. For other workloads this number
can be adjusted as needed.
-
+
+
.Default
`1000`
-
+
[[hbase.regionserver.thrift.framed]]
*`hbase.regionserver.thrift.framed`*::
+
@@ -1710,12 +1713,12 @@ Use Thrift TFramedTransport on the server side.
This is the recommended transport for thrift servers and requires a similar setting
on the client side. Changing this to false will select the default transport,
vulnerable to DoS when malformed requests are issued due to THRIFT-601.
-
+
+
.Default
`false`
-
+
[[hbase.regionserver.thrift.framed.max_frame_size_in_mb]]
*`hbase.regionserver.thrift.framed.max_frame_size_in_mb`*::
+
@@ -1725,7 +1728,7 @@ Default frame size when using framed transport
.Default
`2`
-
+
[[hbase.regionserver.thrift.compact]]
*`hbase.regionserver.thrift.compact`*::
+
@@ -1735,7 +1738,7 @@ Use Thrift TCompactProtocol binary serialization protocol.
.Default
`false`
-
+
[[hbase.data.umask.enable]]
*`hbase.data.umask.enable`*::
+
@@ -1746,7 +1749,7 @@ Enable, if true, that file permissions should be assigned
.Default
`false`
-
+
[[hbase.data.umask]]
*`hbase.data.umask`*::
+
@@ -1757,7 +1760,7 @@ File permissions that should be used to write data
.Default
`000`
-
+
[[hbase.metrics.showTableName]]
*`hbase.metrics.showTableName`*::
+
@@ -1770,7 +1773,7 @@ Whether to include the prefix "tbl.tablename" in per-column family metrics.
.Default
`true`
-
+
[[hbase.metrics.exposeOperationTimes]]
*`hbase.metrics.exposeOperationTimes`*::
+
@@ -1782,7 +1785,7 @@ Whether to report metrics about time taken performing an
.Default
`true`
-
+
[[hbase.snapshot.enabled]]
*`hbase.snapshot.enabled`*::
+
@@ -1792,7 +1795,7 @@ Set to true to allow snapshots to be taken / restored / cloned.
.Default
`true`
-
+
[[hbase.snapshot.restore.take.failsafe.snapshot]]
*`hbase.snapshot.restore.take.failsafe.snapshot`*::
+
@@ -1804,7 +1807,7 @@ Set to true to take a snapshot before the restore operation.
.Default
`true`
-
+
[[hbase.snapshot.restore.failsafe.name]]
*`hbase.snapshot.restore.failsafe.name`*::
+
@@ -1816,7 +1819,7 @@ Name of the failsafe snapshot taken by the restore operation.
.Default
`hbase-failsafe-{snapshot.name}-{restore.timestamp}`
-
+
[[hbase.server.compactchecker.interval.multiplier]]
*`hbase.server.compactchecker.interval.multiplier`*::
+
@@ -1831,7 +1834,7 @@ The number that determines how often we scan to see if compaction is necessary.
.Default
`1000`
-
+
[[hbase.lease.recovery.timeout]]
*`hbase.lease.recovery.timeout`*::
+
@@ -1841,7 +1844,7 @@ How long we wait on dfs lease recovery in total before giving up.
.Default
`900000`
-
+
[[hbase.lease.recovery.dfs.timeout]]
*`hbase.lease.recovery.dfs.timeout`*::
+
@@ -1855,7 +1858,7 @@ How long between dfs recover lease invocations. Should be larger than the sum of
.Default
`64000`
-
+
[[hbase.column.max.version]]
*`hbase.column.max.version`*::
+
@@ -1866,7 +1869,7 @@ New column family descriptors will use this value as the default number of versi
.Default
`1`
-
+
[[hbase.dfs.client.read.shortcircuit.buffer.size]]
*`hbase.dfs.client.read.shortcircuit.buffer.size`*::
+
@@ -1880,12 +1883,12 @@ If the DFSClient configuration
direct memory. So, we set it down from the default. Make
it > the default hbase block size set in the HColumnDescriptor
which is usually 64k.
-
+
+
.Default
`131072`
-
+
[[hbase.regionserver.checksum.verify]]
*`hbase.regionserver.checksum.verify`*::
+
@@ -1900,13 +1903,13 @@ If the DFSClient configuration
fails, we will switch back to using HDFS checksums (so do not disable HDFS
checksums! And besides this feature applies to hfiles only, not to WALs).
If this parameter is set to false, then hbase will not verify any checksums,
- instead it will depend on checksum verification being done in the HDFS client.
-
+ instead it will depend on checksum verification being done in the HDFS client.
+
+
.Default
`true`
-
+
[[hbase.hstore.bytes.per.checksum]]
*`hbase.hstore.bytes.per.checksum`*::
+
@@ -1914,12 +1917,12 @@ If the DFSClient configuration
Number of bytes in a newly created checksum chunk for HBase-level
checksums in hfile blocks.
-
+
+
.Default
`16384`
-
+
[[hbase.hstore.checksum.algorithm]]
*`hbase.hstore.checksum.algorithm`*::
+
@@ -1927,12 +1930,12 @@ If the DFSClient configuration
Name of an algorithm that is used to compute checksums. Possible values
are NULL, CRC32, CRC32C.
-
+
+
.Default
`CRC32`
-
+
[[hbase.status.published]]
*`hbase.status.published`*::
+
@@ -1942,60 +1945,60 @@ If the DFSClient configuration
When a region server dies and its recovery starts, the master will push this information
to the client application, to let them cut the connection immediately instead of waiting
for a timeout.
-
+
+
.Default
`false`
-
+
[[hbase.status.publisher.class]]
*`hbase.status.publisher.class`*::
+
.Description
Implementation of the status publication with a multicast message.
-
+
+
.Default
`org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher`
-
+
[[hbase.status.listener.class]]
*`hbase.status.listener.class`*::
+
.Description
Implementation of the status listener with a multicast message.
-
+
+
.Default
`org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener`
-
+
[[hbase.status.multicast.address.ip]]
*`hbase.status.multicast.address.ip`*::
+
.Description
Multicast address to use for the status publication by multicast.
-
+
+
.Default
`226.1.1.3`
-
+
[[hbase.status.multicast.address.port]]
*`hbase.status.multicast.address.port`*::
+
.Description
Multicast port to use for the status publication by multicast.
-
+
+
.Default
`16100`
-
+
[[hbase.dynamic.jars.dir]]
*`hbase.dynamic.jars.dir`*::
+
@@ -2005,12 +2008,12 @@ If the DFSClient configuration
dynamically by the region server without the need to restart. However,
an already loaded filter/co-processor class would not be un-loaded. See
HBASE-1936 for more details.
-
+
+
.Default
`${hbase.rootdir}/lib`
-
+
[[hbase.security.authentication]]
*`hbase.security.authentication`*::
+
@@ -2018,24 +2021,24 @@ If the DFSClient configuration
Controls whether or not secure authentication is enabled for HBase.
Possible values are 'simple' (no authentication), and 'kerberos'.
-
+
+
.Default
`simple`
-
+
[[hbase.rest.filter.classes]]
*`hbase.rest.filter.classes`*::
+
.Description
Servlet filters for REST service.
-
+
+
.Default
`org.apache.hadoop.hbase.rest.filter.GzipFilter`
-
+
[[hbase.master.loadbalancer.class]]
*`hbase.master.loadbalancer.class`*::
+
@@ -2046,12 +2049,12 @@ If the DFSClient configuration
http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.html
It replaces the DefaultLoadBalancer as the default (since renamed
as the SimpleLoadBalancer).
-
+
+
.Default
`org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer`
-
+
[[hbase.security.exec.permission.checks]]
*`hbase.security.exec.permission.checks`*::
+
@@ -2067,28 +2070,28 @@ If the DFSClient configuration
section of the HBase online manual. For more information on granting or
revoking permissions using the AccessController, see the security
section of the HBase online manual.
-
+
+
.Default
`false`
-
+
[[hbase.procedure.regionserver.classes]]
*`hbase.procedure.regionserver.classes`*::
+
.Description
-A comma-separated list of
- org.apache.hadoop.hbase.procedure.RegionServerProcedureManager procedure managers that are
- loaded by default on the active HRegionServer process. The lifecycle methods (init/start/stop)
- will be called by the active HRegionServer process to perform the specific globally barriered
- procedure. After implementing your own RegionServerProcedureManager, just put it in
+A comma-separated list of
+ org.apache.hadoop.hbase.procedure.RegionServerProcedureManager procedure managers that are
+ loaded by default on the active HRegionServer process. The lifecycle methods (init/start/stop)
+ will be called by the active HRegionServer process to perform the specific globally barriered
+ procedure. After implementing your own RegionServerProcedureManager, just put it in
HBase's classpath and add the fully qualified class name here.
-
+
+
.Default
``
-
+
[[hbase.procedure.master.classes]]
*`hbase.procedure.master.classes`*::
+
@@ -2103,7 +2106,7 @@ A comma-separated list of
.Default
``
-
+
[[hbase.coordinated.state.manager.class]]
*`hbase.coordinated.state.manager.class`*::
+
@@ -2113,7 +2116,7 @@ Fully qualified name of class implementing coordinated state manager.
.Default
`org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager`
-
+
[[hbase.regionserver.storefile.refresh.period]]
*`hbase.regionserver.storefile.refresh.period`*::
+
@@ -2126,12 +2129,12 @@ Fully qualified name of class implementing coordinated state manager.
extra Namenode pressure. If the files cannot be refreshed for longer than HFile TTL
(hbase.master.hfilecleaner.ttl) the requests are rejected. Configuring HFile TTL to a larger
value is also recommended with this setting.
-
+
+
.Default
`0`
-
+
[[hbase.region.replica.replication.enabled]]
*`hbase.region.replica.replication.enabled`*::
+
@@ -2139,36 +2142,36 @@ Fully qualified name of class implementing coordinated state manager.
Whether asynchronous WAL replication to the secondary region replicas is enabled or not.
If this is enabled, a replication peer named "region_replica_replication" will be created
- which will tail the logs and replicate the mutatations to region replicas for tables that
+ which will tail the logs and replicate the mutations to region replicas for tables that
have region replication > 1. If this is enabled once, disabling this replication also
requires disabling the replication peer using shell or ReplicationAdmin java class.
- Replication to secondary region replicas works over standard inter-cluster replication.
- So replication, if disabled explicitly, also has to be enabled by setting "hbase.replication"
+ Replication to secondary region replicas works over standard inter-cluster replication.
+ So replication, if disabled explicitly, also has to be enabled by setting "hbase.replication"
to true for this feature to work.
-
+
+
.Default
`false`
-
+
[[hbase.http.filter.initializers]]
*`hbase.http.filter.initializers`*::
+
.Description
- A comma separated list of class names. Each class in the list must extend
- org.apache.hadoop.hbase.http.FilterInitializer. The corresponding Filter will
- be initialized. Then, the Filter will be applied to all user facing jsp
- and servlet web pages.
+ A comma separated list of class names. Each class in the list must extend
+ org.apache.hadoop.hbase.http.FilterInitializer. The corresponding Filter will
+ be initialized. Then, the Filter will be applied to all user facing jsp
+ and servlet web pages.
The ordering of the list defines the ordering of the filters.
- The default StaticUserWebFilter add a user principal as defined by the
+ The default StaticUserWebFilter add a user principal as defined by the
hbase.http.staticuser.user property.
-
+
+
.Default
`org.apache.hadoop.hbase.http.lib.StaticUserWebFilter`
-
+
[[hbase.security.visibility.mutations.checkauths]]
*`hbase.security.visibility.mutations.checkauths`*::
+
@@ -2176,41 +2179,41 @@ Fully qualified name of class implementing coordinated state manager.
This property if enabled, will check whether the labels in the visibility expression are associated
with the user issuing the mutation
-
+
+
.Default
`false`
-
+
[[hbase.http.max.threads]]
*`hbase.http.max.threads`*::
+
.Description
- The maximum number of threads that the HTTP Server will create in its
+ The maximum number of threads that the HTTP Server will create in its
ThreadPool.
-
+
+
.Default
`10`
-
+
[[hbase.replication.rpc.codec]]
*`hbase.replication.rpc.codec`*::
+
.Description
The codec that is to be used when replication is enabled so that
- the tags are also replicated. This is used along with HFileV3 which
+ the tags are also replicated. This is used along with HFileV3 which
supports tags in them. If tags are not used or if the hfile version used
is HFileV2 then KeyValueCodec can be used as the replication codec. Note that
using KeyValueCodecWithTags for replication when there are no tags causes no harm.
-
+
+
.Default
`org.apache.hadoop.hbase.codec.KeyValueCodecWithTags`
-
+
[[hbase.http.staticuser.user]]
*`hbase.http.staticuser.user`*::
+
@@ -2219,12 +2222,12 @@ Fully qualified name of class implementing coordinated state manager.
The user name to filter as, on static web filters
while rendering content. An example use is the HDFS
web UI (user to be used for browsing files).
-
+
+
.Default
`dr.stack`
-
+
[[hbase.regionserver.handler.abort.on.error.percent]]
*`hbase.regionserver.handler.abort.on.error.percent`*::
+
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/hbase_history.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/hbase_history.adoc b/src/main/asciidoc/_chapters/hbase_history.adoc
index de4aff5..7308b90 100644
--- a/src/main/asciidoc/_chapters/hbase_history.adoc
+++ b/src/main/asciidoc/_chapters/hbase_history.adoc
@@ -29,9 +29,9 @@
:icons: font
:experimental:
-* 2006: link:http://research.google.com/archive/bigtable.html[BigTable] paper published by Google.
-* 2006 (end of year): HBase development starts.
-* 2008: HBase becomes Hadoop sub-project.
-* 2010: HBase becomes Apache top-level project.
+* 2006: link:http://research.google.com/archive/bigtable.html[BigTable] paper published by Google.
+* 2006 (end of year): HBase development starts.
+* 2008: HBase becomes Hadoop sub-project.
+* 2010: HBase becomes Apache top-level project.
:numbered:
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/hbck_in_depth.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/hbck_in_depth.adoc b/src/main/asciidoc/_chapters/hbck_in_depth.adoc
index 1b30c59..1e1f9fb 100644
--- a/src/main/asciidoc/_chapters/hbck_in_depth.adoc
+++ b/src/main/asciidoc/_chapters/hbck_in_depth.adoc
@@ -29,7 +29,7 @@
:experimental:
HBaseFsck (hbck) is a tool for checking for region consistency and table integrity problems and repairing a corrupted HBase.
-It works in two basic modes -- a read-only inconsistency identifying mode and a multi-phase read-write repair mode.
+It works in two basic modes -- a read-only inconsistency identifying mode and a multi-phase read-write repair mode.
=== Running hbck to identify inconsistencies
@@ -42,10 +42,10 @@ $ ./bin/hbase hbck
----
At the end of the commands output it prints OK or tells you the number of INCONSISTENCIES present.
-You may also want to run run hbck a few times because some inconsistencies can be transient (e.g.
+You may also want to run hbck a few times because some inconsistencies can be transient (e.g.
cluster is starting up or a region is splitting). Operationally you may want to run hbck regularly and setup alert (e.g.
via nagios) if it repeatedly reports inconsistencies . A run of hbck will report a list of inconsistencies along with a brief description of the regions and tables affected.
-The using the `-details` option will report more details including a representative listing of all the splits present in all the tables.
+The using the `-details` option will report more details including a representative listing of all the splits present in all the tables.
[source,bourne]
----
@@ -66,9 +66,9 @@ $ ./bin/hbase hbck TableFoo TableBar
=== Inconsistencies
If after several runs, inconsistencies continue to be reported, you may have encountered a corruption.
-These should be rare, but in the event they occur newer versions of HBase include the hbck tool enabled with automatic repair options.
+These should be rare, but in the event they occur newer versions of HBase include the hbck tool enabled with automatic repair options.
-There are two invariants that when violated create inconsistencies in HBase:
+There are two invariants that when violated create inconsistencies in HBase:
* HBase's region consistency invariant is satisfied if every region is assigned and deployed on exactly one region server, and all places where this state kept is in accordance.
* HBase's table integrity invariant is satisfied if for each table, every possible row key resolves to exactly one region.
@@ -77,20 +77,20 @@ Repairs generally work in three phases -- a read-only information gathering phas
Starting from version 0.90.0, hbck could detect region consistency problems report on a subset of possible table integrity problems.
It also included the ability to automatically fix the most common inconsistency, region assignment and deployment consistency problems.
This repair could be done by using the `-fix` command line option.
-These problems close regions if they are open on the wrong server or on multiple region servers and also assigns regions to region servers if they are not open.
+These problems close regions if they are open on the wrong server or on multiple region servers and also assigns regions to region servers if they are not open.
Starting from HBase versions 0.90.7, 0.92.2 and 0.94.0, several new command line options are introduced to aid repairing a corrupted HBase.
-This hbck sometimes goes by the nickname ``uberhbck''. Each particular version of uber hbck is compatible with the HBase's of the same major version (0.90.7 uberhbck can repair a 0.90.4). However, versions <=0.90.6 and versions <=0.92.1 may require restarting the master or failing over to a backup master.
+This hbck sometimes goes by the nickname ``uberhbck''. Each particular version of uber hbck is compatible with the HBase's of the same major version (0.90.7 uberhbck can repair a 0.90.4). However, versions <=0.90.6 and versions <=0.92.1 may require restarting the master or failing over to a backup master.
=== Localized repairs
When repairing a corrupted HBase, it is best to repair the lowest risk inconsistencies first.
These are generally region consistency repairs -- localized single region repairs, that only modify in-memory data, ephemeral zookeeper data, or patch holes in the META table.
Region consistency requires that the HBase instance has the state of the region's data in HDFS (.regioninfo files), the region's row in the hbase:meta table., and region's deployment/assignments on region servers and the master in accordance.
-Options for repairing region consistency include:
+Options for repairing region consistency include:
* `-fixAssignments` (equivalent to the 0.90 `-fix` option) repairs unassigned, incorrectly assigned or multiply assigned regions.
-* `-fixMeta` which removes meta rows when corresponding regions are not present in HDFS and adds new meta rows if they regions are present in HDFS while not in META. To fix deployment and assignment problems you can run this command:
+* `-fixMeta` which removes meta rows when corresponding regions are not present in HDFS and adds new meta rows if they regions are present in HDFS while not in META. To fix deployment and assignment problems you can run this command:
[source,bourne]
----
@@ -177,7 +177,7 @@ $ ./bin/hbase hbck -fixMetaOnly -fixAssignments
==== Special cases: HBase version file is missing
HBase's data on the file system requires a version file in order to start.
-If this flie is missing, you can use the `-fixVersionFile` option to fabricating a new HBase version file.
+If this file is missing, you can use the `-fixVersionFile` option to fabricating a new HBase version file.
This assumes that the version of hbck you are running is the appropriate version for the HBase cluster.
==== Special case: Root and META are corrupt.
@@ -205,8 +205,8 @@ However, there could be some lingering offline split parents sometimes.
They are in META, in HDFS, and not deployed.
But HBase can't clean them up.
In this case, you can use the `-fixSplitParents` option to reset them in META to be online and not split.
-Therefore, hbck can merge them with other regions if fixing overlapping regions option is used.
+Therefore, hbck can merge them with other regions if fixing overlapping regions option is used.
-This option should not normally be used, and it is not in `-fixAll`.
+This option should not normally be used, and it is not in `-fixAll`.
:numbered:
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/mapreduce.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/mapreduce.adoc b/src/main/asciidoc/_chapters/mapreduce.adoc
index 1337c79..75718fd 100644
--- a/src/main/asciidoc/_chapters/mapreduce.adoc
+++ b/src/main/asciidoc/_chapters/mapreduce.adoc
@@ -65,7 +65,7 @@ The dependencies only need to be available on the local `CLASSPATH`.
The following example runs the bundled HBase link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/RowCounter.html[RowCounter] MapReduce job against a table named `usertable`.
If you have not set the environment variables expected in the command (the parts prefixed by a `$` sign and surrounded by curly braces), you can use the actual system paths instead.
Be sure to use the correct version of the HBase JAR for your system.
-The backticks (``` symbols) cause ths shell to execute the sub-commands, setting the output of `hbase classpath` (the command to dump HBase CLASSPATH) to `HADOOP_CLASSPATH`.
+The backticks (``` symbols) cause the shell to execute the sub-commands, setting the output of `hbase classpath` (the command to dump HBase CLASSPATH) to `HADOOP_CLASSPATH`.
This example assumes you use a BASH-compatible shell.
[source,bash]
@@ -279,7 +279,7 @@ That is where the logic for map-task assignment resides.
The following is an example of using HBase as a MapReduce source in read-only manner.
Specifically, there is a Mapper instance but no Reducer, and nothing is being emitted from the Mapper.
-There job would be defined as follows...
+The job would be defined as follows...
[source,java]
----
@@ -592,7 +592,7 @@ public class MyMapper extends TableMapper<Text, LongWritable> {
== Speculative Execution
It is generally advisable to turn off speculative execution for MapReduce jobs that use HBase as a source.
-This can either be done on a per-Job basis through properties, on on the entire cluster.
+This can either be done on a per-Job basis through properties, or on the entire cluster.
Especially for longer running jobs, speculative execution will create duplicate map-tasks which will double-write your data to HBase; this is probably not what you want.
See <<spec.ex,spec.ex>> for more information.
@@ -613,7 +613,7 @@ The following example shows a Cascading `Flow` which "sinks" data into an HBase
// emits two fields: "offset" and "line"
Tap source = new Hfs( new TextLine(), inputFileLhs );
-// store data in a HBase cluster
+// store data in an HBase cluster
// accepts fields "num", "lower", and "upper"
// will automatically scope incoming fields to their proper familyname, "left" or "right"
Fields keyFields = new Fields( "num" );
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/ops_mgt.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/ops_mgt.adoc b/src/main/asciidoc/_chapters/ops_mgt.adoc
index c5f52f5..13835c0 100644
--- a/src/main/asciidoc/_chapters/ops_mgt.adoc
+++ b/src/main/asciidoc/_chapters/ops_mgt.adoc
@@ -79,7 +79,7 @@ There is a Canary class can help users to canary-test the HBase cluster status,
To see the usage, use the `--help` parameter.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -help
+$ ${HBASE_HOME}/bin/hbase canary -help
Usage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table1 [table2]...] | [regionserver1 [regionserver2]..]
where [opts] are:
@@ -126,7 +126,7 @@ Following are some examples based on the previous given case.
==== Canary test for every column family (store) of every region of every table
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary
+$ ${HBASE_HOME}/bin/hbase canary
3/12/09 03:26:32 INFO tool.Canary: read from region test-01,,1386230156732.0e3c7d77ffb6361ea1b996ac1042ca9a. column family cf1 in 2ms
13/12/09 03:26:32 INFO tool.Canary: read from region test-01,,1386230156732.0e3c7d77ffb6361ea1b996ac1042ca9a. column family cf2 in 2ms
@@ -147,7 +147,7 @@ This is a default behavior of the this tool does.
You can also test one or more specific tables.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary test-01 test-02
+$ ${HBASE_HOME}/bin/hbase canary test-01 test-02
----
==== Canary test with RegionServer granularity
@@ -155,7 +155,7 @@ $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary test-01 test-02
This will pick one small piece of data from each RegionServer, and can also put your RegionServer name as input options for canary-test specific RegionServer.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -regionserver
+$ ${HBASE_HOME}/bin/hbase canary -regionserver
13/12/09 06:05:17 INFO tool.Canary: Read from table:test-01 on region server:rs2 in 72ms
13/12/09 06:05:17 INFO tool.Canary: Read from table:test-02 on region server:rs3 in 34ms
@@ -167,7 +167,7 @@ $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -regionserver
This will test both table test-01 and test-02.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -e test-0[1-2]
+$ ${HBASE_HOME}/bin/hbase canary -e test-0[1-2]
----
==== Run canary test as daemon mode
@@ -176,13 +176,13 @@ Run repeatedly with interval defined in option `-interval` whose default value i
This daemon will stop itself and return non-zero error code if any error occurs, due to the default value of option -f is true.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -daemon
+$ ${HBASE_HOME}/bin/hbase canary -daemon
----
Run repeatedly with internal 5 seconds and will not stop itself even if errors occur in the test.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -daemon -interval 50000 -f false
+$ ${HBASE_HOME}/bin/hbase canary -daemon -interval 50000 -f false
----
==== Force timeout if canary test stuck
@@ -192,23 +192,23 @@ Because of this we provide a timeout option to kill the canary test and return a
This run sets the timeout value to 60 seconds, the default value is 600 seconds.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -t 600000
+$ ${HBASE_HOME}/bin/hbase canary -t 600000
----
==== Enable write sniffing in canary
By default, the canary tool only check the read operations, it's hard to find the problem in the
write path. To enable the write sniffing, you can run canary with the `-writeSniffing` option.
-When the write sniffing is enabled, the canary tool will create a hbase table and make sure the
+When the write sniffing is enabled, the canary tool will create an hbase table and make sure the
regions of the table distributed on all region servers. In each sniffing period, the canary will
try to put data to these regions to check the write availability of each region server.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -writeSniffing
+$ ${HBASE_HOME}/bin/hbase canary -writeSniffing
----
The default write table is `hbase:canary` and can be specified by the option `-writeTable`.
----
-$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -writeSniffing -writeTable ns:canary
+$ ${HBASE_HOME}/bin/hbase canary -writeSniffing -writeTable ns:canary
----
The default value size of each put is 10 bytes and you can set it by the config key:
@@ -351,7 +351,7 @@ You can invoke it via the HBase cli with the 'wal' command.
[NOTE]
====
Prior to version 2.0, the WAL Pretty Printer was called the `HLogPrettyPrinter`, after an internal name for HBase's write ahead log.
-In those versions, you can pring the contents of a WAL using the same configuration as above, but with the 'hlog' command.
+In those versions, you can print the contents of a WAL using the same configuration as above, but with the 'hlog' command.
----
$ ./bin/hbase hlog hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012
@@ -523,7 +523,7 @@ row9 c1 c2
row10 c1 c2
----
-For ImportTsv to use this imput file, the command line needs to look like this:
+For ImportTsv to use this input file, the command line needs to look like this:
----
@@ -781,7 +781,7 @@ To decommission a loaded RegionServer, run the following: +$
====
The `HOSTNAME` passed to _graceful_stop.sh_ must match the hostname that hbase is using to identify RegionServers.
Check the list of RegionServers in the master UI for how HBase is referring to servers.
-Its usually hostname but can also be FQDN.
+It's usually hostname but can also be FQDN.
Whatever HBase is using, this is what you should pass the _graceful_stop.sh_ decommission script.
If you pass IPs, the script is not yet smart enough to make a hostname (or FQDN) of it and so it will fail when it checks if server is currently running; the graceful unloading of regions will not run.
====
@@ -821,12 +821,12 @@ Hence, it is better to manage the balancer apart from `graceful_stop` reenabling
[[draining.servers]]
==== Decommissioning several Regions Servers concurrently
-If you have a large cluster, you may want to decommission more than one machine at a time by gracefully stopping mutiple RegionServers concurrently.
+If you have a large cluster, you may want to decommission more than one machine at a time by gracefully stopping multiple RegionServers concurrently.
To gracefully drain multiple regionservers at the same time, RegionServers can be put into a "draining" state.
This is done by marking a RegionServer as a draining node by creating an entry in ZooKeeper under the _hbase_root/draining_ znode.
This znode has format `name,port,startcode` just like the regionserver entries under _hbase_root/rs_ znode.
-Without this facility, decommissioning mulitple nodes may be non-optimal because regions that are being drained from one region server may be moved to other regionservers that are also draining.
+Without this facility, decommissioning multiple nodes may be non-optimal because regions that are being drained from one region server may be moved to other regionservers that are also draining.
Marking RegionServers to be in the draining state prevents this from happening.
See this link:http://inchoate-clatter.blogspot.com/2012/03/hbase-ops-automation.html[blog
post] for more details.
@@ -991,7 +991,7 @@ To configure metrics for a given region server, edit the _conf/hadoop-metrics2-h
Restart the region server for the changes to take effect.
To change the sampling rate for the default sink, edit the line beginning with `*.period`.
-To filter which metrics are emitted or to extend the metrics framework, see link:http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html
+To filter which metrics are emitted or to extend the metrics framework, see http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html
.HBase Metrics and Ganglia
[NOTE]
@@ -1014,15 +1014,15 @@ Rather than listing each metric which HBase emits by default, you can browse thr
Different metrics are exposed for the Master process and each region server process.
.Procedure: Access a JSON Output of Available Metrics
-. After starting HBase, access the region server's web UI, at `http://REGIONSERVER_HOSTNAME:60030` by default (or port 16030 in HBase 1.0+).
+. After starting HBase, access the region server's web UI, at pass:[http://REGIONSERVER_HOSTNAME:60030] by default (or port 16030 in HBase 1.0+).
. Click the [label]#Metrics Dump# link near the top.
The metrics for the region server are presented as a dump of the JMX bean in JSON format.
This will dump out all metrics names and their values.
- To include metrics descriptions in the listing -- this can be useful when you are exploring what is available -- add a query string of `?description=true` so your URL becomes `http://REGIONSERVER_HOSTNAME:60030/jmx?description=true`.
+ To include metrics descriptions in the listing -- this can be useful when you are exploring what is available -- add a query string of `?description=true` so your URL becomes pass:[http://REGIONSERVER_HOSTNAME:60030/jmx?description=true].
Not all beans and attributes have descriptions.
-. To view metrics for the Master, connect to the Master's web UI instead (defaults to `http://localhost:60010` or port 16010 in HBase 1.0+) and click its [label]#Metrics
+. To view metrics for the Master, connect to the Master's web UI instead (defaults to pass:[http://localhost:60010] or port 16010 in HBase 1.0+) and click its [label]#Metrics
Dump# link.
- To include metrics descriptions in the listing -- this can be useful when you are exploring what is available -- add a query string of `?description=true` so your URL becomes `http://REGIONSERVER_HOSTNAME:60010/jmx?description=true`.
+ To include metrics descriptions in the listing -- this can be useful when you are exploring what is available -- add a query string of `?description=true` so your URL becomes pass:[http://REGIONSERVER_HOSTNAME:60010/jmx?description=true].
Not all beans and attributes have descriptions.
@@ -1341,9 +1341,9 @@ disable_peer <ID>::
remove_peer <ID>::
Disable and remove a replication relationship. HBase will no longer send edits to that peer cluster or keep track of WALs.
enable_table_replication <TABLE_NAME>::
- Enable the table replication switch for all it's column families. If the table is not found in the destination cluster then it will create one with the same name and column families.
+ Enable the table replication switch for all its column families. If the table is not found in the destination cluster then it will create one with the same name and column families.
disable_table_replication <TABLE_NAME>::
- Disable the table replication switch for all it's column families.
+ Disable the table replication switch for all its column families.
=== Verifying Replicated Data
@@ -1462,7 +1462,7 @@ Speed is also limited by total size of the list of edits to replicate per slave,
With this configuration, a master cluster region server with three slaves would use at most 192 MB to store data to replicate.
This does not account for the data which was filtered but not garbage collected.
-Once the maximum size of edits has been buffered or the reader reaces the end of the WAL, the source thread stops reading and chooses at random a sink to replicate to (from the list that was generated by keeping only a subset of slave region servers). It directly issues a RPC to the chosen region server and waits for the method to return.
+Once the maximum size of edits has been buffered or the reader reaches the end of the WAL, the source thread stops reading and chooses at random a sink to replicate to (from the list that was generated by keeping only a subset of slave region servers). It directly issues a RPC to the chosen region server and waits for the method to return.
If the RPC was successful, the source determines whether the current file has been emptied or it contains more data which needs to be read.
If the file has been emptied, the source deletes the znode in the queue.
Otherwise, it registers the new offset in the log's znode.
@@ -1778,7 +1778,7 @@ but still suboptimal compared to a mechanism which allows large requests to be s
into multiple smaller ones.
HBASE-10993 introduces such a system for deprioritizing long-running scanners. There
-are two types of queues,`fifo` and `deadline`.To configure the type of queue used,
+are two types of queues, `fifo` and `deadline`. To configure the type of queue used,
configure the `hbase.ipc.server.callqueue.type` property in `hbase-site.xml`. There
is no way to estimate how long each request may take, so de-prioritization only affects
scans, and is based on the number of “next” calls a scan request has made. An assumption
@@ -2049,7 +2049,7 @@ Aside from the disk space necessary to store the data, one RS may not be able to
[[ops.capacity.nodes.throughput]]
==== Read/Write throughput
-Number of nodes can also be driven by required thoughput for reads and/or writes.
+Number of nodes can also be driven by required throughput for reads and/or writes.
The throughput one can get per node depends a lot on data (esp.
key/value sizes) and request patterns, as well as node and system configuration.
Planning should be done for peak load if it is likely that the load would be the main driver of the increase of the node count.
@@ -2214,7 +2214,7 @@ or in code it would be as follows:
[source,java]
----
-void rename(Admin admin, String oldTableName, String newTableName) {
+void rename(Admin admin, String oldTableName, TableName newTableName) {
String snapshotName = randomName();
admin.disableTable(oldTableName);
admin.snapshot(snapshotName, oldTableName);
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/other_info.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/other_info.adoc b/src/main/asciidoc/_chapters/other_info.adoc
index 046b747..6143876 100644
--- a/src/main/asciidoc/_chapters/other_info.adoc
+++ b/src/main/asciidoc/_chapters/other_info.adoc
@@ -31,50 +31,50 @@
[[other.info.videos]]
=== HBase Videos
-.Introduction to HBase
-* link:http://www.cloudera.com/content/cloudera/en/resources/library/presentation/chicago_data_summit_apache_hbase_an_introduction_todd_lipcon.html[Introduction to HBase] by Todd Lipcon (Chicago Data Summit 2011).
-* link:http://www.cloudera.com/videos/intorduction-hbase-todd-lipcon[Introduction to HBase] by Todd Lipcon (2010).
-link:http://www.cloudera.com/videos/hadoop-world-2011-presentation-video-building-realtime-big-data-services-at-facebook-with-hadoop-and-hbase[Building Real Time Services at Facebook with HBase] by Jonathan Gray (Hadoop World 2011).
+.Introduction to HBase
+* link:http://www.cloudera.com/content/cloudera/en/resources/library/presentation/chicago_data_summit_apache_hbase_an_introduction_todd_lipcon.html[Introduction to HBase] by Todd Lipcon (Chicago Data Summit 2011).
+* link:http://www.cloudera.com/videos/intorduction-hbase-todd-lipcon[Introduction to HBase] by Todd Lipcon (2010).
+link:http://www.cloudera.com/videos/hadoop-world-2011-presentation-video-building-realtime-big-data-services-at-facebook-with-hadoop-and-hbase[Building Real Time Services at Facebook with HBase] by Jonathan Gray (Hadoop World 2011).
-link:http://www.cloudera.com/videos/hw10_video_how_stumbleupon_built_and_advertising_platform_using_hbase_and_hadoop[HBase and Hadoop, Mixing Real-Time and Batch Processing at StumbleUpon] by JD Cryans (Hadoop World 2010).
+link:http://www.cloudera.com/videos/hw10_video_how_stumbleupon_built_and_advertising_platform_using_hbase_and_hadoop[HBase and Hadoop, Mixing Real-Time and Batch Processing at StumbleUpon] by JD Cryans (Hadoop World 2010).
[[other.info.pres]]
=== HBase Presentations (Slides)
-link:http://www.cloudera.com/content/cloudera/en/resources/library/hadoopworld/hadoop-world-2011-presentation-video-advanced-hbase-schema-design.html[Advanced HBase Schema Design] by Lars George (Hadoop World 2011).
+link:http://www.cloudera.com/content/cloudera/en/resources/library/hadoopworld/hadoop-world-2011-presentation-video-advanced-hbase-schema-design.html[Advanced HBase Schema Design] by Lars George (Hadoop World 2011).
-link:http://www.slideshare.net/cloudera/chicago-data-summit-apache-hbase-an-introduction[Introduction to HBase] by Todd Lipcon (Chicago Data Summit 2011).
+link:http://www.slideshare.net/cloudera/chicago-data-summit-apache-hbase-an-introduction[Introduction to HBase] by Todd Lipcon (Chicago Data Summit 2011).
-link:http://www.slideshare.net/cloudera/hw09-practical-h-base-getting-the-most-from-your-h-base-install[Getting The Most From Your HBase Install] by Ryan Rawson, Jonathan Gray (Hadoop World 2009).
+link:http://www.slideshare.net/cloudera/hw09-practical-h-base-getting-the-most-from-your-h-base-install[Getting The Most From Your HBase Install] by Ryan Rawson, Jonathan Gray (Hadoop World 2009).
[[other.info.papers]]
=== HBase Papers
-link:http://research.google.com/archive/bigtable.html[BigTable] by Google (2006).
+link:http://research.google.com/archive/bigtable.html[BigTable] by Google (2006).
-link:http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html[HBase and HDFS Locality] by Lars George (2010).
+link:http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html[HBase and HDFS Locality] by Lars George (2010).
-link:http://ianvarley.com/UT/MR/Varley_MastersReport_Full_2009-08-07.pdf[No Relation: The Mixed Blessings of Non-Relational Databases] by Ian Varley (2009).
+link:http://ianvarley.com/UT/MR/Varley_MastersReport_Full_2009-08-07.pdf[No Relation: The Mixed Blessings of Non-Relational Databases] by Ian Varley (2009).
[[other.info.sites]]
=== HBase Sites
-link:http://www.cloudera.com/blog/category/hbase/[Cloudera's HBase Blog] has a lot of links to useful HBase information.
+link:http://www.cloudera.com/blog/category/hbase/[Cloudera's HBase Blog] has a lot of links to useful HBase information.
-* link:http://www.cloudera.com/blog/2010/04/cap-confusion-problems-with-partition-tolerance/[CAP Confusion] is a relevant entry for background information on distributed storage systems.
+* link:http://www.cloudera.com/blog/2010/04/cap-confusion-problems-with-partition-tolerance/[CAP Confusion] is a relevant entry for background information on distributed storage systems.
-link:http://wiki.apache.org/hadoop/HBase/HBasePresentations[HBase Wiki] has a page with a number of presentations.
+link:http://wiki.apache.org/hadoop/HBase/HBasePresentations[HBase Wiki] has a page with a number of presentations.
-link:http://refcardz.dzone.com/refcardz/hbase[HBase RefCard] from DZone.
+link:http://refcardz.dzone.com/refcardz/hbase[HBase RefCard] from DZone.
[[other.info.books]]
=== HBase Books
-link:http://shop.oreilly.com/product/0636920014348.do[HBase: The Definitive Guide] by Lars George.
+link:http://shop.oreilly.com/product/0636920014348.do[HBase: The Definitive Guide] by Lars George.
[[other.info.books.hadoop]]
=== Hadoop Books
-link:http://shop.oreilly.com/product/9780596521981.do[Hadoop: The Definitive Guide] by Tom White.
+link:http://shop.oreilly.com/product/9780596521981.do[Hadoop: The Definitive Guide] by Tom White.
:numbered:
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/performance.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/performance.adoc b/src/main/asciidoc/_chapters/performance.adoc
index bf0e790..5155f0a 100644
--- a/src/main/asciidoc/_chapters/performance.adoc
+++ b/src/main/asciidoc/_chapters/performance.adoc
@@ -88,7 +88,7 @@ Multiple rack configurations carry the same potential issues as multiple switche
* Poor switch capacity performance
* Insufficient uplink to another rack
-If the the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing more of your cluster across racks.
+If the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing more of your cluster across racks.
The easiest way to avoid issues when spanning multiple racks is to use port trunking to create a bonded uplink to other racks.
The downside of this method however, is in the overhead of ports that could potentially be used.
An example of this is, creating an 8Gbps port channel from rack A to rack B, using 8 of your 24 ports to communicate between racks gives you a poor ROI, using too few however can mean you're not getting the most out of your cluster.
@@ -102,12 +102,12 @@ Are all the network interfaces functioning correctly? Are you sure? See the Trou
[[perf.network.call_me_maybe]]
=== Network Consistency and Partition Tolerance
-The link:http://en.wikipedia.org/wiki/CAP_theorem[CAP Theorem] states that a distributed system can maintain two out of the following three charateristics:
-- *C*onsistency -- all nodes see the same data.
+The link:http://en.wikipedia.org/wiki/CAP_theorem[CAP Theorem] states that a distributed system can maintain two out of the following three characteristics:
+- *C*onsistency -- all nodes see the same data.
- *A*vailability -- every request receives a response about whether it succeeded or failed.
- *P*artition tolerance -- the system continues to operate even if some of its components become unavailable to the others.
-HBase favors consistency and partition tolerance, where a decision has to be made. Coda Hale explains why partition tolerance is so important, in http://codahale.com/you-cant-sacrifice-partition-tolerance/.
+HBase favors consistency and partition tolerance, where a decision has to be made. Coda Hale explains why partition tolerance is so important, in http://codahale.com/you-cant-sacrifice-partition-tolerance/.
Robert Yokota used an automated testing framework called link:https://aphyr.com/tags/jepsen[Jepson] to test HBase's partition tolerance in the face of network partitions, using techniques modeled after Aphyr's link:https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions[Call Me Maybe] series. The results, available as a link:https://rayokota.wordpress.com/2015/09/30/call-me-maybe-hbase/[blog post] and an link:https://rayokota.wordpress.com/2015/09/30/call-me-maybe-hbase-addendum/[addendum], show that HBase performs correctly.
@@ -556,7 +556,7 @@ When writing a lot of data to an HBase table from a MR job (e.g., with link:http
When a Reducer step is used, all of the output (Puts) from the Mapper will get spooled to disk, then sorted/shuffled to other Reducers that will most likely be off-node.
It's far more efficient to just write directly to HBase.
-For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step (e.g., summarize values then write out result). This is a different processing problem than from the the above case.
+For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step (e.g., summarize values then write out result). This is a different processing problem than from the above case.
[[perf.one.region]]
=== Anti-Pattern: One Hot Region
@@ -565,7 +565,7 @@ If all your data is being written to one region at a time, then re-read the sect
Also, if you are pre-splitting regions and all your data is _still_ winding up in a single region even though your keys aren't monotonically increasing, confirm that your keyspace actually works with the split strategy.
There are a variety of reasons that regions may appear "well split" but won't work with your data.
-As the HBase client communicates directly with the RegionServers, this can be obtained via link:hhttp://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#getRegionLocation(byte[])[Table.getRegionLocation].
+As the HBase client communicates directly with the RegionServers, this can be obtained via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#getRegionLocation(byte%5B%5D)[Table.getRegionLocation].
See <<precreate.regions>>, as well as <<perf.configurations>>
@@ -607,7 +607,7 @@ When columns are selected explicitly with `scan.addColumn`, HBase will schedule
When rows have few columns and each column has only a few versions this can be inefficient.
A seek operation is generally slower if does not seek at least past 5-10 columns/versions or 512-1024 bytes.
-In order to opportunistically look ahead a few columns/versions to see if the next column/version can be found that way before a seek operation is scheduled, a new attribute `Scan.HINT_LOOKAHEAD` can be set the on Scan object.
+In order to opportunistically look ahead a few columns/versions to see if the next column/version can be found that way before a seek operation is scheduled, a new attribute `Scan.HINT_LOOKAHEAD` can be set on the Scan object.
The following code instructs the RegionServer to attempt two iterations of next before a seek is scheduled:
[source,java]
@@ -731,7 +731,7 @@ However, if hedged reads are enabled, the client waits some configurable amount
Whichever read returns first is used, and the other read request is discarded.
Hedged reads can be helpful for times where a rare slow read is caused by a transient error such as a failing disk or flaky network connection.
-Because a HBase RegionServer is a HDFS client, you can enable hedged reads in HBase, by adding the following properties to the RegionServer's hbase-site.xml and tuning the values to suit your environment.
+Because an HBase RegionServer is a HDFS client, you can enable hedged reads in HBase, by adding the following properties to the RegionServer's hbase-site.xml and tuning the values to suit your environment.
.Configuration for Hedged Reads
* `dfs.client.hedged.read.threadpool.size` - the number of threads dedicated to servicing hedged reads.
@@ -782,7 +782,8 @@ Be aware that `Table.delete(Delete)` doesn't use the writeBuffer.
It will execute an RegionServer RPC with each invocation.
For a large number of deletes, consider `Table.delete(List)`.
-See http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#delete%28org.apache.hadoop.hbase.client.Delete%29
+See
++++<a href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#delete%28org.apache.hadoop.hbase.client.Delete%29">hbase.client.Delete</a>+++.
[[perf.hdfs]]
== HDFS
@@ -869,7 +870,7 @@ If you are running on EC2 and post performance questions on the dist-list, pleas
== Collocating HBase and MapReduce
It is often recommended to have different clusters for HBase and MapReduce.
-A better qualification of this is: don't collocate a HBase that serves live requests with a heavy MR workload.
+A better qualification of this is: don't collocate an HBase that serves live requests with a heavy MR workload.
OLTP and OLAP-optimized systems have conflicting requirements and one will lose to the other, usually the former.
For example, short latency-sensitive disk reads will have to wait in line behind longer reads that are trying to squeeze out as much throughput as possible.
MR jobs that write to HBase will also generate flushes and compactions, which will in turn invalidate blocks in the <<block.cache>>.
[3/4] hbase git commit: updating docs from master
Posted by nd...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/cp.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/cp.adoc b/src/main/asciidoc/_chapters/cp.adoc
index 45944b4..5f50b68 100644
--- a/src/main/asciidoc/_chapters/cp.adoc
+++ b/src/main/asciidoc/_chapters/cp.adoc
@@ -27,254 +27,212 @@
:icons: font
:experimental:
-HBase Coprocessors are modeled after the Coprocessors which are part of Google's BigTable
-(http://research.google.com/people/jeff/SOCC2010-keynote-slides.pdf pages 41-42.). +
-Coprocessor is a framework that provides an easy way to run your custom code directly on
-Region Server.
-The information in this chapter is primarily sourced and heavily reused from:
+HBase Coprocessors are modeled after Google BigTable's coprocessor implementation
+(http://research.google.com/people/jeff/SOCC2010-keynote-slides.pdf pages 41-42.).
+
+The coprocessor framework provides mechanisms for running your custom code directly on
+the RegionServers managing your data. Efforts are ongoing to bridge gaps between HBase's
+implementation and BigTable's architecture. For more information see
+link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047].
+
+The information in this chapter is primarily sourced and heavily reused from the following
+resources:
. Mingjie Lai's blog post
link:https://blogs.apache.org/hbase/entry/coprocessor_introduction[Coprocessor Introduction].
. Gaurav Bhardwaj's blog post
link:http://www.3pillarglobal.com/insights/hbase-coprocessors[The How To Of HBase Coprocessors].
+[WARNING]
+.Use Coprocessors At Your Own Risk
+====
+Coprocessors are an advanced feature of HBase and are intended to be used by system
+developers only. Because coprocessor code runs directly on the RegionServer and has
+direct access to your data, they introduce the risk of data corruption, man-in-the-middle
+attacks, or other malicious data access. Currently, there is no mechanism to prevent
+data corruption by coprocessors, though work is underway on
+link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047].
++
+In addition, there is no resource isolation, so a well-intentioned but misbehaving
+coprocessor can severely degrade cluster performance and stability.
+====
+== Coprocessor Overview
-== Coprocessor Framework
-
-When working with any data store (like RDBMS or HBase) you fetch the data (in case of RDBMS you
-might use SQL query and in case of HBase you use either Get or Scan). To fetch only relevant data
-you filter it (for RDBMS you put conditions in 'WHERE' predicate and in HBase you use
-link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/Filter.html[Filter]).
-After fetching the desired data, you perform your business computation on the data.
-This scenario is close to ideal for "small data", where few thousand rows and a bunch of columns
-are returned from the data store. Now imagine a scenario where there are billions of rows and
-millions of columns and you want to perform some computation which requires all the data, like
-calculating average or sum. Even if you are interested in just few columns, you still have to
-fetch all the rows. There are a few drawbacks in this approach as described below:
-
-. In this approach the data transfer (from data store to client side) will become the bottleneck,
-and the time required to complete the operation is limited by the rate at which data transfer
-takes place.
-. It's not always possible to hold so much data in memory and perform computation.
-. Bandwidth is one of the most precious resources in any data center. Operations like this may
-saturate your data center’s bandwidth and will severely impact the performance of your cluster.
-. Your client code is becoming thick as you are maintaining the code for calculating average or
-summation on client side. Not a major drawback when talking of severe issues like
-performance/bandwidth but still worth giving consideration.
-
-In a scenario like this it's better to move the computation (i.e. user's custom code) to the data
-itself (Region Server). Coprocessor helps you achieve this but you can do more than that.
-There is another advantage that your code runs in parallel (i.e. on all Regions).
-To give an idea of Coprocessor's capabilities, different people give different analogies.
-The three most famous analogies for Coprocessor are:
-[[cp_analogies]]
-Triggers and Stored Procedure:: This is the most common analogy for Coprocessor. Observer
-Coprocessor is compared to triggers because like triggers they execute your custom code when
-certain event occurs (like Get or Put etc.). Similarly Endpoints Coprocessor is compared to the
-stored procedures and you can perform custom computation on data directly inside the region server.
+In HBase, you fetch data using a `Get` or `Scan`, whereas in an RDBMS you use a SQL
+query. In order to fetch only the relevant data, you filter it using a HBase
+link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/Filter.html[Filter]
+, whereas in an RDBMS you use a `WHERE` predicate.
-MapReduce:: As in MapReduce you move the computation to the data in the same way. Coprocessor
-executes your custom computation directly on Region Servers, i.e. where data resides. That's why
-some people compare Coprocessor to a small MapReduce jobs.
+After fetching the data, you perform computations on it. This paradigm works well
+for "small data" with a few thousand rows and several columns. However, when you scale
+to billions of rows and millions of columns, moving large amounts of data across your
+network will create bottlenecks at the network layer, and the client needs to be powerful
+enough and have enough memory to handle the large amounts of data and the computations.
+In addition, the client code can grow large and complex.
-AOP:: Some people compare it to _Aspect Oriented Programming_ (AOP). As in AOP, you apply advice
-(on occurrence of specific event) by intercepting the request and then running some custom code
-(probably cross-cutting concerns) and then forwarding the request on its path as if nothing
-happened (or even return it back). Similarly in Coprocessor you have this facility of intercepting
-the request and running custom code and then forwarding it on its path (or returning it).
+In this scenario, coprocessors might make sense. You can put the business computation
+code into a coprocessor which runs on the RegionServer, in the same location as the
+data, and returns the result to the client.
+This is only one scenario where using coprocessors can provide benefit. Following
+are some analogies which may help to explain some of the benefits of coprocessors.
-Although Coprocessor derives its roots from Google's Bigtable but it deviates from it largely in
-its design. Currently there are efforts going on to bridge this gap. For more information see
-link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047].
+[[cp_analogies]]
+=== Coprocessor Analogies
-In HBase, to implement a Coprocessor certain steps must be followed as described below:
+Triggers and Stored Procedure::
+ An Observer coprocessor is similar to a trigger in a RDBMS in that it executes
+ your code either before or after a specific event (such as a `Get` or `Put`)
+ occurs. An endpoint coprocessor is similar to a stored procedure in a RDBMS
+ because it allows you to perform custom computations on the data on the
+ RegionServer itself, rather than on the client.
-. Either your class should extend one of the Coprocessor classes (like
-// Below URL is more than 100 characters long.
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver]
-) or it should implement Coprocessor interfaces (like
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/Coprocessor.html[Coprocessor],
-// Below URL is more than 100 characters long.
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/CoprocessorService.html[CoprocessorService]).
+MapReduce::
+ MapReduce operates on the principle of moving the computation to the location of
+ the data. Coprocessors operate on the same principal.
-. Load the Coprocessor: Currently there are two ways to load the Coprocessor. +
-Static:: Loading from configuration
-Dynammic:: Loading via 'hbase shell' or via Java code using HTableDescriptor class). +
-For more details see <<cp_loading,Loading Coprocessors>>.
+AOP::
+ If you are familiar with Aspect Oriented Programming (AOP), you can think of a coprocessor
+ as applying advice by intercepting a request and then running some custom code,
+ before passing the request on to its final destination (or even changing the destination).
-. Finally your client-side code to call the Coprocessor. This is the easiest step, as HBase
-handles the Coprocessor transparently and you don't have to do much to call the Coprocessor.
+=== Coprocessor Implementation Overview
-The framework API is provided in the
-// Below URL is more than 100 characters long.
-link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/package-summary.html[coprocessor]
-package. +
-Coprocessors are not designed to be used by the end users but by developers. Coprocessors are
-executed directly on region server; therefore a faulty/malicious code can bring your region server
-down. Currently there is no mechanism to prevent this, but there are efforts going on for this.
-For more, see link:https://issues.apache.org/jira/browse/HBASE-4047[HBASE-4047]. +
-Two different types of Coprocessors are provided by the framework, based on their functionality.
+. Either your class should extend one of the Coprocessor classes, such as
+link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver],
+or it should implement the link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/Coprocessor.html[Coprocessor]
+or
+link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/CoprocessorService.html[CoprocessorService]
+interface.
+. Load the coprocessor, either statically (from the configuration) or dynamically,
+using HBase Shell. For more details see <<cp_loading,Loading Coprocessors>>.
+. Call the coprocessor from your client-side code. HBase handles the coprocessor
+trapsparently.
+
+The framework API is provided in the
+link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/package-summary.html[coprocessor]
+package.
== Types of Coprocessors
-Coprocessor can be broadly divided into two categories: Observer and Endpoint.
-
-=== Observer
-Observer Coprocessor are easy to understand. People coming from RDBMS background can compare them
-to the triggers available in relational databases. Folks coming from programming background can
-visualize it like advice (before and after only) available in AOP (Aspect Oriented Programming).
-See <<cp_analogies, Coprocessor Analogy>> +
-Coprocessors allows you to hook your custom code in two places during the life cycle of an event. +
-First is just _before_ the occurrence of the event (just like 'before' advice in AOP or triggers
-like 'before update'). All methods providing this kind feature will start with the prefix `pre`. +
-For example if you want your custom code to get executed just before the `Put` operation, you can
-use the override the
-// Below URL is more than 100 characters long.
-link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#prePut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`prePut`]
-method of
-// Below URL is more than 100 characters long.
-link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionCoprocessor].
-This method has following signature:
-[source,java]
-----
-public void prePut (final ObserverContext e, final Put put, final WALEdit edit,final Durability
-durability) throws IOException;
-----
+=== Observer Coprocessors
-Secondly, the Observer Coprocessor also provides hooks for your code to get executed just _after_
-the occurrence of the event (similar to after advice in AOP terminology or 'after update' triggers
-). The methods giving this functionality will start with the prefix `post`. For example, if you
-want your code to be executed after the 'Put' operation, you should consider overriding
-// Below URL is more than 100 characters long.
-link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#postPut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`postPut`]
-method of
-// Below URL is more than 100 characters long.
-link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionCoprocessor]:
-[source,java]
-----
-public void postPut(final ObserverContext e, final Put put, final WALEdit edit, final Durability
-durability) throws IOException;
-----
-
-In short, the following conventions are generally followed: +
-Override _preXXX()_ method if you want your code to be executed just before the occurrence of the
-event. +
-Override _postXXX()_ method if you want your code to be executed just after the occurrence of the
-event. +
+Observer coprocessors are triggered either before or after a specific event occurs.
+Observers that happen before an event use methods that start with a `pre` prefix,
+such as link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#prePut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`prePut`]. Observers that happen just after an event override methods that start
+with a `post` prefix, such as link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html#postPut%28org.apache.hadoop.hbase.coprocessor.ObserverContext,%20org.apache.hadoop.hbase.client.Put,%20org.apache.hadoop.hbase.regionserver.wal.WALEdit,%20org.apache.hadoop.hbase.client.Durability%29[`postPut`].
-.Use Cases for Observer Coprocessors:
-Few use cases of the Observer Coprocessor are:
-. *Security*: Before performing any operation (like 'Get', 'Put') you can check for permission in
-the 'preXXX' methods.
+==== Use Cases for Observer Coprocessors
+Security::
+ Before performing a `Get` or `Put` operation, you can check for permission using
+ `preGet` or `prePut` methods.
-. *Referential Integrity*: Unlike traditional RDBMS, HBase doesn't have the concept of referential
-integrity (foreign key). Suppose for example you have a requirement that whenever you insert a
-record in 'users' table, a corresponding entry should also be created in 'user_daily_attendance'
-table. One way you could solve this is by using two 'Put' one for each table, this way you are
-throwing the responsibility (of the referential integrity) to the user. A better way is to use
-Coprocessor and overriding 'postPut' method in which you write the code to insert the record in
-'user_daily_attendance' table. This way client code is more lean and clean.
+Referential Integrity::
+ HBase does not directly support the RDBMS concept of refential integrity, also known
+ as foreign keys. You can use a coprocessor to enforce such integrity. For instance,
+ if you have a business rule that every insert to the `users` table must be followed
+ by a corresponding entry in the `user_daily_attendance` table, you could implement
+ a coprocessor to use the `prePut` method on `user` to insert a record into `user_daily_attendance`.
-. *Secondary Index*: Coprocessor can be used to maintain secondary indexes. For more information
-see link:http://wiki.apache.org/hadoop/Hbase/SecondaryIndexing[SecondaryIndexing].
+Secondary Indexes::
+ You can use a coprocessor to maintain secondary indexes. For more information, see
+ link:http://wiki.apache.org/hadoop/Hbase/SecondaryIndexing[SecondaryIndexing].
==== Types of Observer Coprocessor
-Observer Coprocessor comes in following flavors:
-
-. *RegionObserver*: This Coprocessor provides the facility to hook your code when the events on
-region are triggered. Most common example include 'preGet' and 'postGet' for 'Get' operation and
-'prePut' and 'postPut' for 'Put' operation. For exhaustive list of supported methods (events) see
-// Below URL is more than 100 characters long.
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionObserver].
-
-. *Region Server Observer*: Provides hook for the events related to the RegionServer, such as
-stopping the RegionServer and performing operations before or after merges, commits, or rollbacks.
-For more details please refer
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionServerObserver.html[RegionServerObserver].
-
-. *Master Observer*: This observer provides hooks for DDL like operation, such as create, delete,
-modify table. For entire list of available methods see
-// Below URL is more than 100 characters long.
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/MasterObserver.html[MasterObserver].
-
-. *WAL Observer*: Provides hooks for WAL (Write-Ahead-Log) related operation. It has only two
-method 'preWALWrite()' and 'postWALWrite()'. For more details see
-// Below URL is more than 100 characters long.
-link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/WALObserver.html[WALObserver].
-
-For example see <<cp_example,Examples>>
+RegionObserver::
+ A RegionObserver coprocessor allows you to observe events on a region, such as `Get`
+ and `Put` operations. See
+ link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html[RegionObserver].
+ Consider overriding the convenience class
+ link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver],
+ which implements the `RegionObserver` interface and will not break if new methods are added.
+
+RegionServerObserver::
+ A RegionServerObserver allows you to observe events related to the RegionServer's
+ operation, such as starting, stopping, or performing merges, commits, or rollbacks.
+ See
+ link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionServerObserver.html[RegionServerObserver].
+ Consider overriding the convenience class
+ link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseMasterRegionServerObserver.html[BaseMasterRegionServerObserver]
+ which implements both `MasterObserver` and `RegionServerObserver` interfaces and
+ will not break if new methods are added.
+
+MasterOvserver::
+ A MasterObserver allows you to observe events related to the HBase Master, such
+ as table creation, deletion, or schema modification. See
+ link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/MasterObserver.html[MasterObserver].
+ Consider overriding the convenience class
+ link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseMasterRegionServerObserver.html[BaseMasterRegionServerObserver],
+ which implements both `MasterObserver` and `RegionServerObserver` interfaces and
+ will not break if new methods are added.
+
+WalObserver::
+ A WalObserver allows you to observe events related to writes to the Write-Ahead
+ Log (WAL). See
+ link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/WALObserver.html[WALObserver].
+ Consider overriding the convenience class
+ link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseWALObserver.html[BaseWALObserver],
+ which implements the `WalObserver` interface and will not break if new methods are added.
+
+<<cp_example,Examples>> provides working examples of observer coprocessors.
=== Endpoint Coprocessor
-Endpoint Coprocessor can be compared to stored procedure found in RDBMS.
-See <<cp_analogies, Coprocessor Analogy>>. They help in performing computation which is not
-possible either through Observer Coprocessor or otherwise. For example, calculating average or
-summation over the entire table that spans across multiple regions. They do so by providing a hook
-for your custom code and then running it across all regions. +
-With Endpoints Coprocessor you can create your own dynamic RPC protocol and thus can provide
-communication between client and region server, hence enabling you to run your custom code on
-region server (on each region of a table). +
-Unlike observer Coprocessor (where your custom code is
-executed transparently when events like 'Get' operation occurs), in Endpoint Coprocessor you have
-to explicitly invoke the Coprocessor by using the
-// Below URL is more than 100 characters long.
+Endpoint processors allow you to perform computation at the location of the data.
+See <<cp_analogies, Coprocessor Analogy>>. An example is the need to calculate a running
+average or summation for an entire table which spans hundreds of regions.
+
+In contract to observer coprocessors, where your code is run transparently, endpoint
+coprocessors must be explicitly invoked using the
link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/Table.html#coprocessorService%28java.lang.Class,%20byte%5B%5D,%20byte%5B%5D,%20org.apache.hadoop.hbase.client.coprocessor.Batch.Call%29[CoprocessorService()]
method available in
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/Table.html[Table]
-(or
-// Below URL is more than 100 characters long.
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTableInterface.html[HTableInterface]
+link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/Table.html[Table],
+link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTableInterface.html[HTableInterface],
or
-link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTable.html[HTable]).
+link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTable.html[HTable].
-From version 0.96, implementing Endpoint Coprocessor is not straight forward. Now it is done with
-the help of Google's Protocol Buffer. For more details on Protocol Buffer, please see
+Starting with HBase 0.96, endpoint coprocessors are implemented using Google Protocol
+Buffers (protobuf). For more details on protobuf, see Google's
link:https://developers.google.com/protocol-buffers/docs/proto[Protocol Buffer Guide].
-Endpoints Coprocessor written in version 0.94 are not compatible with with version 0.96 or later
-(for more details, see
-link:https://issues.apache.org/jira/browse/HBASE-5448[HBASE-5448]),
-so if your are upgrading your HBase cluster from version 0.94 (or before) to 0.96 (or later) you
-have to rewrite your Endpoint coprocessor.
-
-For example see <<cp_example,Examples>>
+Endpoints Coprocessor written in version 0.94 are not compatible with version 0.96 or later.
+See
+link:https://issues.apache.org/jira/browse/HBASE-5448[HBASE-5448]). To upgrade your
+HBase cluster from 0.94 or earlier to 0.96 or later, you need to reimplement your
+coprocessor.
+<<cp_example,Examples>> provides working examples of endpoint coprocessors.
[[cp_loading]]
== Loading Coprocessors
-_Loading of Coprocessor refers to the process of making your custom Coprocessor implementation
-available to the the HBase, so that when a requests comes in or an event takes place the desired
-functionality implemented in your custom code gets executed. +
-Coprocessor can be loaded broadly in two ways. One is static (loading through configuration files)
-and the other one is dynamic loading (using hbase shell or java code).
+To make your coprocessor available to HBase, it must be _loaded_, either statically
+(through the HBase configuration) or dynamically (using HBase Shell or the Java API).
=== Static Loading
-Static loading means that your Coprocessor will take effect only when you restart your HBase and
-there is a reason for it. In this you make changes 'hbase-site.xml' and therefore have to restart
-HBase for your changes to take place. +
-Following are the steps for loading Coprocessor statically.
-. Define the Coprocessor in hbase-site.xml: Define a <property> element which consist of two
-sub elements <name> and <value> respectively.
+Follow these steps to statically load your coprocessor. Keep in mind that you must
+restart HBase to unload a coprocessor that has been loaded statically.
+
+. Define the Coprocessor in _hbase-site.xml_, with a <property> element with a <name>
+and a <value> sub-element. The <name> should be one of the following:
+
-.. <name> can have one of the following values:
+- `hbase.coprocessor.region.classes` for RegionObservers and Endpoints.
+- `hbase.coprocessor.wal.classes` for WALObservers.
+- `hbase.coprocessor.master.classes` for MasterObservers.
+
-... 'hbase.coprocessor.region.classes' for RegionObservers and Endpoints.
-... 'hbase.coprocessor.wal.classes' for WALObservers.
-... 'hbase.coprocessor.master.classes' for MasterObservers.
-.. <value> must contain the fully qualified class name of your class implmenting the Coprocessor.
+<value> must contain the fully-qualified class name of your coprocessor's implementation
+class.
+
For example to load a Coprocessor (implemented in class SumEndPoint.java) you have to create
-following entry in RegionServer's 'hbase-site.xml' file (generally located under 'conf' directiory):
+following entry in RegionServer's 'hbase-site.xml' file (generally located under 'conf' directory):
+
[source,xml]
----
@@ -283,6 +241,7 @@ following entry in RegionServer's 'hbase-site.xml' file (generally located under
<value>org.myname.hbase.coprocessor.endpoint.SumEndPoint</value>
</property>
----
++
If multiple classes are specified for loading, the class names must be comma-separated.
The framework attempts to load all the configured classes using the default class loader.
Therefore, the jar file must reside on the server-side HBase classpath.
@@ -297,34 +256,32 @@ When calling out to registered observers, the framework executes their callbacks
sorted order of their priority. +
Ties are broken arbitrarily.
-. Put your code on classpth of HBase: There are various ways to do so, like adding jars on
-classpath etc. One easy way to do this is to drop the jar (containing you code and all the
-dependencies) in 'lib' folder of the HBase installation.
-
-. Restart the HBase.
+. Put your code HBase's classpath. One easy way to do this is to drop the jar
+ (containing you code and all the dependencies) into the `lib/` directory in the
+ HBase installation.
+. Restart HBase.
-==== Unloading Static Coprocessor
-Unloading static Coprocessor is easy. Following are the steps:
-. Delete the Coprocessor's entry from the 'hbase-site.xml' i.e. remove the <property> tag.
+=== Static Unloading
-. Restart the Hbase.
+. Delete the coprocessor's <property> element, including sub-elements, from `hbase-site.xml`.
+. Restart HBase.
+. Optionally, remove the coprocessor's JAR file from the classpath or HBase's `lib/`
+ directory.
-. Optionally remove the Coprocessor jar file from the classpath (or from the lib directory if you
-copied it over there). Removing the coprocessor JARs from HBase’s classpath is a good practice.
=== Dynamic Loading
-Dynamic loading refers to the process of loading Coprocessor without restarting HBase. This may
-sound better than the static loading (and in some scenarios it may) but there is a caveat, dynamic
-loaded Coprocessor applies to the table only for which it was loaded while same is not true for
-static loading as it applies to all the tables. Due to this difference sometimes dynamically
-loaded Coprocessor are also called *Table Coprocessor* (as they applies only to a single table)
-while statically loaded Coprocessor are called *System Coprocessor* (as they applies to all the
-tables). +
-To dynamically load the Coprocessor you have to take the table offline hence during this time you
-won't be able to process any request involving this table. +
-There are three ways to dynamically load Coprocessor as shown below:
+
+You can also load a coprocessor dynamically, without restarting HBase. This may seem
+preferable to static loading, but dynamically loaded coprocessors are loaded on a
+per-table basis, and are only available to the table for which they were loaded. For
+this reason, dynamically loaded tables are sometimes called *Table Coprocessor*.
+
+In addition, dynamically loading a coprocessor acts as a schema change on the table,
+and the table must be taken offline to load the coprocessor.
+
+There are three ways to dynamically load Coprocessor.
[NOTE]
.Assumptions
@@ -332,26 +289,25 @@ There are three ways to dynamically load Coprocessor as shown below:
The below mentioned instructions makes the following assumptions:
* A JAR called `coprocessor.jar` contains the Coprocessor implementation along with all of its
-dependencies if any.
+dependencies.
* The JAR is available in HDFS in some location like
`hdfs://<namenode>:<port>/user/<hadoop-user>/coprocessor.jar`.
====
-. *Using Shell*: You can load the Coprocessor using the HBase shell as follows:
-.. Disable Table: Take table offline by disabling it. Suppose if the table name is 'users', then
-to disable it enter following command:
+==== Using HBase Shell
+
+. Disable the table using HBase Shell:
+
[source]
----
-hbase(main):001:0> disable 'users'
+hbase> disable 'users'
----
-.. Load the Coprocessor: The Coprocessor jar should be on HDFS and should be accessible to HBase,
-to load the Coprocessor use following command:
+. Load the Coprocessor, using a command like the following:
+
[source]
----
-hbase(main):002:0> alter 'users', METHOD => 'table_att', 'Coprocessor'=>'hdfs://<namenode>:<port>/
+hbase alter 'users', METHOD => 'table_att', 'Coprocessor'=>'hdfs://<namenode>:<port>/
user/<hadoop-user>/coprocessor.jar| org.myname.hbase.Coprocessor.RegionObserverExample|1073741823|
arg1=1,arg2=2'
----
@@ -370,30 +326,25 @@ observers registered at the same hook using priorities. This field can be left b
case the framework will assign a default priority value.
* Arguments (Optional): This field is passed to the Coprocessor implementation. This is optional.
-.. Enable the table: To enable table type following command:
+. Enable the table.
+
----
hbase(main):003:0> enable 'users'
----
-.. Verification: This is optional but generally good practice to see if your Coprocessor is
-loaded successfully. Enter following command:
+
+. Verify that the coprocessor loaded:
+
----
hbase(main):04:0> describe 'users'
----
+
-You must see some output like this:
-+
-----
-DESCRIPTION ENABLED
-'users', {TABLE_ATTRIBUTES => {coprocessor$1 => true 'hdfs://<namenode>:<port>/user/<hadoop-user>/
-coprocessor.jar| org.myname.hbase.Coprocessor.RegionObserverExample|1073741823|'}, {NAME =>
-'personalDet'.....
-----
+The coprocessor should be listed in the `TABLE_ATTRIBUTES`.
+==== Using the Java API (all HBase versions)
+
+The following Java code shows how to use the `setValue()` method of `HTableDescriptor`
+to load a coprocessor on the `users` table.
-. *Using setValue()* method of HTableDescriptor: This is done entirely in Java as follows:
-+
[source,java]
----
TableName tableName = TableName.valueOf("users");
@@ -416,12 +367,14 @@ admin.modifyTable(tableName, hTableDescriptor);
admin.enableTable(tableName);
----
-. *Using addCoprocessor()* method of HTableDescriptor: This method is available from 0.96 version
-onwards.
-+
+==== Using the Java API (HBase 0.96+ only)
+
+In HBase 0.96 and newer, the `addCoprocessor()` method of `HTableDescriptor` provides
+an easier way to load a coprocessor dynamically.
+
[source,java]
----
-String tableName = "users";
+TableName tableName = TableName.valueOf("users");
String path = "hdfs://<namenode>:<port>/user/<hadoop-user>/coprocessor.jar";
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
@@ -439,26 +392,42 @@ admin.modifyTable(tableName, hTableDescriptor);
admin.enableTable(tableName);
----
-====
WARNING: There is no guarantee that the framework will load a given Coprocessor successfully.
For example, the shell command neither guarantees a jar file exists at a particular location nor
verifies whether the given class is actually contained in the jar file.
-====
-==== Unloading Dynamic Coprocessor
-. Using shell: Run following command from HBase shell to remove Coprocessor from a table.
+=== Dynamic Unloading
+
+==== Using HBase Shell
+
+. Disable the table.
++
+[source]
+----
+hbase> disable 'users'
+----
+
+. Alter the table to remove the coprocessor.
+
[source]
----
-hbase(main):003:0> alter 'users', METHOD => 'table_att_unset',
-hbase(main):004:0* NAME => 'coprocessor$1'
+hbase> alter 'users', METHOD => 'table_att_unset', NAME => 'coprocessor$1'
----
-. Using HtableDescriptor: Simply reload the table definition _without_ setting the value of
-Coprocessor either in setValue() or addCoprocessor() methods. This will remove the Coprocessor
-attached to this table, if any. For example:
+. Enable the table.
+
+[source]
+----
+hbase> enable 'users'
+----
+
+==== Using the Java API
+
+Reload the table definition without setting the value of the coprocessor either by
+using `setValue()` or `addCoprocessor()` methods. This will remove any coprocessor
+attached to the table.
+
[source,java]
----
TableName tableName = TableName.valueOf("users");
@@ -477,26 +446,23 @@ hTableDescriptor.addFamily(columnFamily2);
admin.modifyTable(tableName, hTableDescriptor);
admin.enableTable(tableName);
----
-+
-Optionally you can also use removeCoprocessor() method of HTableDescriptor class.
+In HBase 0.96 and newer, you can instead use the `removeCoprocessor()` method of the
+`HTableDescriptor` class.
[[cp_example]]
== Examples
-HBase ships Coprocessor examples for Observer Coprocessor see
-// Below URL is more than 100 characters long.
+HBase ships examples for Observer Coprocessor in
link:http://hbase.apache.org/xref/org/apache/hadoop/hbase/coprocessor/example/ZooKeeperScanPolicyObserver.html[ZooKeeperScanPolicyObserver]
-and for Endpoint Coprocessor see
-// Below URL is more than 100 characters long.
+and for Endpoint Coprocessor in
link:http://hbase.apache.org/xref/org/apache/hadoop/hbase/coprocessor/example/RowCountEndpoint.html[RowCountEndpoint]
A more detailed example is given below.
-For the sake of example let's take an hypothetical case. Suppose there is a HBase table called
-'users'. The table has two column families 'personalDet' and 'salaryDet' containing personal
-details and salary details respectively. Below is the graphical representation of the 'users'
-table.
+These examples assume a table called `users`, which has two column families `personalDet`
+and `salaryDet`, containing personal and salary details. Below is the graphical representation
+of the `users` table.
.Users Table
[width="100%",cols="7",options="header,footer"]
@@ -509,26 +475,22 @@ table.
|====================
-
=== Observer Example
-For the purpose of demonstration of Coprocessor we are assuming that 'admin' is a special person
-and his details shouldn't be visible or returned to any client querying the 'users' table. +
-To implement this functionality we will take the help of Observer Coprocessor.
-Following are the implementation steps:
+
+The following Observer coprocessor prevents the details of the user `admin` from being
+returned in a `Get` or `Scan` of the `users` table.
. Write a class that extends the
link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.html[BaseRegionObserver]
class.
-. Override the 'preGetOp()' method (Note that 'preGet()' method is now deprecated). The reason for
-overriding this method is to check if the client has queried for the rowkey with value 'admin' or
-not. If the client has queried rowkey with 'admin' value then return the call without allowing the
-system to perform the get operation thus saving on performance, otherwise process the request as
-normal.
+. Override the `preGetOp()` method (the `preGet()` method is deprecated) to check
+whether the client has queried for the rowkey with value `admin`. If so, return an
+empty result. Otherwise, process the request as normal.
-. Put your code and dependencies in the jar file.
+. Put your code and dependencies in a JAR file.
-. Place the jar in HDFS where HBase can locate it.
+. Place the JAR in HDFS where HBase can locate it.
. Load the Coprocessor.
@@ -536,8 +498,7 @@ normal.
Following are the implementation of the above steps:
-. For Step 1 and Step 2, below is the code.
-+
+
[source,java]
----
public class RegionObserverExample extends BaseRegionObserver {
@@ -568,10 +529,10 @@ public class RegionObserverExample extends BaseRegionObserver {
}
}
----
-Overriding the 'preGetOp()' will only work for 'Get' operation. For 'Scan' operation it won't help
-you. To deal with it you have to override another method called 'preScannerOpen()' method, and
-add a Filter explicitly for admin as shown below:
-+
+
+Overriding the `preGetOp()` will only work for `Get` operations. You also need to override
+the `preScannerOpen()` method to filter the `admin` row from scan results.
+
[source,java]
----
@Override
@@ -583,12 +544,11 @@ final RegionScanner s) throws IOException {
return s;
}
----
-+
-This method works but there is a _side effect_. If the client has used any Filter in his scan,
-then that Filter won't have any effect because our filter has replaced it. +
-Another option you can try is to deliberately remove the admin from result. This approach is
-shown below:
-+
+
+This method works but there is a _side effect_. If the client has used a filter in
+its scan, that filter will be replaced by this filter. Instead, you can explicitly
+remove any `admin` results from the scan:
+
[source,java]
----
@Override
@@ -597,9 +557,9 @@ final List results, final int limit, final boolean hasMore) throws IOException {
Result result = null;
Iterator iterator = results.iterator();
while (iterator.hasNext()) {
- result = iterator.next();
+ result = iterator.next();
if (Bytes.equals(result.getRow(), ROWKEY)) {
- iterator.remove();
+ iterator.remove();
break;
}
}
@@ -607,76 +567,12 @@ final List results, final int limit, final boolean hasMore) throws IOException {
}
----
-. Step 3: It's pretty convenient to export the above program in a jar file. Let's assume that was
-exported in a file called 'coprocessor.jar'.
-
-. Step 4: Copy the jar to HDFS. You may use command like this:
-+
-[source]
-----
-hadoop fs -copyFromLocal coprocessor.jar coprocessor.jar
-----
-
-. Step 5: Load the Coprocessor, see <<cp_loading,Loading of Coprocessor>>.
-
-. Step 6: Run the following program to test. The first part is testing 'Get' and second 'Scan'.
-+
-[source,java]
-----
-Configuration conf = HBaseConfiguration.create();
-// Use below code for HBase verion 1.x.x or above.
-Connection connection = ConnectionFactory.createConnection(conf);
-TableName tableName = TableName.valueOf("users");
-Table table = connection.getTable(tableName);
-
-//Use below code HBase verion 0.98.xx or below.
-//HConnection connection = HConnectionManager.createConnection(conf);
-//HTableInterface table = connection.getTable("users");
-
-Get get = new Get(Bytes.toBytes("admin"));
-Result result = table.get(get);
-for (Cell c : result.rawCells()) {
- System.out.println(Bytes.toString(CellUtil.cloneRow(c))
- + "==> " + Bytes.toString(CellUtil.cloneFamily(c))
- + "{" + Bytes.toString(CellUtil.cloneQualifier(c))
- + ":" + Bytes.toLong(CellUtil.cloneValue(c)) + "}");
-}
-Scan scan = new Scan();
-ResultScanner scanner = table.getScanner(scan);
-for (Result res : scanner) {
- for (Cell c : res.rawCells()) {
- System.out.println(Bytes.toString(CellUtil.cloneRow(c))
- + " ==> " + Bytes.toString(CellUtil.cloneFamily(c))
- + " {" + Bytes.toString(CellUtil.cloneQualifier(c))
- + ":" + Bytes.toLong(CellUtil.cloneValue(c))
- + "}");
- }
-}
-----
-
=== Endpoint Example
-In our hypothetical example (See Users Table), to demonstrate the Endpoint Coprocessor we see a
-trivial use case in which we will try to calculate the total (Sum) of gross salary of all
-employees. One way of implementing Endpoint Coprocessor (for version 0.96 and above) is as follows:
+Still using the `users` table, this example implements a coprocessor to calculate
+the sum of all employee salaries, using an endpoint coprocessor.
. Create a '.proto' file defining your service.
-
-. Execute the 'protoc' command to generate the Java code from the above '.proto' file.
-
-. Write a class that should:
-.. Extend the above generated service class.
-.. It should also implement two interfaces Coprocessor and CoprocessorService.
-.. Override the service method.
-
-. Load the Coprocessor.
-
-. Write a client code to call Coprocessor.
-
-Implementation detail of the above steps is as follows:
-
-. Step 1: Create a 'proto' file to define your service, request and response. Let's call this file
-"sum.proto". Below is the content of the 'sum.proto' file.
+
[source]
----
@@ -700,26 +596,25 @@ service SumService {
}
----
-. Step 2: Compile the proto file using proto compiler (for detailed instructions see the
-link:https://developers.google.com/protocol-buffers/docs/overview[official documentation]).
+. Execute the `protoc` command to generate the Java code from the above .proto' file.
+
[source]
----
+$ mkdir src
$ protoc --java_out=src ./sum.proto
----
+
-[note]
-----
-(Note: It is necessary for you to create the src folder).
-This will generate a class call "Sum.java".
-----
+This will generate a class call `Sum.java`.
-. Step 3: Write your Endpoint Coprocessor: Firstly your class should extend the service just
-defined above (i.e. Sum.SumService). Second it should implement Coprocessor and CoprocessorService
-interfaces. Third, override the 'getService()', 'start()', 'stop()' and 'getSum()' methods.
-Below is the full code:
+. Write a class that extends the generated service class, implement the `Coprocessor`
+and `CoprocessorService` classes, and override the service method.
+
-[source,java]
+WARNING: If you load a coprocessor from `hbase-site.xml` and then load the same coprocessor
+again using HBase Shell, it will be loaded a second time. The same class will
+exist twice, and the second instance will have a higher ID (and thus a lower priority).
+The effect is that the duplicate coprocessor is effectively ignored.
++
+[source, java]
----
public class SumEndPoint extends SumService implements Coprocessor, CoprocessorService {
@@ -779,22 +674,16 @@ public class SumEndPoint extends SumService implements Coprocessor, CoprocessorS
}
}
----
-
-. Step 4: Load the Coprocessor. See <<cp_loading,loading of Coprocessor>>.
-
-. Step 5: Now we have to write the client code to test it. To do so in your main method, write the
-following code as shown below:
+
-[source,java]
+[source, java]
----
-
Configuration conf = HBaseConfiguration.create();
-// Use below code for HBase verion 1.x.x or above.
+// Use below code for HBase version 1.x.x or above.
Connection connection = ConnectionFactory.createConnection(conf);
TableName tableName = TableName.valueOf("users");
Table table = connection.getTable(tableName);
-//Use below code HBase verion 0.98.xx or below.
+//Use below code HBase version 0.98.xx or below.
//HConnection connection = HConnectionManager.createConnection(conf);
//HTableInterface table = connection.getTable("users");
@@ -821,6 +710,86 @@ e.printStackTrace();
}
----
+. Load the Coprocessor.
+
+. Write a client code to call the Coprocessor.
+
+
+== Guidelines For Deploying A Coprocessor
+
+Bundling Coprocessors::
+ You can bundle all classes for a coprocessor into a
+ single JAR on the RegionServer's classpath, for easy deployment. Otherwise,
+ place all dependencies on the RegionServer's classpath so that they can be
+ loaded during RegionServer start-up. The classpath for a RegionServer is set
+ in the RegionServer's `hbase-env.sh` file.
+Automating Deployment::
+ You can use a tool such as Puppet, Chef, or
+ Ansible to ship the JAR for the coprocessor to the required location on your
+ RegionServers' filesystems and restart each RegionServer, to automate
+ coprocessor deployment. Details for such set-ups are out of scope of this
+ document.
+Updating a Coprocessor::
+ Deploying a new version of a given coprocessor is not as simple as disabling it,
+ replacing the JAR, and re-enabling the coprocessor. This is because you cannot
+ reload a class in a JVM unless you delete all the current references to it.
+ Since the current JVM has reference to the existing coprocessor, you must restart
+ the JVM, by restarting the RegionServer, in order to replace it. This behavior
+ is not expected to change.
+Coprocessor Logging::
+ The Coprocessor framework does not provide an API for logging beyond standard Java
+ logging.
+Coprocessor Configuration::
+ If you do not want to load coprocessors from the HBase Shell, you can add their configuration
+ properties to `hbase-site.xml`. In <<load_coprocessor_in_shell>>, two arguments are
+ set: `arg1=1,arg2=2`. These could have been added to `hbase-site.xml` as follows:
+[source,xml]
+----
+<property>
+ <name>arg1</name>
+ <value>1</value>
+</property>
+<property>
+ <name>arg2</name>
+ <value>2</value>
+</property>
+----
+Then you can read the configuration using code like the following:
+[source,java]
+----
+Configuration conf = HBaseConfiguration.create();
+// Use below code for HBase version 1.x.x or above.
+Connection connection = ConnectionFactory.createConnection(conf);
+TableName tableName = TableName.valueOf("users");
+Table table = connection.getTable(tableName);
+
+//Use below code HBase version 0.98.xx or below.
+//HConnection connection = HConnectionManager.createConnection(conf);
+//HTableInterface table = connection.getTable("users");
+
+Get get = new Get(Bytes.toBytes("admin"));
+Result result = table.get(get);
+for (Cell c : result.rawCells()) {
+ System.out.println(Bytes.toString(CellUtil.cloneRow(c))
+ + "==> " + Bytes.toString(CellUtil.cloneFamily(c))
+ + "{" + Bytes.toString(CellUtil.cloneQualifier(c))
+ + ":" + Bytes.toLong(CellUtil.cloneValue(c)) + "}");
+}
+Scan scan = new Scan();
+ResultScanner scanner = table.getScanner(scan);
+for (Result res : scanner) {
+ for (Cell c : res.rawCells()) {
+ System.out.println(Bytes.toString(CellUtil.cloneRow(c))
+ + " ==> " + Bytes.toString(CellUtil.cloneFamily(c))
+ + " {" + Bytes.toString(CellUtil.cloneQualifier(c))
+ + ":" + Bytes.toLong(CellUtil.cloneValue(c))
+ + "}");
+ }
+}
+----
+
+
+
== Monitor Time Spent in Coprocessors
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/datamodel.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/datamodel.adoc b/src/main/asciidoc/_chapters/datamodel.adoc
index 7e9f0ab..66d2801 100644
--- a/src/main/asciidoc/_chapters/datamodel.adoc
+++ b/src/main/asciidoc/_chapters/datamodel.adoc
@@ -93,7 +93,7 @@ The colon character (`:`) delimits the column family from the column family _qua
|===
|Row Key |Time Stamp |ColumnFamily `contents` |ColumnFamily `anchor`|ColumnFamily `people`
|"com.cnn.www" |t9 | |anchor:cnnsi.com = "CNN" |
-|"com.cnn.www" |t8 | |anchor:my.look.ca = "CNN.com" |
+|"com.cnn.www" |t8 | |anchor:my.look.ca = "CNN.com" |
|"com.cnn.www" |t6 | contents:html = "<html>..." | |
|"com.cnn.www" |t5 | contents:html = "<html>..." | |
|"com.cnn.www" |t3 | contents:html = "<html>..." | |
@@ -171,7 +171,7 @@ For more information about the internals of how Apache HBase stores data, see <<
A namespace is a logical grouping of tables analogous to a database in relation database systems.
This abstraction lays the groundwork for upcoming multi-tenancy related features:
-* Quota Management (link:https://issues.apache.org/jira/browse/HBASE-8410[HBASE-8410]) - Restrict the amount of resources (ie regions, tables) a namespace can consume.
+* Quota Management (link:https://issues.apache.org/jira/browse/HBASE-8410[HBASE-8410]) - Restrict the amount of resources (i.e. regions, tables) a namespace can consume.
* Namespace Security Administration (link:https://issues.apache.org/jira/browse/HBASE-9206[HBASE-9206]) - Provide another level of security administration for tenants.
* Region server groups (link:https://issues.apache.org/jira/browse/HBASE-6721[HBASE-6721]) - A namespace/table can be pinned onto a subset of RegionServers thus guaranteeing a course level of isolation.
@@ -257,7 +257,7 @@ For example, the columns _courses:history_ and _courses:math_ are both members o
The colon character (`:`) delimits the column family from the column family qualifier.
The column family prefix must be composed of _printable_ characters.
The qualifying tail, the column family _qualifier_, can be made of any arbitrary bytes.
-Column families must be declared up front at schema definition time whereas columns do not need to be defined at schema time but can be conjured on the fly while the table is up an running.
+Column families must be declared up front at schema definition time whereas columns do not need to be defined at schema time but can be conjured on the fly while the table is up and running.
Physically, all column family members are stored together on the filesystem.
Because tunings and storage specifications are done at the column family level, it is advised that all column family members have the same general access pattern and size characteristics.
@@ -279,7 +279,7 @@ Gets are executed via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hba
=== Put
-link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html[Put] either adds new rows to a table (if the key is new) or can update existing rows (if the key already exists). Puts are executed via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#put(org.apache.hadoop.hbase.client.Put)[Table.put] (writeBuffer) or link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#batch(java.util.List, java.lang.Object[])[Table.batch] (non-writeBuffer).
+link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html[Put] either adds new rows to a table (if the key is new) or can update existing rows (if the key already exists). Puts are executed via link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#put(org.apache.hadoop.hbase.client.Put)[Table.put] (writeBuffer) or link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#batch(java.util.List,%20java.lang.Object%5B%5D)[Table.batch] (non-writeBuffer).
[[scan]]
=== Scans
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/developer.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc
index 3cfc02f..d633569 100644
--- a/src/main/asciidoc/_chapters/developer.adoc
+++ b/src/main/asciidoc/_chapters/developer.adoc
@@ -90,7 +90,7 @@ We used to be on SVN.
We migrated.
See link:https://issues.apache.org/jira/browse/INFRA-7768[Migrate Apache HBase SVN Repos to Git].
See link:http://hbase.apache.org/source-repository.html[Source Code
- Management] page for contributor and committer links or seach for HBase on the link:http://git.apache.org/[Apache Git] page.
+ Management] page for contributor and committer links or search for HBase on the link:http://git.apache.org/[Apache Git] page.
== IDEs
@@ -133,7 +133,7 @@ If you cloned the project via git, download and install the Git plugin (EGit). A
==== HBase Project Setup in Eclipse using `m2eclipse`
The easiest way is to use the +m2eclipse+ plugin for Eclipse.
-Eclipse Indigo or newer includes +m2eclipse+, or you can download it from link:http://www.eclipse.org/m2e//. It provides Maven integration for Eclipse, and even lets you use the direct Maven commands from within Eclipse to compile and test your project.
+Eclipse Indigo or newer includes +m2eclipse+, or you can download it from http://www.eclipse.org/m2e/. It provides Maven integration for Eclipse, and even lets you use the direct Maven commands from within Eclipse to compile and test your project.
To import the project, click and select the HBase root directory. `m2eclipse` locates all the hbase modules for you.
@@ -146,7 +146,7 @@ If you install +m2eclipse+ and import HBase in your workspace, do the following
----
Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project hbase:
-'An Ant BuildException has occured: Replace: source file .../target/classes/hbase-default.xml
+'An Ant BuildException has occurred: Replace: source file .../target/classes/hbase-default.xml
doesn't exist
----
+
@@ -213,7 +213,7 @@ For additional information on setting up Eclipse for HBase development on Window
=== IntelliJ IDEA
-You can set up IntelliJ IDEA for similar functinoality as Eclipse.
+You can set up IntelliJ IDEA for similar functionality as Eclipse.
Follow these steps.
. Select
@@ -227,7 +227,7 @@ Using the Eclipse Code Formatter plugin for IntelliJ IDEA, you can import the HB
=== Other IDEs
-It would be userful to mirror the <<eclipse,eclipse>> set-up instructions for other IDEs.
+It would be useful to mirror the <<eclipse,eclipse>> set-up instructions for other IDEs.
If you would like to assist, please have a look at link:https://issues.apache.org/jira/browse/HBASE-11704[HBASE-11704].
[[build]]
@@ -331,13 +331,13 @@ Tests may not all pass so you may need to pass `-DskipTests` unless you are incl
====
You will see ERRORs like the above title if you pass the _default_ profile; e.g.
if you pass +hadoop.profile=1.1+ when building 0.96 or +hadoop.profile=2.0+ when building hadoop 0.98; just drop the hadoop.profile stipulation in this case to get your build to run again.
-This seems to be a maven pecularity that is probably fixable but we've not spent the time trying to figure it.
+This seems to be a maven peculiarity that is probably fixable but we've not spent the time trying to figure it.
====
Similarly, for 3.0, you would just replace the profile value.
-Note that Hadoop-3.0.0-SNAPSHOT does not currently have a deployed maven artificat - you will need to build and install your own in your local maven repository if you want to run against this profile.
+Note that Hadoop-3.0.0-SNAPSHOT does not currently have a deployed maven artifact - you will need to build and install your own in your local maven repository if you want to run against this profile.
-In earilier versions of Apache HBase, you can build against older versions of Apache Hadoop, notably, Hadoop 0.22.x and 0.23.x.
+In earlier versions of Apache HBase, you can build against older versions of Apache Hadoop, notably, Hadoop 0.22.x and 0.23.x.
If you are running, for example HBase-0.94 and wanted to build against Hadoop 0.23.x, you would run with:
[source,bourne]
@@ -415,7 +415,7 @@ mvn -DskipTests package assembly:single deploy
==== Build Gotchas
If you see `Unable to find resource 'VM_global_library.vm'`, ignore it.
-Its not an error.
+It's not an error.
It is link:http://jira.codehaus.org/browse/MSITE-286[officially
ugly] though.
@@ -504,7 +504,7 @@ For building earlier versions, the process is different.
See this section under the respective release documentation folders.
.Point Releases
-If you are making a point release (for example to quickly address a critical incompatability or security problem) off of a release branch instead of a development branch, the tagging instructions are slightly different.
+If you are making a point release (for example to quickly address a critical incompatibility or security problem) off of a release branch instead of a development branch, the tagging instructions are slightly different.
I'll prefix those special steps with _Point Release Only_.
.Before You Begin
@@ -516,7 +516,7 @@ You should also have tried recent branch tips out on a cluster under load, perha
[NOTE]
====
At this point you should tag the previous release branch (ex: 0.96.1) with the new point release tag (e.g.
-0.96.1.1 tag). Any commits with changes for the point release should be appled to the new tag.
+0.96.1.1 tag). Any commits with changes for the point release should be applied to the new tag.
====
The Hadoop link:http://wiki.apache.org/hadoop/HowToRelease[How To
@@ -584,8 +584,8 @@ $ mvn clean install -DskipTests assembly:single -Dassembly.file=hbase-assembly/s
Extract the tarball and make sure it looks good.
A good test for the src tarball being 'complete' is to see if you can build new tarballs from this source bundle.
If the source tarball is good, save it off to a _version directory_, a directory somewhere where you are collecting all of the tarballs you will publish as part of the release candidate.
-For example if you were building a hbase-0.96.0 release candidate, you might call the directory _hbase-0.96.0RC0_.
-Later you will publish this directory as our release candidate up on http://people.apache.org/~YOU.
+For example if you were building an hbase-0.96.0 release candidate, you might call the directory _hbase-0.96.0RC0_.
+Later you will publish this directory as our release candidate up on pass:[http://people.apache.org/~YOU].
. Build the binary tarball.
+
@@ -1146,7 +1146,7 @@ However, maven will do this for us; just use: +mvn
This is very similar to how you specify running a subset of unit tests (see above), but use the property `it.test` instead of `test`.
To just run `IntegrationTestClassXYZ.java`, use: +mvn
- failsafe:integration-test -Dit.test=IntegrationTestClassXYZ+ The next thing you might want to do is run groups of integration tests, say all integration tests that are named IntegrationTestClassX*.java: +mvn failsafe:integration-test -Dit.test=*ClassX*+ This runs everything that is an integration test that matches *ClassX*. This means anything matching: "**/IntegrationTest*ClassX*". You can also run multiple groups of integration tests using comma-delimited lists (similar to unit tests). Using a list of matches still supports full regex matching for each of the groups.This would look something like: +mvn
+ failsafe:integration-test -Dit.test=IntegrationTestClassXYZ+ The next thing you might want to do is run groups of integration tests, say all integration tests that are named IntegrationTestClassX*.java: +mvn failsafe:integration-test -Dit.test=*ClassX*+ This runs everything that is an integration test that matches *ClassX*. This means anything matching: "**/IntegrationTest*ClassX*". You can also run multiple groups of integration tests using comma-delimited lists (similar to unit tests). Using a list of matches still supports full regex matching for each of the groups. This would look something like: +mvn
failsafe:integration-test -Dit.test=*ClassX*, *ClassY+
[[maven.build.commands.integration.tests.distributed]]
@@ -1183,8 +1183,9 @@ For other deployment options, a ClusterManager can be implemented and plugged in
[[maven.build.commands.integration.tests.destructive]]
==== Destructive integration / system tests (ChaosMonkey)
-HBase 0.96 introduced a tool named `ChaosMonkey`, modeled after link:http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
-[same-named tool by Netflix's Chaos Monkey tool]. ChaosMonkey simulates real-world
+HBase 0.96 introduced a tool named `ChaosMonkey`, modeled after
+link:http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html[same-named tool by Netflix's Chaos Monkey tool].
+ChaosMonkey simulates real-world
faults in a running cluster by killing or disconnecting random servers, or injecting
other failures into the environment. You can use ChaosMonkey as a stand-alone tool
to run a policy while other tests are running. In some environments, ChaosMonkey is
@@ -1262,8 +1263,8 @@ HBase ships with several ChaosMonkey policies, available in the
[[chaos.monkey.properties]]
==== Configuring Individual ChaosMonkey Actions
-Since HBase version 1.0.0 (link:https://issues.apache.org/jira/browse/HBASE-11348
-[HBASE-11348]), ChaosMonkey integration tests can be configured per test run.
+Since HBase version 1.0.0 (link:https://issues.apache.org/jira/browse/HBASE-11348[HBASE-11348]),
+ChaosMonkey integration tests can be configured per test run.
Create a Java properties file in the HBase classpath and pass it to ChaosMonkey using
the `-monkeyProps` configuration flag. Configurable properties, along with their default
values if applicable, are listed in the `org.apache.hadoop.hbase.chaos.factories.MonkeyConstants`
@@ -1344,6 +1345,10 @@ NOTE: End-of-life releases are not included in this list.
|===
| Release
| Release Manager
+
+| 0.94
+| Lars Hofhansl
+
| 0.98
| Andrew Purtell
@@ -1604,7 +1609,7 @@ All are subject to challenge of course but until then, please hold to the rules
ZooKeeper state should transient (treat it like memory). If ZooKeeper state is deleted, hbase should be able to recover and essentially be in the same state.
-* .ExceptionsThere are currently a few exceptions that we need to fix around whether a table is enabled or disabled.
+* .Exceptions: There are currently a few exceptions that we need to fix around whether a table is enabled or disabled.
* Replication data is currently stored only in ZooKeeper.
Deleting ZooKeeper data related to replication may cause replication to be disabled.
Do not delete the replication tree, _/hbase/replication/_.
@@ -1866,9 +1871,9 @@ If the push fails for any reason, fix the problem or ask for help.
Do not do a +git push --force+.
+
Before you can commit a patch, you need to determine how the patch was created.
-The instructions and preferences around the way to create patches have changed, and there will be a transition periond.
+The instructions and preferences around the way to create patches have changed, and there will be a transition period.
+
-* .Determine How a Patch Was CreatedIf the first few lines of the patch look like the headers of an email, with a From, Date, and Subject, it was created using +git format-patch+.
+* .Determine How a Patch Was Created: If the first few lines of the patch look like the headers of an email, with a From, Date, and Subject, it was created using +git format-patch+.
This is the preference, because you can reuse the submitter's commit message.
If the commit message is not appropriate, you can still use the commit, then run the command +git
rebase -i origin/master+, and squash and reword as appropriate.
@@ -1971,7 +1976,7 @@ When the amending author is different from the original committer, add notice of
from master to branch].
[[committer.tests]]
-====== Committers are responsible for making sure commits do not break thebuild or tests
+====== Committers are responsible for making sure commits do not break the build or tests
If a committer commits a patch, it is their responsibility to make sure it passes the test suite.
It is helpful if contributors keep an eye out that their patch does not break the hbase build and/or tests, but ultimately, a contributor cannot be expected to be aware of all the particular vagaries and interconnections that occur in a project like HBase.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/external_apis.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/external_apis.adoc b/src/main/asciidoc/_chapters/external_apis.adoc
index 44603f0..43a428a 100644
--- a/src/main/asciidoc/_chapters/external_apis.adoc
+++ b/src/main/asciidoc/_chapters/external_apis.adoc
@@ -77,7 +77,7 @@ of the <<security>> chapter.
=== Using REST Endpoints
-The following examples use the placeholder server `http://example.com:8000`, and
+The following examples use the placeholder server pass:[http://example.com:8000], and
the following commands can all be run using `curl` or `wget` commands. You can request
plain text (the default), XML , or JSON output by adding no header for plain text,
or the header "Accept: text/xml" for XML or "Accept: application/json" for JSON.
@@ -741,7 +741,7 @@ the data, and deletes the table.
[source,jython]
----
import java.lang
-from org.apache.hadoop.hbase import HBaseConfiguration, HTableDescriptor, HColumnDescriptor, HConstants
+from org.apache.hadoop.hbase import HBaseConfiguration, HTableDescriptor, HColumnDescriptor, HConstants, TableName
from org.apache.hadoop.hbase.client import HBaseAdmin, HTable, Get
from org.apache.hadoop.hbase.io import Cell, RowResult
@@ -753,7 +753,7 @@ conf = HBaseConfiguration()
# Create a table named 'test' that has two column families,
# one named 'content, and the other 'anchor'. The colons
# are required for column family names.
-tablename = "test"
+tablename = TableName.valueOf("test")
desc = HTableDescriptor(tablename)
desc.addFamily(HColumnDescriptor("content:"))
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/faq.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/faq.adoc b/src/main/asciidoc/_chapters/faq.adoc
index 22e4ad3..a622650 100644
--- a/src/main/asciidoc/_chapters/faq.adoc
+++ b/src/main/asciidoc/_chapters/faq.adoc
@@ -46,7 +46,7 @@ What is the history of HBase?::
=== Upgrading
How do I upgrade Maven-managed projects from HBase 0.94 to HBase 0.96+?::
- In HBase 0.96, the project moved to a modular structure. Adjust your project's dependencies to rely upon the `hbase-client` module or another module as appropriate, rather than a single JAR. You can model your Maven depency after one of the following, depending on your targeted version of HBase. See Section 3.5, “Upgrading from 0.94.x to 0.96.x” or Section 3.3, “Upgrading from 0.96.x to 0.98.x” for more information.
+ In HBase 0.96, the project moved to a modular structure. Adjust your project's dependencies to rely upon the `hbase-client` module or another module as appropriate, rather than a single JAR. You can model your Maven dependency after one of the following, depending on your targeted version of HBase. See Section 3.5, “Upgrading from 0.94.x to 0.96.x” or Section 3.3, “Upgrading from 0.96.x to 0.98.x” for more information.
+
.Maven Dependency for HBase 0.98
[source,xml]
@@ -55,18 +55,18 @@ How do I upgrade Maven-managed projects from HBase 0.94 to HBase 0.96+?::
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.98.5-hadoop2</version>
-</dependency>
-----
-+
-.Maven Dependency for HBase 0.96
+</dependency>
+----
++
+.Maven Dependency for HBase 0.96
[source,xml]
----
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.96.2-hadoop2</version>
-</dependency>
-----
+</dependency>
+----
+
.Maven Dependency for HBase 0.94
[source,xml]
@@ -75,9 +75,9 @@ How do I upgrade Maven-managed projects from HBase 0.94 to HBase 0.96+?::
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.94.3</version>
-</dependency>
-----
-
+</dependency>
+----
+
=== Architecture
How does HBase handle Region-RegionServer assignment and locality?::
@@ -91,7 +91,7 @@ Where can I learn about the rest of the configuration options?::
See <<configuration>>.
=== Schema Design / Data Access
-
+
How should I design my schema in HBase?::
See <<datamodel>> and <<schema>>.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/getting_started.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/getting_started.adoc b/src/main/asciidoc/_chapters/getting_started.adoc
index 4134831..1b38e6e 100644
--- a/src/main/asciidoc/_chapters/getting_started.adoc
+++ b/src/main/asciidoc/_chapters/getting_started.adoc
@@ -57,7 +57,7 @@ Prior to HBase 0.94.x, HBase expected the loopback IP address to be 127.0.0.1. U
.Example /etc/hosts File for Ubuntu
====
-The following _/etc/hosts_ file works correctly for HBase 0.94.x and earlier, on Ubuntu. Use this as a template if you run into trouble.
+The following _/etc/hosts_ file works correctly for HBase 0.94.x and earlier, on Ubuntu. Use this as a template if you run into trouble.
[listing]
----
127.0.0.1 localhost
[4/4] hbase git commit: updating docs from master
Posted by nd...@apache.org.
updating docs from master
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/c07ddc6d
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/c07ddc6d
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/c07ddc6d
Branch: refs/heads/branch-1.1
Commit: c07ddc6dbeb225b7145b9d884a6a5cc5022331d7
Parents: 5a1cfc1
Author: Nick Dimiduk <nd...@apache.org>
Authored: Sat Jan 16 16:00:17 2016 -0800
Committer: Nick Dimiduk <nd...@apache.org>
Committed: Sat Jan 16 16:00:17 2016 -0800
----------------------------------------------------------------------
.../asciidoc/_chapters/appendix_acl_matrix.adoc | 2 +-
.../appendix_contributing_to_documentation.adoc | 13 +-
.../_chapters/appendix_hfile_format.adoc | 22 +-
src/main/asciidoc/_chapters/architecture.adoc | 233 +++---
src/main/asciidoc/_chapters/asf.adoc | 4 +-
src/main/asciidoc/_chapters/case_studies.adoc | 2 +-
src/main/asciidoc/_chapters/community.adoc | 34 +-
src/main/asciidoc/_chapters/compression.adoc | 40 +-
src/main/asciidoc/_chapters/configuration.adoc | 33 +-
src/main/asciidoc/_chapters/cp.adoc | 715 +++++++++----------
src/main/asciidoc/_chapters/datamodel.adoc | 8 +-
src/main/asciidoc/_chapters/developer.adoc | 49 +-
src/main/asciidoc/_chapters/external_apis.adoc | 6 +-
src/main/asciidoc/_chapters/faq.adoc | 22 +-
.../asciidoc/_chapters/getting_started.adoc | 2 +-
src/main/asciidoc/_chapters/hbase-default.adoc | 527 +++++++-------
src/main/asciidoc/_chapters/hbase_history.adoc | 8 +-
src/main/asciidoc/_chapters/hbck_in_depth.adoc | 24 +-
src/main/asciidoc/_chapters/mapreduce.adoc | 8 +-
src/main/asciidoc/_chapters/ops_mgt.adoc | 54 +-
src/main/asciidoc/_chapters/other_info.adoc | 34 +-
src/main/asciidoc/_chapters/performance.adoc | 21 +-
src/main/asciidoc/_chapters/rpc.adoc | 22 +-
src/main/asciidoc/_chapters/schema_design.adoc | 125 +++-
src/main/asciidoc/_chapters/security.adoc | 60 +-
src/main/asciidoc/_chapters/shell.adoc | 2 +-
src/main/asciidoc/_chapters/spark.adoc | 451 ++++++++++++
src/main/asciidoc/_chapters/tracing.adoc | 30 +-
.../asciidoc/_chapters/troubleshooting.adoc | 12 +-
src/main/asciidoc/_chapters/unit_testing.adoc | 32 +-
src/main/asciidoc/_chapters/upgrading.adoc | 8 +-
src/main/asciidoc/_chapters/zookeeper.adoc | 30 +-
src/main/asciidoc/book.adoc | 1 +
33 files changed, 1607 insertions(+), 1027 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc b/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
index cb285f3..698ae82 100644
--- a/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
+++ b/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
@@ -65,7 +65,7 @@ Possible permissions include the following:
For the most part, permissions work in an expected way, with the following caveats:
Having Write permission does not imply Read permission.::
- It is possible and sometimes desirable for a user to be able to write data that same user cannot read. One such example is a log-writing process.
+ It is possible and sometimes desirable for a user to be able to write data that same user cannot read. One such example is a log-writing process.
The [systemitem]+hbase:meta+ table is readable by every user, regardless of the user's other grants or restrictions.::
This is a requirement for HBase to function correctly.
`CheckAndPut` and `CheckAndDelete` operations will fail if the user does not have both Write and Read permission.::
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc b/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
index 1b674e7..4588e95 100644
--- a/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
+++ b/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
@@ -125,7 +125,7 @@ This directory also stores images used in the HBase Reference Guide.
The website's pages are written in an HTML-like XML dialect called xdoc, which
has a reference guide at
-link:http://maven.apache.org/archives/maven-1.x/plugins/xdoc/reference/xdocs.html.
+http://maven.apache.org/archives/maven-1.x/plugins/xdoc/reference/xdocs.html.
You can edit these files in a plain-text editor, an IDE, or an XML editor such
as XML Mind XML Editor (XXE) or Oxygen XML Author.
@@ -159,7 +159,7 @@ artifacts to the 0.94/ directory of the `asf-site` branch.
The HBase Reference Guide is written in Asciidoc and built using link:http://asciidoctor.org[AsciiDoctor].
The following cheat sheet is included for your reference. More nuanced and comprehensive documentation
-is available at link:http://asciidoctor.org/docs/user-manual/.
+is available at http://asciidoctor.org/docs/user-manual/.
.AsciiDoc Cheat Sheet
[cols="1,1,a",options="header"]
@@ -186,7 +186,8 @@ is available at link:http://asciidoctor.org/docs/user-manual/.
include\::path/to/app.rb[]
----
................
-| Include only part of a separate file | Similar to Javadoc | See link:http://asciidoctor.org/docs/user-manual/#by-tagged-regions
+| Include only part of a separate file | Similar to Javadoc
+| See http://asciidoctor.org/docs/user-manual/#by-tagged-regions
| Filenames, directory names, new terms | italic | \_hbase-default.xml_
| External naked URLs | A link with the URL as link text |
----
@@ -285,7 +286,11 @@ Title:: content
Title::
content
----
-| Sidebars, quotes, or other blocks of text | a block of text, formatted differently from the default | Delimited using different delimiters, see link:http://asciidoctor.org/docs/user-manual/#built-in-blocks-summary. Some of the examples above use delimiters like \...., ----,====.
+| Sidebars, quotes, or other blocks of text
+| a block of text, formatted differently from the default
+| Delimited using different delimiters,
+see http://asciidoctor.org/docs/user-manual/#built-in-blocks-summary.
+Some of the examples above use delimiters like \...., ----,====.
........
[example]
====
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/appendix_hfile_format.adoc b/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
index d73ddfb..18eafe6 100644
--- a/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
+++ b/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
@@ -192,8 +192,11 @@ This format applies to intermediate-level and leaf index blocks of a version 2 m
Every non-root index block is structured as follows.
. numEntries: the number of entries (int).
-. entryOffsets: the ``secondary index'' of offsets of entries in the block, to facilitate a quick binary search on the key (numEntries + 1 int values). The last value is the total length of all entries in this index block.
- For example, in a non-root index block with entry sizes 60, 80, 50 the ``secondary index'' will contain the following int array: {0, 60, 140, 190}.
+. entryOffsets: the "secondary index" of offsets of entries in the block, to facilitate
+ a quick binary search on the key (`numEntries + 1` int values). The last value
+ is the total length of all entries in this index block. For example, in a non-root
+ index block with entry sizes 60, 80, 50 the "secondary index" will contain the
+ following int array: `{0, 60, 140, 190}`.
. Entries.
Each entry contains:
+
@@ -222,7 +225,7 @@ In contrast with version 1, in a version 2 HFile Bloom filter metadata is stored
==== File Info format in versions 1 and 2
-The file info block is a serialized link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/HbaseMapWritable.html[HbaseMapWritable] (essentially a map from byte arrays to byte arrays) with the following keys, among others.
+The file info block is a serialized map from byte arrays to byte arrays, with the following keys, among others.
StoreFile-level logic adds more keys to this.
[cols="1,1", frame="all"]
@@ -232,9 +235,11 @@ StoreFile-level logic adds more keys to this.
|hfile.AVG_VALUE_LEN| The average value length in the file (int)
|===
-File info format did not change in version 2.
-However, we moved the file info to the final section of the file, which can be loaded as one block at the time the HFile is being opened.
-Also, we do not store comparator in the version 2 file info anymore.
+In version 2, we did not change the file format, but we moved the file info to
+the final section of the file, which can be loaded as one block when the HFile
+is being opened.
+
+Also, we do not store the comparator in the version 2 file info anymore.
Instead, we store it in the fixed file trailer.
This is because we need to know the comparator at the time of parsing the load-on-open section of the HFile.
@@ -249,7 +254,8 @@ However, the version is always stored as the last four-byte integer in the file.
|===
| Version 1 | Version 2
| |File info offset (long)
-| Data index offset (long)| loadOnOpenOffset (long) /The offset of the sectionthat we need toload when opening the file./
+| Data index offset (long)
+| loadOnOpenOffset (long) /The offset of the section that we need to load when opening the file./
| | Number of data index entries (int)
| metaIndexOffset (long) /This field is not being used by the version 1 reader, so we removed it from version 2./ | uncompressedDataIndexSize (long) /The total uncompressed size of the whole data block index, including root-level, intermediate-level, and leaf-level blocks./
| | Number of meta index entries (int)
@@ -257,7 +263,7 @@ However, the version is always stored as the last four-byte integer in the file.
| numEntries (int) | numEntries (long)
| Compression codec: 0 = LZO, 1 = GZ, 2 = NONE (int) | Compression codec: 0 = LZO, 1 = GZ, 2 = NONE (int)
| | The number of levels in the data block index (int)
-| | firstDataBlockOffset (long) /The offset of the first first data block. Used when scanning./
+| | firstDataBlockOffset (long) /The offset of the first data block. Used when scanning./
| | lastDataBlockEnd (long) /The offset of the first byte after the last key/value data block. We don't need to go beyond this offset when scanning./
| Version: 1 (int) | Version: 2 (int)
|===
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/architecture.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/architecture.adoc b/src/main/asciidoc/_chapters/architecture.adoc
index 8122e11..103f624 100644
--- a/src/main/asciidoc/_chapters/architecture.adoc
+++ b/src/main/asciidoc/_chapters/architecture.adoc
@@ -41,7 +41,8 @@ Technically speaking, HBase is really more a "Data Store" than "Data Base" becau
However, HBase has many features which supports both linear and modular scaling.
HBase clusters expand by adding RegionServers that are hosted on commodity class servers.
If a cluster expands from 10 to 20 RegionServers, for example, it doubles both in terms of storage and as well as processing capacity.
-RDBMS can scale well, but only up to a point - specifically, the size of a single database server - and for the best performance requires specialized hardware and storage devices.
+An RDBMS can scale well, but only up to a point - specifically, the size of a single database
+server - and for the best performance requires specialized hardware and storage devices.
HBase features of note are:
* Strongly consistent reads/writes: HBase is not an "eventually consistent" DataStore.
@@ -140,7 +141,7 @@ If a region has both an empty start and an empty end key, it is the only region
In the (hopefully unlikely) event that programmatic processing of catalog metadata
is required, see the
-link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/Writables.html#getHRegionInfo%28byte[]%29[Writables]
++++<a href="http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/Writables.html#getHRegionInfo%28byte%5B%5D%29">Writables</a>+++
utility.
[[arch.catalog.startup]]
@@ -172,7 +173,7 @@ The API changed in HBase 1.0. For connection configuration information, see <<cl
==== API as of HBase 1.0.0
-Its been cleaned up and users are returned Interfaces to work against rather than particular types.
+It's been cleaned up and users are returned Interfaces to work against rather than particular types.
In HBase 1.0, obtain a `Connection` object from `ConnectionFactory` and thereafter, get from it instances of `Table`, `Admin`, and `RegionLocator` on an as-need basis.
When done, close the obtained instances.
Finally, be sure to cleanup your `Connection` instance before exiting.
@@ -295,7 +296,11 @@ scan.setFilter(list);
[[client.filter.cv.scvf]]
==== SingleColumnValueFilter
-link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.html[SingleColumnValueFilter] can be used to test column values for equivalence (`CompareOp.EQUAL`), inequality (`CompareOp.NOT_EQUAL`), or ranges (e.g., `CompareOp.GREATER`). The following is example of testing equivalence a column to a String value "my value"...
+A SingleColumnValueFilter (see:
+http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.html)
+can be used to test column values for equivalence (`CompareOp.EQUAL`),
+inequality (`CompareOp.NOT_EQUAL`), or ranges (e.g., `CompareOp.GREATER`). The following is an
+example of testing equivalence of a column to a String value "my value"...
[source,java]
----
@@ -694,7 +699,8 @@ Here are others that you may have to take into account:
Catalog Tables::
The `-ROOT-` (prior to HBase 0.96, see <<arch.catalog.root,arch.catalog.root>>) and `hbase:meta` tables are forced into the block cache and have the in-memory priority which means that they are harder to evict.
- The former never uses more than a few hundreds bytes while the latter can occupy a few MBs (depending on the number of regions).
+ The former never uses more than a few hundred bytes while the latter can occupy a few MBs
+ (depending on the number of regions).
HFiles Indexes::
An _HFile_ is the file format that HBase uses to store data in HDFS.
@@ -878,7 +884,10 @@ image::region_split_process.png[Region Split Process]
. The Master learns about this znode, since it has a watcher for the parent `region-in-transition` znode.
. The RegionServer creates a sub-directory named `.splits` under the parent’s `region` directory in HDFS.
. The RegionServer closes the parent region and marks the region as offline in its local data structures. *THE SPLITTING REGION IS NOW OFFLINE.* At this point, client requests coming to the parent region will throw `NotServingRegionException`. The client will retry with some backoff. The closing region is flushed.
-. The RegionServer creates region directories under the `.splits` directory, for daughter regions A and B, and creates necessary data structures. Then it splits the store files, in the sense that it creates two link:http://www.google.com/url?q=http%3A%2F%2Fhbase.apache.org%2Fapidocs%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fio%2FReference.html&sa=D&sntz=1&usg=AFQjCNEkCbADZ3CgKHTtGYI8bJVwp663CA[Reference] files per store file in the parent region. Those reference files will point to the parent regions'files.
+. The RegionServer creates region directories under the `.splits` directory, for daughter
+regions A and B, and creates necessary data structures. Then it splits the store files,
+in the sense that it creates two Reference files per store file in the parent region.
+Those reference files will point to the parent region's files.
. The RegionServer creates the actual region directory in HDFS, and moves the reference files for each daughter.
. The RegionServer sends a `Put` request to the `.META.` table, to set the parent as offline in the `.META.` table and add information about daughter regions. At this point, there won’t be individual entries in `.META.` for the daughters. Clients will see that the parent region is split if they scan `.META.`, but won’t know about the daughters until they appear in `.META.`. Also, if this `Put` to `.META`. succeeds, the parent will be effectively split. If the RegionServer fails before this RPC succeeds, Master and the next Region Server opening the region will clean dirty state about the region split. After the `.META.` update, though, the region split will be rolled-forward by Master.
. The RegionServer opens daughters A and B in parallel.
@@ -931,7 +940,7 @@ To configure MultiWAL for a RegionServer, set the value of the property `hbase.w
</property>
----
-Restart the RegionServer for the changes to take effect.
+Restart the RegionServer for the changes to take effect.
To disable MultiWAL for a RegionServer, unset the property and restart the RegionServer.
@@ -1008,7 +1017,8 @@ If you set the `hbase.hlog.split.skip.errors` option to `true`, errors are treat
* Processing of the WAL will continue
If the `hbase.hlog.split.skip.errors` option is set to `false`, the default, the exception will be propagated and the split will be logged as failed.
-See link:https://issues.apache.org/jira/browse/HBASE-2958[HBASE-2958 When hbase.hlog.split.skip.errors is set to false, we fail the split but thats it].
+See link:https://issues.apache.org/jira/browse/HBASE-2958[HBASE-2958 When
+hbase.hlog.split.skip.errors is set to false, we fail the split but that's it].
We need to do more than just fail split if this flag is set.
====== How EOFExceptions are treated when splitting a crashed RegionServer's WALs
@@ -1117,7 +1127,8 @@ Based on the state of the task whose data is changed, the split log manager does
Each RegionServer runs a daemon thread called the _split log worker_, which does the work to split the logs.
The daemon thread starts when the RegionServer starts, and registers itself to watch HBase znodes.
If any splitlog znode children change, it notifies a sleeping worker thread to wake up and grab more tasks.
-If if a worker's current task's node data is changed, the worker checks to see if the task has been taken by another worker.
+If a worker's current task's node data is changed,
+the worker checks to see if the task has been taken by another worker.
If so, the worker thread stops work on the current task.
+
The worker monitors the splitlog znode constantly.
@@ -1127,7 +1138,7 @@ At this point, the split log worker scans for another unclaimed task.
+
.How the Split Log Worker Approaches a Task
* It queries the task state and only takes action if the task is in `TASK_UNASSIGNED `state.
-* If the task is is in `TASK_UNASSIGNED` state, the worker attempts to set the state to `TASK_OWNED` by itself.
+* If the task is in `TASK_UNASSIGNED` state, the worker attempts to set the state to `TASK_OWNED` by itself.
If it fails to set the state, another worker will try to grab it.
The split log manager will also ask all workers to rescan later if the task remains unassigned.
* If the worker succeeds in taking ownership of the task, it tries to get the task state again to make sure it really gets it asynchronously.
@@ -1135,7 +1146,7 @@ At this point, the split log worker scans for another unclaimed task.
** Get the HBase root folder, create a temp folder under the root, and split the log file to the temp folder.
** If the split was successful, the task executor sets the task to state `TASK_DONE`.
** If the worker catches an unexpected IOException, the task is set to state `TASK_ERR`.
-** If the worker is shutting down, set the the task to state `TASK_RESIGNED`.
+** If the worker is shutting down, set the task to state `TASK_RESIGNED`.
** If the task is taken by another worker, just log it.
@@ -1326,7 +1337,7 @@ image::region_states.png[]
. Before assigning a region, the master moves the region to `OFFLINE` state automatically if it is in `CLOSED` state.
. When a RegionServer is about to split a region, it notifies the master.
The master moves the region to be split from `OPEN` to `SPLITTING` state and add the two new regions to be created to the RegionServer.
- These two regions are in `SPLITING_NEW` state initially.
+ These two regions are in `SPLITTING_NEW` state initially.
. After notifying the master, the RegionServer starts to split the region.
Once past the point of no return, the RegionServer notifies the master again so the master can update the `hbase:meta` table.
However, the master does not update the region states until it is notified by the server that the split is done.
@@ -1404,8 +1415,8 @@ hbase> create 'test', {METHOD => 'table_att', CONFIG => {'SPLIT_POLICY' => 'org.
----
The default split policy can be overwritten using a custom
-link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/RegionSplitPolicy.html
-[RegionSplitPolicy(HBase 0.94+)]. Typically a custom split policy should extend HBase's default split policy:
+link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/RegionSplitPolicy.html[RegionSplitPolicy(HBase 0.94+)].
+Typically a custom split policy should extend HBase's default split policy:
link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html[ConstantSizeRegionSplitPolicy].
The policy can be set globally through the HBaseConfiguration used or on a per table basis:
@@ -1806,60 +1817,116 @@ This list is not exhaustive.
To tune these parameters from the defaults, edit the _hbase-default.xml_ file.
For a full list of all configuration parameters available, see <<config.files,config.files>>
-[cols="1,1a,1", options="header"]
-|===
-| Parameter
-| Description
-| Default
-
-|`hbase.hstore.compaction.min`
-| The minimum number of StoreFiles which must be eligible for compaction before compaction can run. The goal of tuning `hbase.hstore.compaction.min` is to avoid ending up with too many tiny StoreFiles to compact. Setting this value to 2 would cause a minor compaction each time you have two StoreFiles in a Store, and this is probably not appropriate. If you set this value too high, all the other values will need to be adjusted accordingly. For most cases, the default value is appropriate. In previous versions of HBase, the parameter hbase.hstore.compaction.min was called `hbase.hstore.compactionThreshold`.
-|3
-
-|`hbase.hstore.compaction.max`
-| The maximum number of StoreFiles which will be selected for a single minor compaction, regardless of the number of eligible StoreFiles. Effectively, the value of hbase.hstore.compaction.max controls the length of time it takes a single compaction to complete. Setting it larger means that more StoreFiles are included in a compaction. For most cases, the default value is appropriate.
-|10
-
-|`hbase.hstore.compaction.min.size`
-| A StoreFile smaller than this size will always be eligible for minor compaction. StoreFiles this size or larger are evaluated by `hbase.hstore.compaction.ratio` to determine if they are eligible. Because this limit represents the "automatic include" limit for all StoreFiles smaller than this value, this value may need to be reduced in write-heavy environments where many files in the 1-2 MB range are being flushed, because every StoreFile will be targeted for compaction and the resulting StoreFiles may still be under the minimum size and require further compaction. If this parameter is lowered, the ratio check is triggered more quickly. This addressed some issues seen in earlier versions of HBase but changing this parameter is no longer necessary in most situations.
-|128 MB
-
-|`hbase.hstore.compaction.max.size`
-| An StoreFile larger than this size will be excluded from compaction. The effect of raising `hbase.hstore.compaction.max.size` is fewer, larger StoreFiles that do not get compacted often. If you feel that compaction is happening too often without much benefit, you can try raising this value.
-|`Long.MAX_VALUE`
-
-|`hbase.hstore.compaction.ratio`
-| For minor compaction, this ratio is used to determine whether a given StoreFile which is larger than `hbase.hstore.compaction.min.size` is eligible for compaction. Its effect is to limit compaction of large StoreFile. The value of `hbase.hstore.compaction.ratio` is expressed as a floating-point decimal.
-
-* A large ratio, such as 10, will produce a single giant StoreFile. Conversely, a value of .25, will produce behavior similar to the BigTable compaction algorithm, producing four StoreFiles.
-* A moderate value of between 1.0 and 1.4 is recommended. When tuning this value, you are balancing write costs with read costs. Raising the value (to something like 1.4) will have more write costs, because you will compact larger StoreFiles. However, during reads, HBase will need to seek through fewer StoreFiles to accomplish the read. Consider this approach if you cannot take advantage of <<bloom>>.
-* Alternatively, you can lower this value to something like 1.0 to reduce the background cost of writes, and use to limit the number of StoreFiles touched during reads. For most cases, the default value is appropriate.
-| `1.2F`
-
-|`hbase.hstore.compaction.ratio.offpeak`
-| The compaction ratio used during off-peak compactions, if off-peak hours are also configured (see below). Expressed as a floating-point decimal. This allows for more aggressive (or less aggressive, if you set it lower than `hbase.hstore.compaction.ratio`) compaction during a set time period. Ignored if off-peak is disabled (default). This works the same as hbase.hstore.compaction.ratio.
-| `5.0F`
+`hbase.hstore.compaction.min`::
+ The minimum number of StoreFiles which must be eligible for compaction before compaction can run.
+ The goal of tuning `hbase.hstore.compaction.min` is to avoid ending up with too many tiny StoreFiles
+ to compact. Setting this value to 2 would cause a minor compaction each time you have two StoreFiles
+ in a Store, and this is probably not appropriate. If you set this value too high, all the other
+ values will need to be adjusted accordingly. For most cases, the default value is appropriate.
+ In previous versions of HBase, the parameter `hbase.hstore.compaction.min` was called
+ `hbase.hstore.compactionThreshold`.
++
+*Default*: 3
+
+`hbase.hstore.compaction.max`::
+ The maximum number of StoreFiles which will be selected for a single minor compaction,
+ regardless of the number of eligible StoreFiles. Effectively, the value of
+ `hbase.hstore.compaction.max` controls the length of time it takes a single
+ compaction to complete. Setting it larger means that more StoreFiles are included
+ in a compaction. For most cases, the default value is appropriate.
++
+*Default*: 10
+
+`hbase.hstore.compaction.min.size`::
+ A StoreFile smaller than this size will always be eligible for minor compaction.
+ StoreFiles this size or larger are evaluated by `hbase.hstore.compaction.ratio`
+ to determine if they are eligible. Because this limit represents the "automatic
+ include" limit for all StoreFiles smaller than this value, this value may need
+ to be reduced in write-heavy environments where many files in the 1-2 MB range
+ are being flushed, because every StoreFile will be targeted for compaction and
+ the resulting StoreFiles may still be under the minimum size and require further
+ compaction. If this parameter is lowered, the ratio check is triggered more quickly.
+ This addressed some issues seen in earlier versions of HBase but changing this
+ parameter is no longer necessary in most situations.
++
+*Default*:128 MB
-| `hbase.offpeak.start.hour`
-| The start of off-peak hours, expressed as an integer between 0 and 23, inclusive. Set to -1 to disable off-peak.
-| `-1` (disabled)
+`hbase.hstore.compaction.max.size`::
+ A StoreFile larger than this size will be excluded from compaction. The effect of
+ raising `hbase.hstore.compaction.max.size` is fewer, larger StoreFiles that do not
+ get compacted often. If you feel that compaction is happening too often without
+ much benefit, you can try raising this value.
++
+*Default*: `Long.MAX_VALUE`
-| `hbase.offpeak.end.hour`
-| The end of off-peak hours, expressed as an integer between 0 and 23, inclusive. Set to -1 to disable off-peak.
-| `-1` (disabled)
+`hbase.hstore.compaction.ratio`::
+ For minor compaction, this ratio is used to determine whether a given StoreFile
+ which is larger than `hbase.hstore.compaction.min.size` is eligible for compaction.
+ Its effect is to limit compaction of large StoreFile. The value of
+ `hbase.hstore.compaction.ratio` is expressed as a floating-point decimal.
++
+* A large ratio, such as 10, will produce a single giant StoreFile. Conversely,
+ a value of .25, will produce behavior similar to the BigTable compaction algorithm,
+ producing four StoreFiles.
+* A moderate value of between 1.0 and 1.4 is recommended. When tuning this value,
+ you are balancing write costs with read costs. Raising the value (to something like
+ 1.4) will have more write costs, because you will compact larger StoreFiles.
+ However, during reads, HBase will need to seek through fewer StoreFiles to
+ accomplish the read. Consider this approach if you cannot take advantage of <<bloom>>.
+* Alternatively, you can lower this value to something like 1.0 to reduce the
+ background cost of writes, and use to limit the number of StoreFiles touched
+ during reads. For most cases, the default value is appropriate.
++
+*Default*: `1.2F`
+
+`hbase.hstore.compaction.ratio.offpeak`::
+ The compaction ratio used during off-peak compactions, if off-peak hours are
+ also configured (see below). Expressed as a floating-point decimal. This allows
+ for more aggressive (or less aggressive, if you set it lower than
+ `hbase.hstore.compaction.ratio`) compaction during a set time period. Ignored
+ if off-peak is disabled (default). This works the same as
+ `hbase.hstore.compaction.ratio`.
++
+*Default*: `5.0F`
-| `hbase.regionserver.thread.compaction.throttle`
-| There are two different thread pools for compactions, one for large compactions and the other for small compactions. This helps to keep compaction of lean tables (such as `hbase:meta`) fast. If a compaction is larger than this threshold, it goes into the large compaction pool. In most cases, the default value is appropriate.
-| `2 x hbase.hstore.compaction.max x hbase.hregion.memstore.flush.size` (which defaults to `128`)
+`hbase.offpeak.start.hour`::
+ The start of off-peak hours, expressed as an integer between 0 and 23, inclusive.
+ Set to -1 to disable off-peak.
++
+*Default*: `-1` (disabled)
-| `hbase.hregion.majorcompaction`
-| Time between major compactions, expressed in milliseconds. Set to 0 to disable time-based automatic major compactions. User-requested and size-based major compactions will still run. This value is multiplied by `hbase.hregion.majorcompaction.jitter` to cause compaction to start at a somewhat-random time during a given window of time.
-| 7 days (`604800000` milliseconds)
+`hbase.offpeak.end.hour`::
+ The end of off-peak hours, expressed as an integer between 0 and 23, inclusive.
+ Set to -1 to disable off-peak.
++
+*Default*: `-1` (disabled)
+
+`hbase.regionserver.thread.compaction.throttle`::
+ There are two different thread pools for compactions, one for large compactions
+ and the other for small compactions. This helps to keep compaction of lean tables
+ (such as `hbase:meta`) fast. If a compaction is larger than this threshold,
+ it goes into the large compaction pool. In most cases, the default value is
+ appropriate.
++
+*Default*: `2 x hbase.hstore.compaction.max x hbase.hregion.memstore.flush.size`
+(which defaults to `128`)
+
+`hbase.hregion.majorcompaction`::
+ Time between major compactions, expressed in milliseconds. Set to 0 to disable
+ time-based automatic major compactions. User-requested and size-based major
+ compactions will still run. This value is multiplied by
+ `hbase.hregion.majorcompaction.jitter` to cause compaction to start at a
+ somewhat-random time during a given window of time.
++
+*Default*: 7 days (`604800000` milliseconds)
-| `hbase.hregion.majorcompaction.jitter`
-| A multiplier applied to hbase.hregion.majorcompaction to cause compaction to occur a given amount of time either side of `hbase.hregion.majorcompaction`. The smaller the number, the closer the compactions will happen to the `hbase.hregion.majorcompaction` interval. Expressed as a floating-point decimal.
-| `.50F`
-|===
+`hbase.hregion.majorcompaction.jitter`::
+ A multiplier applied to hbase.hregion.majorcompaction to cause compaction to
+ occur a given amount of time either side of `hbase.hregion.majorcompaction`.
+ The smaller the number, the closer the compactions will happen to the
+ `hbase.hregion.majorcompaction` interval. Expressed as a floating-point decimal.
++
+*Default*: `.50F`
[[compaction.file.selection.old]]
===== Compaction File Selection
@@ -1916,8 +1983,8 @@ Why?
* 100 -> No, because sum(50, 23, 12, 12) * 1.0 = 97.
* 50 -> No, because sum(23, 12, 12) * 1.0 = 47.
* 23 -> Yes, because sum(12, 12) * 1.0 = 24.
-* 12 -> Yes, because the previous file has been included, and because this does not exceed the the max-file limit of 5
-* 12 -> Yes, because the previous file had been included, and because this does not exceed the the max-file limit of 5.
+* 12 -> Yes, because the previous file has been included, and because this does not exceed the max-file limit of 5
+* 12 -> Yes, because the previous file had been included, and because this does not exceed the max-file limit of 5.
[[compaction.file.selection.example2]]
====== Minor Compaction File Selection - Example #2 (Not Enough Files ToCompact)
@@ -2178,7 +2245,7 @@ See link:http://blog.cloudera.com/blog/2013/09/how-to-use-hbase-bulk-loading-and
[[arch.bulk.load.adv]]
=== Advanced Usage
-Although the `importtsv` tool is useful in many cases, advanced users may want to generate data programatically, or import data from other formats.
+Although the `importtsv` tool is useful in many cases, advanced users may want to generate data programmatically, or import data from other formats.
To get started doing so, dig into `ImportTsv.java` and check the JavaDoc for HFileOutputFormat.
The import step of the bulk load can also be done programmatically.
@@ -2274,8 +2341,8 @@ In terms of semantics, TIMELINE consistency as implemented by HBase differs from
.Timeline Consistency
image::timeline_consistency.png[Timeline Consistency]
-To better understand the TIMELINE semantics, lets look at the above diagram.
-Lets say that there are two clients, and the first one writes x=1 at first, then x=2 and x=3 later.
+To better understand the TIMELINE semantics, let's look at the above diagram.
+Let's say that there are two clients, and the first one writes x=1 at first, then x=2 and x=3 later.
As above, all writes are handled by the primary region replica.
The writes are saved in the write ahead log (WAL), and replicated to the other replicas asynchronously.
In the above diagram, notice that replica_id=1 received 2 updates, and its data shows that x=2, while the replica_id=2 only received a single update, and its data shows that x=1.
@@ -2308,18 +2375,18 @@ To serve the region data from multiple replicas, HBase opens the regions in seco
The regions opened in secondary mode will share the same data files with the primary region replica, however each secondary region replica will have its own MemStore to keep the unflushed data (only primary region can do flushes). Also to serve reads from secondary regions, the blocks of data files may be also cached in the block caches for the secondary regions.
=== Where is the code
-This feature is delivered in two phases, Phase 1 and 2. The first phase is done in time for HBase-1.0.0 release. Meaning that using HBase-1.0.x, you can use all the features that are marked for Phase 1. Phase 2 is committed in HBase-1.1.0, meaning all HBase versions after 1.1.0 should contain Phase 2 items.
+This feature is delivered in two phases, Phase 1 and 2. The first phase is done in time for HBase-1.0.0 release. Meaning that using HBase-1.0.x, you can use all the features that are marked for Phase 1. Phase 2 is committed in HBase-1.1.0, meaning all HBase versions after 1.1.0 should contain Phase 2 items.
=== Propagating writes to region replicas
-As discussed above writes only go to the primary region replica. For propagating the writes from the primary region replica to the secondaries, there are two different mechanisms. For read-only tables, you do not need to use any of the following methods. Disabling and enabling the table should make the data available in all region replicas. For mutable tables, you have to use *only* one of the following mechanisms: storefile refresher, or async wal replication. The latter is recommeded.
+As discussed above writes only go to the primary region replica. For propagating the writes from the primary region replica to the secondaries, there are two different mechanisms. For read-only tables, you do not need to use any of the following methods. Disabling and enabling the table should make the data available in all region replicas. For mutable tables, you have to use *only* one of the following mechanisms: storefile refresher, or async wal replication. The latter is recommended.
==== StoreFile Refresher
-The first mechanism is store file refresher which is introduced in HBase-1.0+. Store file refresher is a thread per region server, which runs periodically, and does a refresh operation for the store files of the primary region for the secondary region replicas. If enabled, the refresher will ensure that the secondary region replicas see the new flushed, compacted or bulk loaded files from the primary region in a timely manner. However, this means that only flushed data can be read back from the secondary region replicas, and after the refresher is run, making the secondaries lag behind the primary for an a longer time.
+The first mechanism is store file refresher which is introduced in HBase-1.0+. Store file refresher is a thread per region server, which runs periodically, and does a refresh operation for the store files of the primary region for the secondary region replicas. If enabled, the refresher will ensure that the secondary region replicas see the new flushed, compacted or bulk loaded files from the primary region in a timely manner. However, this means that only flushed data can be read back from the secondary region replicas, and after the refresher is run, making the secondaries lag behind the primary for an a longer time.
-For turning this feature on, you should configure `hbase.regionserver.storefile.refresh.period` to a non-zero value. See Configuration section below.
+For turning this feature on, you should configure `hbase.regionserver.storefile.refresh.period` to a non-zero value. See Configuration section below.
==== Asnyc WAL replication
-The second mechanism for propagation of writes to secondaries is done via “Async WAL Replication” feature and is only available in HBase-1.1+. This works similarly to HBase’s multi-datacenter replication, but instead the data from a region is replicated to the secondary regions. Each secondary replica always receives and observes the writes in the same order that the primary region committed them. In some sense, this design can be thought of as “in-cluster replication”, where instead of replicating to a different datacenter, the data goes to secondary regions to keep secondary region’s in-memory state up to date. The data files are shared between the primary region and the other replicas, so that there is no extra storage overhead. However, the secondary regions will have recent non-flushed data in their memstores, which increases the memory overhead. The primary region writes flush, compaction, and bulk load events to its WAL as well, which are also replicated through w
al replication to secondaries. When they observe the flush/compaction or bulk load event, the secondary regions replay the event to pick up the new files and drop the old ones.
+The second mechanism for propagation of writes to secondaries is done via “Async WAL Replication” feature and is only available in HBase-1.1+. This works similarly to HBase’s multi-datacenter replication, but instead the data from a region is replicated to the secondary regions. Each secondary replica always receives and observes the writes in the same order that the primary region committed them. In some sense, this design can be thought of as “in-cluster replication”, where instead of replicating to a different datacenter, the data goes to secondary regions to keep secondary region’s in-memory state up to date. The data files are shared between the primary region and the other replicas, so that there is no extra storage overhead. However, the secondary regions will have recent non-flushed data in their memstores, which increases the memory overhead. The primary region writes flush, compaction, and bulk load events to its WAL as well, which are also replicated through w
al replication to secondaries. When they observe the flush/compaction or bulk load event, the secondary regions replay the event to pick up the new files and drop the old ones.
Committing writes in the same order as in primary ensures that the secondaries won’t diverge from the primary regions data, but since the log replication is asynchronous, the data might still be stale in secondary regions. Since this feature works as a replication endpoint, the performance and latency characteristics is expected to be similar to inter-cluster replication.
@@ -2332,18 +2399,18 @@ Asyn WAL Replication feature will add a new replication peer named `region_repli
hbase> disable_peer 'region_replica_replication'
----
-=== Store File TTL
-In both of the write propagation approaches mentioned above, store files of the primary will be opened in secondaries independent of the primary region. So for files that the primary compacted away, the secondaries might still be referring to these files for reading. Both features are using HFileLinks to refer to files, but there is no protection (yet) for guaranteeing that the file will not be deleted prematurely. Thus, as a guard, you should set the configuration property `hbase.master.hfilecleaner.ttl` to a larger value, such as 1 hour to guarantee that you will not receive IOExceptions for requests going to replicas.
+=== Store File TTL
+In both of the write propagation approaches mentioned above, store files of the primary will be opened in secondaries independent of the primary region. So for files that the primary compacted away, the secondaries might still be referring to these files for reading. Both features are using HFileLinks to refer to files, but there is no protection (yet) for guaranteeing that the file will not be deleted prematurely. Thus, as a guard, you should set the configuration property `hbase.master.hfilecleaner.ttl` to a larger value, such as 1 hour to guarantee that you will not receive IOExceptions for requests going to replicas.
=== Region replication for META table’s region
-Currently, Async WAL Replication is not done for the META table’s WAL. The meta table’s secondary replicas still refreshes themselves from the persistent store files. Hence the `hbase.regionserver.meta.storefile.refresh.period` needs to be set to a certain non-zero value for refreshing the meta store files. Note that this configuration is configured differently than
-`hbase.regionserver.storefile.refresh.period`.
+Currently, Async WAL Replication is not done for the META table’s WAL. The meta table’s secondary replicas still refreshes themselves from the persistent store files. Hence the `hbase.regionserver.meta.storefile.refresh.period` needs to be set to a certain non-zero value for refreshing the meta store files. Note that this configuration is configured differently than
+`hbase.regionserver.storefile.refresh.period`.
=== Memory accounting
The secondary region replicas refer to the data files of the primary region replica, but they have their own memstores (in HBase-1.1+) and uses block cache as well. However, one distinction is that the secondary region replicas cannot flush the data when there is memory pressure for their memstores. They can only free up memstore memory when the primary region does a flush and this flush is replicated to the secondary. Since in a region server hosting primary replicas for some regions and secondaries for some others, the secondaries might cause extra flushes to the primary regions in the same host. In extreme situations, there can be no memory left for adding new writes coming from the primary via wal replication. For unblocking this situation (and since secondary cannot flush by itself), the secondary is allowed to do a “store file refresh” by doing a file system list operation to pick up new files from primary, and possibly dropping its memstore. This refresh will only be perf
ormed if the memstore size of the biggest secondary region replica is at least `hbase.region.replica.storefile.refresh.memstore.multiplier` (default 4) times bigger than the biggest memstore of a primary replica. One caveat is that if this is performed, the secondary can observe partial row updates across column families (since column families are flushed independently). The default should be good to not do this operation frequently. You can set this value to a large number to disable this feature if desired, but be warned that it might cause the replication to block forever.
=== Secondary replica failover
-When a secondary region replica first comes online, or fails over, it may have served some edits from it’s memstore. Since the recovery is handled differently for secondary replicas, the secondary has to ensure that it does not go back in time before it starts serving requests after assignment. For doing that, the secondary waits until it observes a full flush cycle (start flush, commit flush) or a “region open event” replicated from the primary. Until this happens, the secondary region replica will reject all read requests by throwing an IOException with message “The region's reads are disabled”. However, the other replicas will probably still be available to read, thus not causing any impact for the rpc with TIMELINE consistency. To facilitate faster recovery, the secondary region will trigger a flush request from the primary when it is opened. The configuration property `hbase.region.replica.wait.for.primary.flush` (enabled by default) can be used to disable this featur
e if needed.
+When a secondary region replica first comes online, or fails over, it may have served some edits from its memstore. Since the recovery is handled differently for secondary replicas, the secondary has to ensure that it does not go back in time before it starts serving requests after assignment. For doing that, the secondary waits until it observes a full flush cycle (start flush, commit flush) or a “region open event” replicated from the primary. Until this happens, the secondary region replica will reject all read requests by throwing an IOException with message “The region's reads are disabled”. However, the other replicas will probably still be available to read, thus not causing any impact for the rpc with TIMELINE consistency. To facilitate faster recovery, the secondary region will trigger a flush request from the primary when it is opened. The configuration property `hbase.region.replica.wait.for.primary.flush` (enabled by default) can be used to disable this feature i
f needed.
@@ -2352,7 +2419,7 @@ When a secondary region replica first comes online, or fails over, it may have s
To use highly available reads, you should set the following properties in `hbase-site.xml` file.
There is no specific configuration to enable or disable region replicas.
-Instead you can change the number of region replicas per table to increase or decrease at the table creation or with alter table. The following configuration is for using async wal replication and using meta replicas of 3.
+Instead you can change the number of region replicas per table to increase or decrease at the table creation or with alter table. The following configuration is for using async wal replication and using meta replicas of 3.
==== Server side properties
@@ -2379,7 +2446,7 @@ Instead you can change the number of region replicas per table to increase or de
<name>hbase.region.replica.replication.enabled</name>
<value>true</value>
<description>
- Whether asynchronous WAL replication to the secondary region replicas is enabled or not. If this is enabled, a replication peer named "region_replica_replication" will be created which will tail the logs and replicate the mutatations to region replicas for tables that have region replication > 1. If this is enabled once, disabling this replication also requires disabling the replication peer using shell or ReplicationAdmin java class. Replication to secondary region replicas works over standard inter-cluster replication. So replication, if disabled explicitly, also has to be enabled by setting "hbase.replication"· to true for this feature to work.
+ Whether asynchronous WAL replication to the secondary region replicas is enabled or not. If this is enabled, a replication peer named "region_replica_replication" will be created which will tail the logs and replicate the mutations to region replicas for tables that have region replication > 1. If this is enabled once, disabling this replication also requires disabling the replication peer using shell or ReplicationAdmin java class. Replication to secondary region replicas works over standard inter-cluster replication. So replication, if disabled explicitly, also has to be enabled by setting "hbase.replication"· to true for this feature to work.
</description>
</property>
<property>
@@ -2413,7 +2480,7 @@ Instead you can change the number of region replicas per table to increase or de
</property>
-<property>
+<property>
<name>hbase.region.replica.storefile.refresh.memstore.multiplier</name>
<value>4</value>
<description>
@@ -2476,7 +2543,7 @@ Ensure to set the following for all clients (and servers) that will use region r
</property>
----
-Note HBase-1.0.x users should use `hbase.ipc.client.allowsInterrupt` rather than `hbase.ipc.client.specificThreadForWriting`.
+Note HBase-1.0.x users should use `hbase.ipc.client.allowsInterrupt` rather than `hbase.ipc.client.specificThreadForWriting`.
=== User Interface
@@ -2547,7 +2614,7 @@ hbase> scan 't1', {CONSISTENCY => 'TIMELINE'}
==== Java
-You can set set the consistency for Gets and Scans and do requests as follows.
+You can set the consistency for Gets and Scans and do requests as follows.
[source,java]
----
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/asf.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/asf.adoc b/src/main/asciidoc/_chapters/asf.adoc
index 77eed8f..47c29e5 100644
--- a/src/main/asciidoc/_chapters/asf.adoc
+++ b/src/main/asciidoc/_chapters/asf.adoc
@@ -35,13 +35,13 @@ HBase is a project in the Apache Software Foundation and as such there are respo
[[asf.devprocess]]
=== ASF Development Process
-See the link:http://www.apache.org/dev/#committers[Apache Development Process page] for all sorts of information on how the ASF is structured (e.g., PMC, committers, contributors), to tips on contributing and getting involved, and how open-source works at ASF.
+See the link:http://www.apache.org/dev/#committers[Apache Development Process page] for all sorts of information on how the ASF is structured (e.g., PMC, committers, contributors), to tips on contributing and getting involved, and how open-source works at ASF.
[[asf.reporting]]
=== ASF Board Reporting
Once a quarter, each project in the ASF portfolio submits a report to the ASF board.
This is done by the HBase project lead and the committers.
-See link:http://www.apache.org/foundation/board/reporting[ASF board reporting] for more information.
+See link:http://www.apache.org/foundation/board/reporting[ASF board reporting] for more information.
:numbered:
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/case_studies.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/case_studies.adoc b/src/main/asciidoc/_chapters/case_studies.adoc
index 992414c..b021aa2 100644
--- a/src/main/asciidoc/_chapters/case_studies.adoc
+++ b/src/main/asciidoc/_chapters/case_studies.adoc
@@ -55,7 +55,7 @@ These jobs were consistently found to be waiting on map and reduce tasks assigne
.Datanodes:
* Two 12-core processors
-* Six Enerprise SATA disks
+* Six Enterprise SATA disks
* 24GB of RAM
* Two bonded gigabit NICs
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/community.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/community.adoc b/src/main/asciidoc/_chapters/community.adoc
index 573fb49..ba07df7 100644
--- a/src/main/asciidoc/_chapters/community.adoc
+++ b/src/main/asciidoc/_chapters/community.adoc
@@ -45,18 +45,18 @@ See link:http://search-hadoop.com/m/asM982C5FkS1[HBase, mail # dev - Thoughts
The below policy is something we put in place 09/2012.
It is a suggested policy rather than a hard requirement.
-We want to try it first to see if it works before we cast it in stone.
+We want to try it first to see if it works before we cast it in stone.
Apache HBase is made of link:https://issues.apache.org/jira/browse/HBASE#selectedTab=com.atlassian.jira.plugin.system.project%3Acomponents-panel[components].
Components have one or more <<owner,OWNER>>s.
-See the 'Description' field on the link:https://issues.apache.org/jira/browse/HBASE#selectedTab=com.atlassian.jira.plugin.system.project%3Acomponents-panel[components] JIRA page for who the current owners are by component.
+See the 'Description' field on the link:https://issues.apache.org/jira/browse/HBASE#selectedTab=com.atlassian.jira.plugin.system.project%3Acomponents-panel[components] JIRA page for who the current owners are by component.
Patches that fit within the scope of a single Apache HBase component require, at least, a +1 by one of the component's owners before commit.
-If owners are absent -- busy or otherwise -- two +1s by non-owners will suffice.
+If owners are absent -- busy or otherwise -- two +1s by non-owners will suffice.
-Patches that span components need at least two +1s before they can be committed, preferably +1s by owners of components touched by the x-component patch (TODO: This needs tightening up but I think fine for first pass).
+Patches that span components need at least two +1s before they can be committed, preferably +1s by owners of components touched by the x-component patch (TODO: This needs tightening up but I think fine for first pass).
-Any -1 on a patch by anyone vetos a patch; it cannot be committed until the justification for the -1 is addressed.
+Any -1 on a patch by anyone vetoes a patch; it cannot be committed until the justification for the -1 is addressed.
[[hbase.fix.version.in.jira]]
.How to set fix version in JIRA on issue resolve
@@ -67,13 +67,13 @@ If master is going to be 0.98.0 then:
* Commit only to master: Mark with 0.98
* Commit to 0.95 and master: Mark with 0.98, and 0.95.x
* Commit to 0.94.x and 0.95, and master: Mark with 0.98, 0.95.x, and 0.94.x
-* Commit to 89-fb: Mark with 89-fb.
-* Commit site fixes: no version
+* Commit to 89-fb: Mark with 89-fb.
+* Commit site fixes: no version
[[hbase.when.to.close.jira]]
.Policy on when to set a RESOLVED JIRA as CLOSED
-We link:http://search-hadoop.com/m/4cIKs1iwXMS1[agreed] that for issues that list multiple releases in their _Fix Version/s_ field, CLOSE the issue on the release of any of the versions listed; subsequent change to the issue must happen in a new JIRA.
+We link:http://search-hadoop.com/m/4cIKs1iwXMS1[agreed] that for issues that list multiple releases in their _Fix Version/s_ field, CLOSE the issue on the release of any of the versions listed; subsequent change to the issue must happen in a new JIRA.
[[no.permanent.state.in.zk]]
.Only transient state in ZooKeeper!
@@ -81,7 +81,7 @@ We link:http://search-hadoop.com/m/4cIKs1iwXMS1[agreed] that for issues that lis
You should be able to kill the data in zookeeper and hbase should ride over it recreating the zk content as it goes.
This is an old adage around these parts.
We just made note of it now.
-We also are currently in violation of this basic tenet -- replication at least keeps permanent state in zk -- but we are working to undo this breaking of a golden rule.
+We also are currently in violation of this basic tenet -- replication at least keeps permanent state in zk -- but we are working to undo this breaking of a golden rule.
[[community.roles]]
== Community Roles
@@ -90,22 +90,22 @@ We also are currently in violation of this basic tenet -- replication at least k
.Component Owner/Lieutenant
Component owners are listed in the description field on this Apache HBase JIRA link:https://issues.apache.org/jira/browse/HBASE#selectedTab=com.atlassian.jira.plugin.system.project%3Acomponents-panel[components] page.
-The owners are listed in the 'Description' field rather than in the 'Component Lead' field because the latter only allows us list one individual whereas it is encouraged that components have multiple owners.
+The owners are listed in the 'Description' field rather than in the 'Component Lead' field because the latter only allows us list one individual whereas it is encouraged that components have multiple owners.
-Owners or component lieutenants are volunteers who are (usually, but not necessarily) expert in their component domain and may have an agenda on how they think their Apache HBase component should evolve.
+Owners or component lieutenants are volunteers who are (usually, but not necessarily) expert in their component domain and may have an agenda on how they think their Apache HBase component should evolve.
-. Owners will try and review patches that land within their component's scope.
-. If applicable, if an owner has an agenda, they will publish their goals or the design toward which they are driving their component
+. Owners will try and review patches that land within their component's scope.
+. If applicable, if an owner has an agenda, they will publish their goals or the design toward which they are driving their component
If you would like to be volunteer as a component owner, just write the dev list and we'll sign you up.
-Owners do not need to be committers.
+Owners do not need to be committers.
[[hbase.commit.msg.format]]
== Commit Message format
-We link:http://search-hadoop.com/m/Gwxwl10cFHa1[agreed] to the following Git commit message format:
+We link:http://search-hadoop.com/m/Gwxwl10cFHa1[agreed] to the following Git commit message format:
[source]
----
HBASE-xxxxx <title>. (<contributor>)
-----
-If the person making the commit is the contributor, leave off the '(<contributor>)' element.
+----
+If the person making the commit is the contributor, leave off the '(<contributor>)' element.
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/compression.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/compression.adoc b/src/main/asciidoc/_chapters/compression.adoc
index 42d4de5..462bce3 100644
--- a/src/main/asciidoc/_chapters/compression.adoc
+++ b/src/main/asciidoc/_chapters/compression.adoc
@@ -144,15 +144,15 @@ In general, you need to weigh your options between smaller size and faster compr
The Hadoop shared library has a bunch of facility including compression libraries and fast crc'ing. To make this facility available to HBase, do the following. HBase/Hadoop will fall back to use alternatives if it cannot find the native library versions -- or fail outright if you asking for an explicit compressor and there is no alternative available.
-If you see the following in your HBase logs, you know that HBase was unable to locate the Hadoop native libraries:
+If you see the following in your HBase logs, you know that HBase was unable to locate the Hadoop native libraries:
[source]
----
2014-08-07 09:26:20,139 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-----
-If the libraries loaded successfully, the WARN message does not show.
+----
+If the libraries loaded successfully, the WARN message does not show.
-Lets presume your Hadoop shipped with a native library that suits the platform you are running HBase on.
-To check if the Hadoop native library is available to HBase, run the following tool (available in Hadoop 2.1 and greater):
+Let's presume your Hadoop shipped with a native library that suits the platform you are running HBase on.
+To check if the Hadoop native library is available to HBase, run the following tool (available in Hadoop 2.1 and greater):
[source]
----
$ ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker
@@ -165,28 +165,28 @@ lz4: false
bzip2: false
2014-08-26 13:15:38,863 INFO [main] util.ExitUtil: Exiting with status 1
----
-Above shows that the native hadoop library is not available in HBase context.
+Above shows that the native hadoop library is not available in HBase context.
To fix the above, either copy the Hadoop native libraries local or symlink to them if the Hadoop and HBase stalls are adjacent in the filesystem.
You could also point at their location by setting the `LD_LIBRARY_PATH` environment variable.
-Where the JVM looks to find native librarys is "system dependent" (See `java.lang.System#loadLibrary(name)`). On linux, by default, is going to look in _lib/native/PLATFORM_ where `PLATFORM` is the label for the platform your HBase is installed on.
+Where the JVM looks to find native libraries is "system dependent" (See `java.lang.System#loadLibrary(name)`). On linux, by default, is going to look in _lib/native/PLATFORM_ where `PLATFORM` is the label for the platform your HBase is installed on.
On a local linux machine, it seems to be the concatenation of the java properties `os.name` and `os.arch` followed by whether 32 or 64 bit.
HBase on startup prints out all of the java system properties so find the os.name and os.arch in the log.
-For example:
+For example:
[source]
----
...
2014-08-06 15:27:22,853 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2014-08-06 15:27:22,853 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
...
-----
+----
So in this case, the PLATFORM string is `Linux-amd64-64`.
Copying the Hadoop native libraries or symlinking at _lib/native/Linux-amd64-64_ will ensure they are found.
Check with the Hadoop _NativeLibraryChecker_.
-
-Here is example of how to point at the Hadoop libs with `LD_LIBRARY_PATH` environment variable:
+
+Here is example of how to point at the Hadoop libs with `LD_LIBRARY_PATH` environment variable:
[source]
----
$ LD_LIBRARY_PATH=~/hadoop-2.5.0-SNAPSHOT/lib/native ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker
@@ -199,7 +199,7 @@ snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
----
-Set in _hbase-env.sh_ the LD_LIBRARY_PATH environment variable when starting your HBase.
+Set in _hbase-env.sh_ the LD_LIBRARY_PATH environment variable when starting your HBase.
=== Compressor Configuration, Installation, and Use
@@ -210,13 +210,13 @@ Before HBase can use a given compressor, its libraries need to be available.
Due to licensing issues, only GZ compression is available to HBase (via native Java libraries) in a default installation.
Other compression libraries are available via the shared library bundled with your hadoop.
The hadoop native library needs to be findable when HBase starts.
-See
+See
.Compressor Support On the Master
A new configuration setting was introduced in HBase 0.95, to check the Master to determine which data block encoders are installed and configured on it, and assume that the entire cluster is configured the same.
This option, `hbase.master.check.compression`, defaults to `true`.
-This prevents the situation described in link:https://issues.apache.org/jira/browse/HBASE-6370[HBASE-6370], where a table is created or modified to support a codec that a region server does not support, leading to failures that take a long time to occur and are difficult to debug.
+This prevents the situation described in link:https://issues.apache.org/jira/browse/HBASE-6370[HBASE-6370], where a table is created or modified to support a codec that a region server does not support, leading to failures that take a long time to occur and are difficult to debug.
If `hbase.master.check.compression` is enabled, libraries for all desired compressors need to be installed and configured on the Master, even if the Master does not run a region server.
@@ -232,7 +232,7 @@ See <<brand.new.compressor,brand.new.compressor>>).
HBase cannot ship with LZO because of incompatibility between HBase, which uses an Apache Software License (ASL) and LZO, which uses a GPL license.
See the link:http://wiki.apache.org/hadoop/UsingLzoCompression[Using LZO
- Compression] wiki page for information on configuring LZO support for HBase.
+ Compression] wiki page for information on configuring LZO support for HBase.
If you depend upon LZO compression, consider configuring your RegionServers to fail to start if LZO is not available.
See <<hbase.regionserver.codecs,hbase.regionserver.codecs>>.
@@ -244,19 +244,19 @@ LZ4 support is bundled with Hadoop.
Make sure the hadoop shared library (libhadoop.so) is accessible when you start HBase.
After configuring your platform (see <<hbase.native.platform,hbase.native.platform>>), you can make a symbolic link from HBase to the native Hadoop libraries.
This assumes the two software installs are colocated.
-For example, if my 'platform' is Linux-amd64-64:
+For example, if my 'platform' is Linux-amd64-64:
[source,bourne]
----
$ cd $HBASE_HOME
$ mkdir lib/native
$ ln -s $HADOOP_HOME/lib/native lib/native/Linux-amd64-64
-----
+----
Use the compression tool to check that LZ4 is installed on all nodes.
Start up (or restart) HBase.
-Afterward, you can create and alter tables to enable LZ4 as a compression codec.:
+Afterward, you can create and alter tables to enable LZ4 as a compression codec.:
----
hbase(main):003:0> alter 'TestTable', {NAME => 'info', COMPRESSION => 'LZ4'}
-----
+----
[[snappy.compression.installation]]
.Install Snappy Support
@@ -347,7 +347,7 @@ You must specify either `-write` or `-update-read` as your first parameter, and
====
----
-$ bin/hbase org.apache.hadoop.hbase.util.LoadTestTool -h
+$ bin/hbase org.apache.hadoop.hbase.util.LoadTestTool -h
usage: bin/hbase org.apache.hadoop.hbase.util.LoadTestTool <options>
Options:
-batchupdate Whether to use batch as opposed to separate
http://git-wip-us.apache.org/repos/asf/hbase/blob/c07ddc6d/src/main/asciidoc/_chapters/configuration.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/configuration.adoc b/src/main/asciidoc/_chapters/configuration.adoc
index 5a4a6ec..495232f 100644
--- a/src/main/asciidoc/_chapters/configuration.adoc
+++ b/src/main/asciidoc/_chapters/configuration.adoc
@@ -162,7 +162,7 @@ For example, assuming that a schema had 3 ColumnFamilies per region with an aver
+
Another related setting is the number of processes a user is allowed to run at once. In Linux and Unix, the number of processes is set using the `ulimit -u` command. This should not be confused with the `nproc` command, which controls the number of CPUs available to a given user. Under load, a `ulimit -u` that is too low can cause OutOfMemoryError exceptions. See Jack Levin's major HDFS issues thread on the hbase-users mailing list, from 2011.
+
-Configuring the maximum number of file descriptors and processes for the user who is running the HBase process is an operating system configuration, rather than an HBase configuration. It is also important to be sure that the settings are changed for the user that actually runs HBase. To see which user started HBase, and that user's ulimit configuration, look at the first line of the HBase log for that instance. A useful read setting config on you hadoop cluster is Aaron Kimballs' Configuration Parameters: What can you just ignore?
+Configuring the maximum number of file descriptors and processes for the user who is running the HBase process is an operating system configuration, rather than an HBase configuration. It is also important to be sure that the settings are changed for the user that actually runs HBase. To see which user started HBase, and that user's ulimit configuration, look at the first line of the HBase log for that instance. A useful read setting config on your hadoop cluster is Aaron Kimball's Configuration Parameters: What can you just ignore?
+
.`ulimit` Settings on Ubuntu
====
@@ -222,7 +222,8 @@ Use the following legend to interpret this table:
|Hadoop-2.3.x | NT | S | NT | NT | NT
|Hadoop-2.4.x | NT | S | S | S | S
|Hadoop-2.5.x | NT | S | S | S | S
-|Hadoop-2.6.x | X | X | X | X | X
+|Hadoop-2.6.0 | X | X | X | X | X
+|Hadoop-2.6.1+ | NT | NT | NT | NT | S
|Hadoop-2.7.0 | X | X | X | X | X
|Hadoop-2.7.1+ | NT | NT | NT | NT | S
|===
@@ -233,7 +234,7 @@ Use the following legend to interpret this table:
Hadoop distributions based on the 2.6.x line *must* have
link:https://issues.apache.org/jira/browse/HADOOP-11710[HADOOP-11710] applied if you plan to run
HBase on top of an HDFS Encryption Zone. Failure to do so will result in cluster failure and
-data loss.
+data loss. This patch is present in Apache Hadoop releases 2.6.1+.
====
.Hadoop 2.7.x
@@ -410,7 +411,7 @@ Zookeeper binds to a well known port so clients may talk to HBase.
=== Distributed
-Distributed mode can be subdivided into distributed but all daemons run on a single node -- a.k.a _pseudo-distributed_ -- and _fully-distributed_ where the daemons are spread across all nodes in the cluster.
+Distributed mode can be subdivided into distributed but all daemons run on a single node -- a.k.a. _pseudo-distributed_ -- and _fully-distributed_ where the daemons are spread across all nodes in the cluster.
The _pseudo-distributed_ vs. _fully-distributed_ nomenclature comes from Hadoop.
Pseudo-distributed mode can run against the local filesystem or it can run against an instance of the _Hadoop Distributed File System_ (HDFS). Fully-distributed mode can ONLY run on HDFS.
@@ -540,7 +541,7 @@ HBase logs can be found in the _logs_ subdirectory.
Check them out especially if HBase had trouble starting.
HBase also puts up a UI listing vital attributes.
-By default it's deployed on the Master host at port 16010 (HBase RegionServers listen on port 16020 by default and put up an informational HTTP server at port 16030). If the Master is running on a host named `master.example.org` on the default port, point your browser at _http://master.example.org:16010_ to see the web interface.
+By default it's deployed on the Master host at port 16010 (HBase RegionServers listen on port 16020 by default and put up an informational HTTP server at port 16030). If the Master is running on a host named `master.example.org` on the default port, point your browser at pass:[http://master.example.org:16010] to see the web interface.
Prior to HBase 0.98 the master UI was deployed on port 60010, and the HBase RegionServers UI on port 60030.
@@ -564,7 +565,7 @@ If you are running a distributed operation, be sure to wait until HBase has shut
=== _hbase-site.xml_ and _hbase-default.xml_
Just as in Hadoop where you add site-specific HDFS configuration to the _hdfs-site.xml_ file, for HBase, site specific customizations go into the file _conf/hbase-site.xml_.
-For the list of configurable properties, see <<hbase_default_configurations,hbase default configurations>> below or view the raw _hbase-default.xml_ source file in the HBase source code at _src/main/resources_.
+For the list of configurable properties, see <<hbase_default_configurations,hbase default configurations>> below or view the raw _hbase-default.xml_ source file in the HBase source code at _src/main/resources_.
Not all configuration options make it out to _hbase-default.xml_.
Configuration that it is thought rare anyone would change can exist only in code; the only way to turn up such configurations is via a reading of the source code itself.
@@ -572,7 +573,7 @@ Configuration that it is thought rare anyone would change can exist only in code
Currently, changes here will require a cluster restart for HBase to notice the change.
// hbase/src/main/asciidoc
//
-include::../../../../target/asciidoc/hbase-default.adoc[]
+include::{docdir}/../../../target/asciidoc/hbase-default.adoc[]
[[hbase.env.sh]]
@@ -604,7 +605,7 @@ ZooKeeper is where all these values are kept.
Thus clients require the location of the ZooKeeper ensemble before they can do anything else.
Usually this the ensemble location is kept out in the _hbase-site.xml_ and is picked up by the client from the `CLASSPATH`.
-If you are configuring an IDE to run a HBase client, you should include the _conf/_ directory on your classpath so _hbase-site.xml_ settings can be found (or add _src/test/resources_ to pick up the hbase-site.xml used by tests).
+If you are configuring an IDE to run an HBase client, you should include the _conf/_ directory on your classpath so _hbase-site.xml_ settings can be found (or add _src/test/resources_ to pick up the hbase-site.xml used by tests).
Minimally, a client of HBase needs several libraries in its `CLASSPATH` when connecting to a cluster, including:
[source]
@@ -621,7 +622,7 @@ slf4j-log4j (slf4j-log4j12-1.5.8.jar)
zookeeper (zookeeper-3.4.2.jar)
----
-An example basic _hbase-site.xml_ for client only might look as follows:
+An example basic _hbase-site.xml_ for client only might look as follows:
[source,xml]
----
<?xml version="1.0"?>
@@ -917,7 +918,7 @@ See <<master.processes.loadbalancer,master.processes.loadbalancer>> for more inf
==== Disabling Blockcache
Do not turn off block cache (You'd do it by setting `hbase.block.cache.size` to zero). Currently we do not do well if you do this because the RegionServer will spend all its time loading HFile indices over and over again.
-If your working set it such that block cache does you no good, at least size the block cache such that HFile indices will stay up in the cache (you can get a rough idea on the size you need by surveying RegionServer UIs; you'll see index block size accounted near the top of the webpage).
+If your working set is such that block cache does you no good, at least size the block cache such that HFile indices will stay up in the cache (you can get a rough idea on the size you need by surveying RegionServer UIs; you'll see index block size accounted near the top of the webpage).
[[nagles]]
==== link:http://en.wikipedia.org/wiki/Nagle's_algorithm[Nagle's] or the small package problem
@@ -930,7 +931,7 @@ You might also see the graphs on the tail of link:https://issues.apache.org/jira
==== Better Mean Time to Recover (MTTR)
This section is about configurations that will make servers come back faster after a fail.
-See the Deveraj Das an Nicolas Liochon blog post link:http://hortonworks.com/blog/introduction-to-hbase-mean-time-to-recover-mttr/[Introduction to HBase Mean Time to Recover (MTTR)] for a brief introduction.
+See the Deveraj Das and Nicolas Liochon blog post link:http://hortonworks.com/blog/introduction-to-hbase-mean-time-to-recover-mttr/[Introduction to HBase Mean Time to Recover (MTTR)] for a brief introduction.
The issue link:https://issues.apache.org/jira/browse/HBASE-8389[HBASE-8354 forces Namenode into loop with lease recovery requests] is messy but has a bunch of good discussion toward the end on low timeouts and how to effect faster recovery including citation of fixes added to HDFS. Read the Varun Sharma comments.
The below suggested configurations are Varun's suggestions distilled and tested.
@@ -1002,7 +1003,7 @@ See the link:http://docs.oracle.com/javase/6/docs/technotes/guides/management/ag
Historically, besides above port mentioned, JMX opens two additional random TCP listening ports, which could lead to port conflict problem. (See link:https://issues.apache.org/jira/browse/HBASE-10289[HBASE-10289] for details)
As an alternative, You can use the coprocessor-based JMX implementation provided by HBase.
-To enable it in 0.99 or above, add below property in _hbase-site.xml_:
+To enable it in 0.99 or above, add below property in _hbase-site.xml_:
[source,xml]
----
@@ -1033,7 +1034,7 @@ The registry port can be shared with connector port in most cases, so you only n
However if you want to use SSL communication, the 2 ports must be configured to different values.
By default the password authentication and SSL communication is disabled.
-To enable password authentication, you need to update _hbase-env.sh_ like below:
+To enable password authentication, you need to update _hbase-env.sh_ like below:
[source,bash]
----
export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.authenticate=true \
@@ -1060,7 +1061,7 @@ keytool -export -alias jconsole -keystore myKeyStore -file jconsole.cert
keytool -import -alias jconsole -keystore jconsoleKeyStore -file jconsole.cert
----
-And then update _hbase-env.sh_ like below:
+And then update _hbase-env.sh_ like below:
[source,bash]
----
@@ -1082,12 +1083,12 @@ Finally start `jconsole` on the client using the key store:
jconsole -J-Djavax.net.ssl.trustStore=/home/tianq/jconsoleKeyStore
----
-NOTE: To enable the HBase JMX implementation on Master, you also need to add below property in _hbase-site.xml_:
+NOTE: To enable the HBase JMX implementation on Master, you also need to add below property in _hbase-site.xml_:
[source,xml]
----
<property>
- <ame>hbase.coprocessor.master.classes</name>
+ <name>hbase.coprocessor.master.classes</name>
<value>org.apache.hadoop.hbase.JMXListener</value>
</property>
----