You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by an...@apache.org on 2016/10/26 20:07:37 UTC
[8/8] hbase git commit: HBASE-15347 updated asciidoc for 1.3
HBASE-15347 updated asciidoc for 1.3
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/6cb8a436
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/6cb8a436
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/6cb8a436
Branch: refs/heads/branch-1.3
Commit: 6cb8a436c3584445f8e87e0251b8174be7d9dc21
Parents: 505d48a
Author: Mikhail Antonov <an...@apache.org>
Authored: Wed Oct 26 13:07:08 2016 -0700
Committer: Mikhail Antonov <an...@apache.org>
Committed: Wed Oct 26 13:07:08 2016 -0700
----------------------------------------------------------------------
.../asciidoc/_chapters/appendix_acl_matrix.adoc | 4 +-
.../appendix_contributing_to_documentation.adoc | 268 +++---
.../_chapters/appendix_hfile_format.adoc | 176 ++--
src/main/asciidoc/_chapters/architecture.adoc | 423 ++++++---
src/main/asciidoc/_chapters/asf.adoc | 4 +-
src/main/asciidoc/_chapters/case_studies.adoc | 2 +-
src/main/asciidoc/_chapters/community.adoc | 42 +-
src/main/asciidoc/_chapters/compression.adoc | 84 +-
src/main/asciidoc/_chapters/configuration.adoc | 146 +--
src/main/asciidoc/_chapters/cp.adoc | 887 +++++++++++++++---
src/main/asciidoc/_chapters/datamodel.adoc | 11 +-
src/main/asciidoc/_chapters/developer.adoc | 657 +++++++------
src/main/asciidoc/_chapters/external_apis.adoc | 920 ++++++++++++++++++-
src/main/asciidoc/_chapters/faq.adoc | 24 +-
.../asciidoc/_chapters/getting_started.adoc | 25 +-
src/main/asciidoc/_chapters/hbase-default.adoc | 602 +++++-------
src/main/asciidoc/_chapters/hbase_apis.adoc | 8 +-
src/main/asciidoc/_chapters/hbase_history.adoc | 8 +-
src/main/asciidoc/_chapters/hbase_mob.adoc | 236 +++++
src/main/asciidoc/_chapters/hbck_in_depth.adoc | 24 +-
src/main/asciidoc/_chapters/mapreduce.adoc | 57 +-
src/main/asciidoc/_chapters/ops_mgt.adoc | 361 +++++++-
src/main/asciidoc/_chapters/other_info.adoc | 34 +-
src/main/asciidoc/_chapters/performance.adoc | 88 +-
src/main/asciidoc/_chapters/preface.adoc | 54 +-
src/main/asciidoc/_chapters/protobuf.adoc | 153 +++
src/main/asciidoc/_chapters/rpc.adoc | 25 +-
src/main/asciidoc/_chapters/schema_design.adoc | 242 ++++-
src/main/asciidoc/_chapters/security.adoc | 149 ++-
src/main/asciidoc/_chapters/shell.adoc | 64 +-
src/main/asciidoc/_chapters/spark.adoc | 690 ++++++++++++++
.../_chapters/thrift_filter_language.adoc | 3 +-
src/main/asciidoc/_chapters/tracing.adoc | 65 +-
.../asciidoc/_chapters/troubleshooting.adoc | 76 +-
src/main/asciidoc/_chapters/unit_testing.adoc | 74 +-
src/main/asciidoc/_chapters/upgrading.adoc | 33 +-
src/main/asciidoc/_chapters/ycsb.adoc | 1 +
src/main/asciidoc/_chapters/zookeeper.adoc | 85 +-
src/main/asciidoc/book.adoc | 3 +
.../images/hbase_logo_with_orca_large.png | Bin 0 -> 21196 bytes
.../images/hbasecon2016-stack-logo.jpg | Bin 0 -> 32105 bytes
.../resources/images/hbasecon2016-stacked.png | Bin 0 -> 24924 bytes
42 files changed, 5227 insertions(+), 1581 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/6cb8a436/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc b/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
index cb285f3..e222875 100644
--- a/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
+++ b/src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
@@ -65,7 +65,7 @@ Possible permissions include the following:
For the most part, permissions work in an expected way, with the following caveats:
Having Write permission does not imply Read permission.::
- It is possible and sometimes desirable for a user to be able to write data that same user cannot read. One such example is a log-writing process.
+ It is possible and sometimes desirable for a user to be able to write data that same user cannot read. One such example is a log-writing process.
The [systemitem]+hbase:meta+ table is readable by every user, regardless of the user's other grants or restrictions.::
This is a requirement for HBase to function correctly.
`CheckAndPut` and `CheckAndDelete` operations will fail if the user does not have both Write and Read permission.::
@@ -100,7 +100,7 @@ In case the table goes out of date, the unit tests which check for accuracy of p
| | stopMaster | superuser\|global(A)
| | snapshot | superuser\|global(A)\|NS(A)\|TableOwner\|table(A)
| | listSnapshot | superuser\|global(A)\|SnapshotOwner
-| | cloneSnapshot | superuser\|global(A)
+| | cloneSnapshot | superuser\|global(A)\|(SnapshotOwner & TableName matches)
| | restoreSnapshot | superuser\|global(A)\|SnapshotOwner & (NS(A)\|TableOwner\|table(A))
| | deleteSnapshot | superuser\|global(A)\|SnapshotOwner
| | createNamespace | superuser\|global(A)
http://git-wip-us.apache.org/repos/asf/hbase/blob/6cb8a436/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc b/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
index 6b31059..ce6f835 100644
--- a/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
+++ b/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
@@ -30,15 +30,14 @@
:toc: left
:source-language: java
-The Apache HBase project welcomes contributions to all aspects of the project, including the documentation.
+The Apache HBase project welcomes contributions to all aspects of the project,
+including the documentation.
In HBase, documentation includes the following areas, and probably some others:
* The link:http://hbase.apache.org/book.html[HBase Reference
Guide] (this book)
* The link:http://hbase.apache.org/[HBase website]
-* The link:http://wiki.apache.org/hadoop/Hbase[HBase
- Wiki]
* API documentation
* Command-line utility output and help text
* Web UI strings, explicit help text, context-sensitive strings, and others
@@ -46,99 +45,122 @@ In HBase, documentation includes the following areas, and probably some others:
* Comments in source files, configuration files, and others
* Localization of any of the above into target languages other than English
-No matter which area you want to help out with, the first step is almost always to download (typically by cloning the Git repository) and familiarize yourself with the HBase source code.
-The only exception in the list above is the HBase Wiki, which is edited online.
-For information on downloading and building the source, see <<developer,developer>>.
-
-=== Getting Access to the Wiki
-
-The HBase Wiki is not well-maintained and much of its content has been moved into the HBase Reference Guide (this guide). However, some pages on the Wiki are well maintained, and it would be great to have some volunteers willing to help out with the Wiki.
-To request access to the Wiki, register a new account at link:https://wiki.apache.org/hadoop/Hbase?action=newaccount[https://wiki.apache.org/hadoop/Hbase?action=newaccount].
-Contact one of the HBase committers, who can either give you access or refer you to someone who can.
+No matter which area you want to help out with, the first step is almost always
+to download (typically by cloning the Git repository) and familiarize yourself
+with the HBase source code. For information on downloading and building the source,
+see <<developer,developer>>.
=== Contributing to Documentation or Other Strings
-If you spot an error in a string in a UI, utility, script, log message, or elsewhere, or you think something could be made more clear, or you think text needs to be added where it doesn't currently exist, the first step is to file a JIRA.
-Be sure to set the component to `Documentation` in addition any other involved components.
-Most components have one or more default owners, who monitor new issues which come into those queues.
-Regardless of whether you feel able to fix the bug, you should still file bugs where you see them.
+If you spot an error in a string in a UI, utility, script, log message, or elsewhere,
+or you think something could be made more clear, or you think text needs to be added
+where it doesn't currently exist, the first step is to file a JIRA. Be sure to set
+the component to `Documentation` in addition any other involved components. Most
+components have one or more default owners, who monitor new issues which come into
+those queues. Regardless of whether you feel able to fix the bug, you should still
+file bugs where you see them.
If you want to try your hand at fixing your newly-filed bug, assign it to yourself.
-You will need to clone the HBase Git repository to your local system and work on the issue there.
-When you have developed a potential fix, submit it for review.
-If it addresses the issue and is seen as an improvement, one of the HBase committers will commit it to one or more branches, as appropriate.
+You will need to clone the HBase Git repository to your local system and work on
+the issue there. When you have developed a potential fix, submit it for review.
+If it addresses the issue and is seen as an improvement, one of the HBase committers
+will commit it to one or more branches, as appropriate.
+[[submit_doc_patch_procedure]]
.Procedure: Suggested Work flow for Submitting Patches
-This procedure goes into more detail than Git pros will need, but is included in this appendix so that people unfamiliar with Git can feel confident contributing to HBase while they learn.
+This procedure goes into more detail than Git pros will need, but is included
+in this appendix so that people unfamiliar with Git can feel confident contributing
+to HBase while they learn.
. If you have not already done so, clone the Git repository locally.
You only need to do this once.
-. Fairly often, pull remote changes into your local repository by using the `git pull` command, while your master branch is checked out.
+. Fairly often, pull remote changes into your local repository by using the
+`git pull` command, while your tracking branch is checked out.
. For each issue you work on, create a new branch.
- One convention that works well for naming the branches is to name a given branch the same as the JIRA it relates to:
+ One convention that works well for naming the branches is to name a given branch
+ the same as the JIRA it relates to:
+
----
$ git checkout -b HBASE-123456
----
-. Make your suggested changes on your branch, committing your changes to your local repository often.
- If you need to switch to working on a different issue, remember to check out the appropriate branch.
-. When you are ready to submit your patch, first be sure that HBase builds cleanly and behaves as expected in your modified branch.
- If you have made documentation changes, be sure the documentation and website builds by running `mvn clean site`.
-+
-NOTE: Before you use the `site` target the very first time, be sure you have built HBase at least once, in order to fetch all the Maven dependencies you need.
-+
-----
-$ mvn clean install -DskipTests # Builds HBase
-----
-+
-----
-$ mvn clean site -DskipTests # Builds the website and documentation
-----
-+
-If any errors occur, address them.
-
-. If it takes you several days or weeks to implement your fix, or you know that the area of the code you are working in has had a lot of changes lately, make sure you rebase your branch against the remote master and take care of any conflicts before submitting your patch.
+. Make your suggested changes on your branch, committing your changes to your
+local repository often. If you need to switch to working on a different issue,
+remember to check out the appropriate branch.
+. When you are ready to submit your patch, first be sure that HBase builds cleanly
+and behaves as expected in your modified branch.
+. If you have made documentation changes, be sure the documentation and website
+builds by running `mvn clean site`.
+. If it takes you several days or weeks to implement your fix, or you know that
+the area of the code you are working in has had a lot of changes lately, make
+sure you rebase your branch against the remote master and take care of any conflicts
+before submitting your patch.
+
----
-
$ git checkout HBASE-123456
$ git rebase origin/master
----
-. Generate your patch against the remote master.
- Run the following command from the top level of your git repository (usually called `hbase`):
+. Generate your patch against the remote master. Run the following command from
+the top level of your git repository (usually called `hbase`):
+
----
$ git format-patch --stdout origin/master > HBASE-123456.patch
----
+
The name of the patch should contain the JIRA ID.
-Look over the patch file to be sure that you did not change any additional files by accident and that there are no other surprises.
-When you are satisfied, attach the patch to the JIRA and click the btn:[Patch Available] button.
-A reviewer will review your patch.
-If you need to submit a new version of the patch, leave the old one on the JIRA and add a version number to the name of the new patch.
-
+. Look over the patch file to be sure that you did not change any additional files
+by accident and that there are no other surprises.
+. When you are satisfied, attach the patch to the JIRA and click the
+btn:[Patch Available] button. A reviewer will review your patch.
+. If you need to submit a new version of the patch, leave the old one on the
+JIRA and add a version number to the name of the new patch.
. After a change has been committed, there is no need to keep your local branch around.
- Instead you should run `git pull` to get the new change into your master branch.
=== Editing the HBase Website
The source for the HBase website is in the HBase source, in the _src/main/site/_ directory.
-Within this directory, source for the individual pages is in the _xdocs/_ directory, and images referenced in those pages are in the _images/_ directory.
+Within this directory, source for the individual pages is in the _xdocs/_ directory,
+and images referenced in those pages are in the _resources/images/_ directory.
This directory also stores images used in the HBase Reference Guide.
-The website's pages are written in an HTML-like XML dialect called xdoc, which has a reference guide at link:http://maven.apache.org/archives/maven-1.x/plugins/xdoc/reference/xdocs.html.
-You can edit these files in a plain-text editor, an IDE, or an XML editor such as XML Mind XML Editor (XXE) or Oxygen XML Author.
+The website's pages are written in an HTML-like XML dialect called xdoc, which
+has a reference guide at
+http://maven.apache.org/archives/maven-1.x/plugins/xdoc/reference/xdocs.html.
+You can edit these files in a plain-text editor, an IDE, or an XML editor such
+as XML Mind XML Editor (XXE) or Oxygen XML Author.
+
+To preview your changes, build the website using the `mvn clean site -DskipTests`
+command. The HTML output resides in the _target/site/_ directory.
+When you are satisfied with your changes, follow the procedure in
+<<submit_doc_patch_procedure,submit doc patch procedure>> to submit your patch.
-To preview your changes, build the website using the +mvn clean site
- -DskipTests+ command.
-The HTML output resides in the _target/site/_ directory.
-When you are satisfied with your changes, follow the procedure in <<submit_doc_patch_procedure,submit doc patch procedure>> to submit your patch.
+[[website_publish]]
+=== Publishing the HBase Website and Documentation
+
+HBase uses the ASF's `gitpubsub` mechanism.
+. After generating the website and documentation
+artifacts using `mvn clean site site:stage`, check out the `asf-site` repository.
+
+. Remove previously-generated content using the following command:
++
+----
+rm -rf rm -rf *apidocs* *xref* *book* *.html *.pdf* css js
+----
++
+WARNING: Do not remove the `0.94/` directory. To regenerate them, you must check out
+the 0.94 branch and run `mvn clean site site:stage` from there, and then copy the
+artifacts to the 0.94/ directory of the `asf-site` branch.
+
+. Copy the contents of `target/staging` to the branch.
+
+. Add and commit your changes, and submit a patch for review.
=== HBase Reference Guide Style Guide and Cheat Sheet
-The HBase Reference Guide is written in Asciidoc and built using link:http://asciidoctor.org[AsciiDoctor]. The following cheat sheet is included for your reference. More nuanced and comprehensive documentation is available at link:http://asciidoctor.org/docs/user-manual/.
+The HBase Reference Guide is written in Asciidoc and built using link:http://asciidoctor.org[AsciiDoctor].
+The following cheat sheet is included for your reference. More nuanced and comprehensive documentation
+is available at http://asciidoctor.org/docs/user-manual/.
.AsciiDoc Cheat Sheet
[cols="1,1,a",options="header"]
@@ -147,15 +169,15 @@ The HBase Reference Guide is written in Asciidoc and built using link:http://asc
| A paragraph | a paragraph | Just type some text with a blank line at the top and bottom.
| Add line breaks within a paragraph without adding blank lines | Manual line breaks | This will break + at the plus sign. Or prefix the whole paragraph with a line containing '[%hardbreaks]'
| Give a title to anything | Colored italic bold differently-sized text | .MyTitle (no space between the period and the words) on the line before the thing to be titled
-| In-Line Code or commands | monospace | \`text`
+| In-Line Code or commands | monospace | \`text`
| In-line literal content (things to be typed exactly as shown) | bold mono | \*\`typethis`*
| In-line replaceable content (things to substitute with your own values) | bold italic mono | \*\_typesomething_*
-| Code blocks with highlighting | monospace, highlighted, preserve space |
+| Code blocks with highlighting | monospace, highlighted, preserve space |
........
[source,java]
-----
- myAwesomeCode() {
-}
+----
+ myAwesomeCode() {
+}
----
........
| Code block included from a separate file | included just as though it were part of the main file |
@@ -165,51 +187,52 @@ The HBase Reference Guide is written in Asciidoc and built using link:http://asc
include\::path/to/app.rb[]
----
................
-| Include only part of a separate file | Similar to Javadoc | See link:http://asciidoctor.org/docs/user-manual/#by-tagged-regions
+| Include only part of a separate file | Similar to Javadoc
+| See http://asciidoctor.org/docs/user-manual/#by-tagged-regions
| Filenames, directory names, new terms | italic | \_hbase-default.xml_
-| External naked URLs | A link with the URL as link text |
+| External naked URLs | A link with the URL as link text |
----
link:http://www.google.com
----
-| External URLs with text | A link with arbitrary link text |
+| External URLs with text | A link with arbitrary link text |
----
link:http://www.google.com[Google]
----
-| Create an internal anchor to cross-reference | not rendered |
+| Create an internal anchor to cross-reference | not rendered |
----
[[anchor_name]]
----
-| Cross-reference an existing anchor using its default title| an internal hyperlink using the element title if available, otherwise using the anchor name |
+| Cross-reference an existing anchor using its default title| an internal hyperlink using the element title if available, otherwise using the anchor name |
----
<<anchor_name>>
----
-| Cross-reference an existing anchor using custom text | an internal hyperlink using arbitrary text |
+| Cross-reference an existing anchor using custom text | an internal hyperlink using arbitrary text |
----
<<anchor_name,Anchor Text>>
----
-| A block image | The image with alt text |
+| A block image | The image with alt text |
----
-image::sunset.jpg[Alt Text]
+image::sunset.jpg[Alt Text]
----
(put the image in the src/main/site/resources/images directory)
-| An inline image | The image with alt text, as part of the text flow |
+| An inline image | The image with alt text, as part of the text flow |
----
image:sunset.jpg [Alt Text]
----
(only one colon)
-| Link to a remote image | show an image hosted elsewhere |
+| Link to a remote image | show an image hosted elsewhere |
----
-image::http://inkscape.org/doc/examples/tux.svg[Tux,250,350]
+image::http://inkscape.org/doc/examples/tux.svg[Tux,250,350]
----
(or `image:`)
| Add dimensions or a URL to the image | depends | inside the brackets after the alt text, specify width, height and/or link="http://my_link.com"
-| A footnote | subscript link which takes you to the footnote |
+| A footnote | subscript link which takes you to the footnote |
----
Some text.footnote:[The footnote text.]
----
-| A note or warning with no title | The admonition image followed by the admonition |
+| A note or warning with no title | The admonition image followed by the admonition |
----
NOTE:My note here
----
@@ -217,7 +240,7 @@ NOTE:My note here
----
WARNING:My warning here
----
-| A complex note | The note has a title and/or multiple paragraphs and/or code blocks or lists, etc |
+| A complex note | The note has a title and/or multiple paragraphs and/or code blocks or lists, etc |
........
.The Title
[NOTE]
@@ -228,26 +251,26 @@ some source code
----
====
........
-| Bullet lists | bullet lists |
+| Bullet lists | bullet lists |
----
* list item 1
----
(see http://asciidoctor.org/docs/user-manual/#unordered-lists)
-| Numbered lists | numbered list |
+| Numbered lists | numbered list |
----
-. list item 2
+. list item 2
----
(see http://asciidoctor.org/docs/user-manual/#ordered-lists)
-| Checklists | Checked or unchecked boxes |
+| Checklists | Checked or unchecked boxes |
Checked:
----
-- [*]
+- [*]
----
Unchecked:
----
- [ ]
----
-| Multiple levels of lists | bulleted or numbered or combo |
+| Multiple levels of lists | bulleted or numbered or combo |
----
. Numbered (1), at top level
* Bullet (2), nested under 1
@@ -257,14 +280,18 @@ Unchecked:
** Bullet (6), nested under 5
- [x] Checked (7), at top level
----
-| Labelled lists / variablelists | a list item title or summary followed by content |
+| Labelled lists / variablelists | a list item title or summary followed by content |
----
-Title:: content
+Title:: content
Title::
content
----
-| Sidebars, quotes, or other blocks of text | a block of text, formatted differently from the default | Delimited using different delimiters, see link:http://asciidoctor.org/docs/user-manual/#built-in-blocks-summary. Some of the examples above use delimiters like \...., ----,====.
+| Sidebars, quotes, or other blocks of text
+| a block of text, formatted differently from the default
+| Delimited using different delimiters,
+see http://asciidoctor.org/docs/user-manual/#built-in-blocks-summary.
+Some of the examples above use delimiters like \...., ----,====.
........
[example]
====
@@ -288,7 +315,7 @@ ____
........
If you want to insert literal Asciidoc content that keeps being interpreted, when in doubt, use eight dots as the delimiter at the top and bottom.
-| Nested Sections | chapter, section, sub-section, etc |
+| Nested Sections | chapter, section, sub-section, etc |
----
= Book (or chapter if the chapter can be built alone, see the leveloffset info below)
@@ -296,7 +323,7 @@ If you want to insert literal Asciidoc content that keeps being interpreted, whe
=== Section (or subsection, etc)
-==== Subsection
+==== Subsection
----
and so on up to 6 levels (think carefully about going deeper than 4 levels, maybe you can just titled paragraphs or lists instead). Note that you can include a book inside another book by adding the `:leveloffset:+1` macro directive directly before your include, and resetting it to 0 directly after. See the _book.adoc_ source for examples, as this is how this guide handles chapters. *Don't do it for prefaces, glossaries, appendixes, or other special types of chapters.*
@@ -309,7 +336,7 @@ include::[/path/to/file.adoc]
For plenty of examples. see _book.adoc_.
| A table | a table | See http://asciidoctor.org/docs/user-manual/#tables. Generally rows are separated by newlines and columns by pipes
-| Comment out a single line | A line is skipped during rendering |
+| Comment out a single line | A line is skipped during rendering |
`+//+ This line won't show up`
| Comment out a block | A section of the file is skipped during rendering |
----
@@ -317,7 +344,7 @@ For plenty of examples. see _book.adoc_.
Nothing between the slashes will show up.
////
----
-| Highlight text for review | text shows up with yellow background |
+| Highlight text for review | text shows up with yellow background |
----
Test between #hash marks# is highlighted yellow.
----
@@ -326,20 +353,27 @@ Test between #hash marks# is highlighted yellow.
=== Auto-Generated Content
-Some parts of the HBase Reference Guide, most notably <<config.files,config.files>>, are generated automatically, so that this area of the documentation stays in sync with the code.
-This is done by means of an XSLT transform, which you can examine in the source at _src/main/xslt/configuration_to_asciidoc_chapter.xsl_.
-This transforms the _hbase-common/src/main/resources/hbase-default.xml_ file into an Asciidoc output which can be included in the Reference Guide.
-Sometimes, it is necessary to add configuration parameters or modify their descriptions.
-Make the modifications to the source file, and they will be included in the Reference Guide when it is rebuilt.
+Some parts of the HBase Reference Guide, most notably <<config.files,config.files>>,
+are generated automatically, so that this area of the documentation stays in
+sync with the code. This is done by means of an XSLT transform, which you can examine
+in the source at _src/main/xslt/configuration_to_asciidoc_chapter.xsl_. This
+transforms the _hbase-common/src/main/resources/hbase-default.xml_ file into an
+Asciidoc output which can be included in the Reference Guide.
-It is possible that other types of content can and will be automatically generated from HBase source files in the future.
+Sometimes, it is necessary to add configuration parameters or modify their descriptions.
+Make the modifications to the source file, and they will be included in the
+Reference Guide when it is rebuilt.
+It is possible that other types of content can and will be automatically generated
+from HBase source files in the future.
=== Images in the HBase Reference Guide
-You can include images in the HBase Reference Guide. It is important to include an image title if possible, and alternate text always.
-This allows screen readers to navigate to the image and also provides alternative text for the image.
-The following is an example of an image with a title and alternate text. Notice the double colon.
+You can include images in the HBase Reference Guide. It is important to include
+an image title if possible, and alternate text always. This allows screen readers
+to navigate to the image and also provides alternative text for the image.
+The following is an example of an image with a title and alternate text. Notice
+the double colon.
[source,asciidoc]
----
@@ -347,42 +381,53 @@ The following is an example of an image with a title and alternate text. Notice
image::sunset.jpg[Alt Text]
----
-Here is an example of an inline image with alternate text. Notice the single colon. Inline images cannot have titles. They are generally small images like GUI buttons.
+Here is an example of an inline image with alternate text. Notice the single colon.
+Inline images cannot have titles. They are generally small images like GUI buttons.
[source,asciidoc]
----
image:sunset.jpg[Alt Text]
----
-
When doing a local build, save the image to the _src/main/site/resources/images/_ directory.
When you link to the image, do not include the directory portion of the path.
The image will be copied to the appropriate target location during the build of the output.
-When you submit a patch which includes adding an image to the HBase Reference Guide, attach the image to the JIRA.
-If the committer asks where the image should be committed, it should go into the above directory.
+When you submit a patch which includes adding an image to the HBase Reference Guide,
+attach the image to the JIRA. If the committer asks where the image should be
+committed, it should go into the above directory.
=== Adding a New Chapter to the HBase Reference Guide
-If you want to add a new chapter to the HBase Reference Guide, the easiest way is to copy an existing chapter file, rename it, and change the ID (in double brackets) and title. Chapters are located in the _src/main/asciidoc/_chapters/_ directory.
+If you want to add a new chapter to the HBase Reference Guide, the easiest way
+is to copy an existing chapter file, rename it, and change the ID (in double
+brackets) and title. Chapters are located in the _src/main/asciidoc/_chapters/_
+directory.
-Delete the existing content and create the new content.
-Then open the _src/main/asciidoc/book.adoc_ file, which is the main file for the HBase Reference Guide, and copy an existing `include` element to include your new chapter in the appropriate location.
-Be sure to add your new file to your Git repository before creating your patch.
+Delete the existing content and create the new content. Then open the
+_src/main/asciidoc/book.adoc_ file, which is the main file for the HBase Reference
+Guide, and copy an existing `include` element to include your new chapter in the
+appropriate location. Be sure to add your new file to your Git repository before
+creating your patch.
When in doubt, check to see how other files have been included.
=== Common Documentation Issues
-The following documentation issues come up often.
-Some of these are preferences, but others can create mysterious build errors or other problems.
+The following documentation issues come up often. Some of these are preferences,
+but others can create mysterious build errors or other problems.
[qanda]
Isolate Changes for Easy Diff Review.::
- Be careful with pretty-printing or re-formatting an entire XML file, even if the formatting has degraded over time. If you need to reformat a file, do that in a separate JIRA where you do not change any content. Be careful because some XML editors do a bulk-reformat when you open a new file, especially if you use GUI mode in the editor.
+ Be careful with pretty-printing or re-formatting an entire XML file, even if
+ the formatting has degraded over time. If you need to reformat a file, do that
+ in a separate JIRA where you do not change any content. Be careful because some
+ XML editors do a bulk-reformat when you open a new file, especially if you use
+ GUI mode in the editor.
Syntax Highlighting::
- The HBase Reference Guide uses `coderay` for syntax highlighting. To enable syntax highlighting for a given code listing, use the following type of syntax:
+ The HBase Reference Guide uses `coderay` for syntax highlighting. To enable
+ syntax highlighting for a given code listing, use the following type of syntax:
+
........
[source,xml]
@@ -391,5 +436,6 @@ Syntax Highlighting::
----
........
+
-Several syntax types are supported. The most interesting ones for the HBase Reference Guide are `java`, `xml`, `sql`, and `bash`.
+Several syntax types are supported. The most interesting ones for the HBase
+Reference Guide are `java`, `xml`, `sql`, and `bash`.
http://git-wip-us.apache.org/repos/asf/hbase/blob/6cb8a436/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
----------------------------------------------------------------------
diff --git a/src/main/asciidoc/_chapters/appendix_hfile_format.adoc b/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
index b74763c..18eafe6 100644
--- a/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
+++ b/src/main/asciidoc/_chapters/appendix_hfile_format.adoc
@@ -44,16 +44,16 @@ An HFile in version 1 format is structured as follows:
.HFile V1 Format
image::hfile.png[HFile Version 1]
-==== Block index format in version 1
+==== Block index format in version 1
The block index in version 1 is very straightforward.
-For each entry, it contains:
+For each entry, it contains:
. Offset (long)
. Uncompressed size (int)
-. Key (a serialized byte array written using Bytes.writeByteArray)
-.. Key length as a variable-length integer (VInt)
-.. Key bytes
+. Key (a serialized byte array written using Bytes.writeByteArray)
+.. Key length as a variable-length integer (VInt)
+.. Key bytes
The number of entries in the block index is stored in the fixed file trailer, and has to be passed in to the method that reads the block index.
@@ -66,7 +66,7 @@ We fix this limitation in version 2, where we store on-disk block size instead o
Note: this feature was introduced in HBase 0.92
-==== Motivation
+==== Motivation
We found it necessary to revise the HFile format after encountering high memory usage and slow startup times caused by large Bloom filters and block indexes in the region server.
Bloom filters can get as large as 100 MB per HFile, which adds up to 2 GB when aggregated over 20 regions.
@@ -80,7 +80,7 @@ Bloom filter blocks and index blocks (we call these "inline blocks") become inte
HFile is a low-level file format by design, and it should not deal with application-specific details such as Bloom filters, which are handled at StoreFile level.
Therefore, we call Bloom filter blocks in an HFile "inline" blocks.
-We also supply HFile with an interface to write those inline blocks.
+We also supply HFile with an interface to write those inline blocks.
Another format modification aimed at reducing the region server startup time is to use a contiguous "load-on-open" section that has to be loaded in memory at the time an HFile is being opened.
Currently, as an HFile opens, there are separate seek operations to read the trailer, data/meta indexes, and file info.
@@ -91,57 +91,57 @@ In version 2, we seek once to read the trailer and seek again to read everything
==== Overview of Version 2
The version of HBase introducing the above features reads both version 1 and 2 HFiles, but only writes version 2 HFiles.
-A version 2 HFile is structured as follows:
+A version 2 HFile is structured as follows:
.HFile Version 2 Structure
-image:hfilev2.png[HFile Version 2]
+image:hfilev2.png[HFile Version 2]
==== Unified version 2 block format
-In the version 2 every block in the data section contains the following fields:
-
-. 8 bytes: Block type, a sequence of bytes equivalent to version 1's "magic records". Supported block types are:
-.. DATA \u2013 data blocks
-.. LEAF_INDEX \u2013 leaf-level index blocks in a multi-level-block-index
-.. BLOOM_CHUNK \u2013 Bloom filter chunks
-.. META \u2013 meta blocks (not used for Bloom filters in version 2 anymore)
-.. INTERMEDIATE_INDEX \u2013 intermediate-level index blocks in a multi-level blockindex
-.. ROOT_INDEX \u2013 root>level index blocks in a multi>level block index
-.. FILE_INFO \u2013 the ``file info'' block, a small key>value map of metadata
-.. BLOOM_META \u2013 a Bloom filter metadata block in the load>on>open section
+In the version 2 every block in the data section contains the following fields:
+
+. 8 bytes: Block type, a sequence of bytes equivalent to version 1's "magic records". Supported block types are:
+.. DATA \u2013 data blocks
+.. LEAF_INDEX \u2013 leaf-level index blocks in a multi-level-block-index
+.. BLOOM_CHUNK \u2013 Bloom filter chunks
+.. META \u2013 meta blocks (not used for Bloom filters in version 2 anymore)
+.. INTERMEDIATE_INDEX \u2013 intermediate-level index blocks in a multi-level blockindex
+.. ROOT_INDEX \u2013 root>level index blocks in a multi>level block index
+.. FILE_INFO \u2013 the ``file info'' block, a small key>value map of metadata
+.. BLOOM_META \u2013 a Bloom filter metadata block in the load>on>open section
.. TRAILER \u2013 a fixed>size file trailer.
- As opposed to the above, this is not an HFile v2 block but a fixed>size (for each HFile version) data structure
-.. INDEX_V1 \u2013 this block type is only used for legacy HFile v1 block
-. Compressed size of the block's data, not including the header (int).
+ As opposed to the above, this is not an HFile v2 block but a fixed>size (for each HFile version) data structure
+.. INDEX_V1 \u2013 this block type is only used for legacy HFile v1 block
+. Compressed size of the block's data, not including the header (int).
+
-Can be used for skipping the current data block when scanning HFile data.
+Can be used for skipping the current data block when scanning HFile data.
. Uncompressed size of the block's data, not including the header (int)
+
-This is equal to the compressed size if the compression algorithm is NONE
+This is equal to the compressed size if the compression algorithm is NONE
. File offset of the previous block of the same type (long)
+
-Can be used for seeking to the previous data/index block
+Can be used for seeking to the previous data/index block
. Compressed data (or uncompressed data if the compression algorithm is NONE).
The above format of blocks is used in the following HFile sections:
Scanned block section::
The section is named so because it contains all data blocks that need to be read when an HFile is scanned sequentially.
- Also contains leaf block index and Bloom chunk blocks.
+ Also contains leaf block index and Bloom chunk blocks.
Non-scanned block section::
This section still contains unified-format v2 blocks but it does not have to be read when doing a sequential scan.
- This section contains "meta" blocks and intermediate-level index blocks.
+ This section contains "meta" blocks and intermediate-level index blocks.
-We are supporting "meta" blocks in version 2 the same way they were supported in version 1, even though we do not store Bloom filter data in these blocks anymore.
+We are supporting "meta" blocks in version 2 the same way they were supported in version 1, even though we do not store Bloom filter data in these blocks anymore.
==== Block index in version 2
-There are three types of block indexes in HFile version 2, stored in two different formats (root and non-root):
+There are three types of block indexes in HFile version 2, stored in two different formats (root and non-root):
. Data index -- version 2 multi-level block index, consisting of:
-.. Version 2 root index, stored in the data block index section of the file
-.. Optionally, version 2 intermediate levels, stored in the non%root format in the data index section of the file. Intermediate levels can only be present if leaf level blocks are present
-.. Optionally, version 2 leaf levels, stored in the non%root format inline with data blocks
+.. Version 2 root index, stored in the data block index section of the file
+.. Optionally, version 2 intermediate levels, stored in the non%root format in the data index section of the file. Intermediate levels can only be present if leaf level blocks are present
+.. Optionally, version 2 leaf levels, stored in the non%root format inline with data blocks
. Meta index -- version 2 root index format only, stored in the meta index section of the file
. Bloom index -- version 2 root index format only, stored in the ``load-on-open'' section as part of Bloom filter metadata.
@@ -150,19 +150,19 @@ There are three types of block indexes in HFile version 2, stored in two differe
This format applies to:
. Root level of the version 2 data index
-. Entire meta and Bloom indexes in version 2, which are always single-level.
+. Entire meta and Bloom indexes in version 2, which are always single-level.
-A version 2 root index block is a sequence of entries of the following format, similar to entries of a version 1 block index, but storing on-disk size instead of uncompressed size.
+A version 2 root index block is a sequence of entries of the following format, similar to entries of a version 1 block index, but storing on-disk size instead of uncompressed size.
-. Offset (long)
+. Offset (long)
+
-This offset may point to a data block or to a deeper>level index block.
+This offset may point to a data block or to a deeper>level index block.
-. On-disk size (int)
-. Key (a serialized byte array stored using Bytes.writeByteArray)
+. On-disk size (int)
+. Key (a serialized byte array stored using Bytes.writeByteArray)
+
-. Key (VInt)
-. Key bytes
+. Key (VInt)
+. Key bytes
A single-level version 2 block index consists of just a single root index block.
@@ -172,13 +172,13 @@ For the data index and the meta index the number of entries is stored in the tra
For a multi-level block index we also store the following fields in the root index block in the load-on-open section of the HFile, in addition to the data structure described above:
. Middle leaf index block offset
-. Middle leaf block on-disk size (meaning the leaf index block containing the reference to the ``middle'' data block of the file)
+. Middle leaf block on-disk size (meaning the leaf index block containing the reference to the ``middle'' data block of the file)
. The index of the mid-key (defined below) in the middle leaf-level block.
These additional fields are used to efficiently retrieve the mid-key of the HFile used in HFile splits, which we define as the first key of the block with a zero-based index of (n \u2013 1) / 2, if the total number of blocks in the HFile is n.
-This definition is consistent with how the mid-key was determined in HFile version 1, and is reasonable in general, because blocks are likely to be the same size on average, but we don't have any estimates on individual key/value pair sizes.
+This definition is consistent with how the mid-key was determined in HFile version 1, and is reasonable in general, because blocks are likely to be the same size on average, but we don't have any estimates on individual key/value pair sizes.
@@ -189,52 +189,57 @@ When reading the HFile and the mid-key is requested, we retrieve the middle leaf
==== Non-root block index format in version 2
This format applies to intermediate-level and leaf index blocks of a version 2 multi-level data block index.
-Every non-root index block is structured as follows.
-
-. numEntries: the number of entries (int).
-. entryOffsets: the ``secondary index'' of offsets of entries in the block, to facilitate a quick binary search on the key (numEntries + 1 int values). The last value is the total length of all entries in this index block.
- For example, in a non-root index block with entry sizes 60, 80, 50 the ``secondary index'' will contain the following int array: {0, 60, 140, 190}.
+Every non-root index block is structured as follows.
+
+. numEntries: the number of entries (int).
+. entryOffsets: the "secondary index" of offsets of entries in the block, to facilitate
+ a quick binary search on the key (`numEntries + 1` int values). The last value
+ is the total length of all entries in this index block. For example, in a non-root
+ index block with entry sizes 60, 80, 50 the "secondary index" will contain the
+ following int array: `{0, 60, 140, 190}`.
. Entries.
- Each entry contains:
+ Each entry contains:
+
-. Offset of the block referenced by this entry in the file (long)
-. On>disk size of the referenced block (int)
+. Offset of the block referenced by this entry in the file (long)
+. On>disk size of the referenced block (int)
. Key.
- The length can be calculated from entryOffsets.
+ The length can be calculated from entryOffsets.
==== Bloom filters in version 2
-In contrast with version 1, in a version 2 HFile Bloom filter metadata is stored in the load-on-open section of the HFile for quick startup.
+In contrast with version 1, in a version 2 HFile Bloom filter metadata is stored in the load-on-open section of the HFile for quick startup.
-. A compound Bloom filter.
+. A compound Bloom filter.
+
-. Bloom filter version = 3 (int). There used to be a DynamicByteBloomFilter class that had the Bloom filter version number 2
-. The total byte size of all compound Bloom filter chunks (long)
-. Number of hash functions (int
-. Type of hash functions (int)
-. The total key count inserted into the Bloom filter (long)
-. The maximum total number of keys in the Bloom filter (long)
-. The number of chunks (int)
-. Comparator class used for Bloom filter keys, a UTF>8 encoded string stored using Bytes.writeByteArray
-. Bloom block index in the version 2 root block index format
+. Bloom filter version = 3 (int). There used to be a DynamicByteBloomFilter class that had the Bloom filter version number 2
+. The total byte size of all compound Bloom filter chunks (long)
+. Number of hash functions (int
+. Type of hash functions (int)
+. The total key count inserted into the Bloom filter (long)
+. The maximum total number of keys in the Bloom filter (long)
+. The number of chunks (int)
+. Comparator class used for Bloom filter keys, a UTF>8 encoded string stored using Bytes.writeByteArray
+. Bloom block index in the version 2 root block index format
==== File Info format in versions 1 and 2
-The file info block is a serialized link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/io/HbaseMapWritable.html[HbaseMapWritable] (essentially a map from byte arrays to byte arrays) with the following keys, among others.
+The file info block is a serialized map from byte arrays to byte arrays, with the following keys, among others.
StoreFile-level logic adds more keys to this.
[cols="1,1", frame="all"]
|===
|hfile.LASTKEY| The last key of the file (byte array)
|hfile.AVG_KEY_LEN| The average key length in the file (int)
-|hfile.AVG_VALUE_LEN| The average value length in the file (int)
+|hfile.AVG_VALUE_LEN| The average value length in the file (int)
|===
-File info format did not change in version 2.
-However, we moved the file info to the final section of the file, which can be loaded as one block at the time the HFile is being opened.
-Also, we do not store comparator in the version 2 file info anymore.
+In version 2, we did not change the file format, but we moved the file info to
+the final section of the file, which can be loaded as one block when the HFile
+is being opened.
+
+Also, we do not store the comparator in the version 2 file info anymore.
Instead, we store it in the fixed file trailer.
This is because we need to know the comparator at the time of parsing the load-on-open section of the HFile.
@@ -242,14 +247,15 @@ This is because we need to know the comparator at the time of parsing the load-o
The following table shows common and different fields between fixed file trailers in versions 1 and 2.
Note that the size of the trailer is different depending on the version, so it is ``fixed'' only within one version.
-However, the version is always stored as the last four-byte integer in the file.
+However, the version is always stored as the last four-byte integer in the file.
.Differences between HFile Versions 1 and 2
[cols="1,1", frame="all"]
|===
| Version 1 | Version 2
| |File info offset (long)
-| Data index offset (long)| loadOnOpenOffset (long) /The offset of the sectionthat we need toload when opening the file./
+| Data index offset (long)
+| loadOnOpenOffset (long) /The offset of the section that we need to load when opening the file./
| | Number of data index entries (int)
| metaIndexOffset (long) /This field is not being used by the version 1 reader, so we removed it from version 2./ | uncompressedDataIndexSize (long) /The total uncompressed size of the whole data block index, including root-level, intermediate-level, and leaf-level blocks./
| | Number of meta index entries (int)
@@ -257,7 +263,7 @@ However, the version is always stored as the last four-byte integer in the file.
| numEntries (int) | numEntries (long)
| Compression codec: 0 = LZO, 1 = GZ, 2 = NONE (int) | Compression codec: 0 = LZO, 1 = GZ, 2 = NONE (int)
| | The number of levels in the data block index (int)
-| | firstDataBlockOffset (long) /The offset of the first first data block. Used when scanning./
+| | firstDataBlockOffset (long) /The offset of the first data block. Used when scanning./
| | lastDataBlockEnd (long) /The offset of the first byte after the last key/value data block. We don't need to go beyond this offset when scanning./
| Version: 1 (int) | Version: 2 (int)
|===
@@ -290,42 +296,42 @@ This optimization (implemented by the getShortMidpointKey method) is inspired by
Note: this feature was introduced in HBase 0.98
[[hfilev3.motivation]]
-==== Motivation
+==== Motivation
-Version 3 of HFile makes changes needed to ease management of encryption at rest and cell-level metadata (which in turn is needed for cell-level ACLs and cell-level visibility labels). For more information see <<hbase.encryption.server,hbase.encryption.server>>, <<hbase.tags,hbase.tags>>, <<hbase.accesscontrol.configuration,hbase.accesscontrol.configuration>>, and <<hbase.visibility.labels,hbase.visibility.labels>>.
+Version 3 of HFile makes changes needed to ease management of encryption at rest and cell-level metadata (which in turn is needed for cell-level ACLs and cell-level visibility labels). For more information see <<hbase.encryption.server,hbase.encryption.server>>, <<hbase.tags,hbase.tags>>, <<hbase.accesscontrol.configuration,hbase.accesscontrol.configuration>>, and <<hbase.visibility.labels,hbase.visibility.labels>>.
[[hfilev3.overview]]
==== Overview
The version of HBase introducing the above features reads HFiles in versions 1, 2, and 3 but only writes version 3 HFiles.
Version 3 HFiles are structured the same as version 2 HFiles.
-For more information see <<hfilev2.overview,hfilev2.overview>>.
+For more information see <<hfilev2.overview,hfilev2.overview>>.
[[hvilev3.infoblock]]
==== File Info Block in Version 3
-Version 3 added two additional pieces of information to the reserved keys in the file info block.
+Version 3 added two additional pieces of information to the reserved keys in the file info block.
[cols="1,1", frame="all"]
|===
| hfile.MAX_TAGS_LEN | The maximum number of bytes needed to store the serialized tags for any single cell in this hfile (int)
| hfile.TAGS_COMPRESSED | Does the block encoder for this hfile compress tags? (boolean). Should only be present if hfile.MAX_TAGS_LEN is also present.
-|===
+|===
When reading a Version 3 HFile the presence of `MAX_TAGS_LEN` is used to determine how to deserialize the cells within a data block.
-Therefore, consumers must read the file's info block prior to reading any data blocks.
+Therefore, consumers must read the file's info block prior to reading any data blocks.
-When writing a Version 3 HFile, HBase will always include `MAX_TAGS_LEN ` when flushing the memstore to underlying filesystem and when using prefix tree encoding for data blocks, as described in <<compression,compression>>.
+When writing a Version 3 HFile, HBase will always include `MAX_TAGS_LEN ` when flushing the memstore to underlying filesystem and when using prefix tree encoding for data blocks, as described in <<compression,compression>>.
When compacting extant files, the default writer will omit `MAX_TAGS_LEN` if all of the files selected do not themselves contain any cells with tags.
-See <<compaction,compaction>> for details on the compaction file selection algorithm.
+See <<compaction,compaction>> for details on the compaction file selection algorithm.
[[hfilev3.datablock]]
==== Data Blocks in Version 3
Within an HFile, HBase cells are stored in data blocks as a sequence of KeyValues (see <<hfilev1.overview,hfilev1.overview>>, or link:http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html[Lars George's
- excellent introduction to HBase Storage]). In version 3, these KeyValue optionally will include a set of 0 or more tags:
+ excellent introduction to HBase Storage]). In version 3, these KeyValue optionally will include a set of 0 or more tags:
[cols="1,1", frame="all"]
|===
@@ -335,14 +341,14 @@ Within an HFile, HBase cells are stored in data blocks as a sequence of KeyValue
2+| Key bytes (variable)
2+| Value bytes (variable)
| | Tags Length (2 bytes)
-| | Tags bytes (variable)
-|===
+| | Tags bytes (variable)
+|===
If the info block for a given HFile contains an entry for `MAX_TAGS_LEN` each cell will have the length of that cell's tags included, even if that length is zero.
-The actual tags are stored as a sequence of tag length (2 bytes), tag type (1 byte), tag bytes (variable). The format an individual tag's bytes depends on the tag type.
+The actual tags are stored as a sequence of tag length (2 bytes), tag type (1 byte), tag bytes (variable). The format an individual tag's bytes depends on the tag type.
Note that the dependence on the contents of the info block implies that prior to reading any data blocks you must first process a file's info block.
-It also implies that prior to writing a data block you must know if the file's info block will include `MAX_TAGS_LEN`.
+It also implies that prior to writing a data block you must know if the file's info block will include `MAX_TAGS_LEN`.
[[hfilev3.fixedtrailer]]
==== Fixed File Trailer in Version 3
@@ -350,6 +356,6 @@ It also implies that prior to writing a data block you must know if the file's i
The fixed file trailers written with HFile version 3 are always serialized with protocol buffers.
Additionally, it adds an optional field to the version 2 protocol buffer named encryption_key.
If HBase is configured to encrypt HFiles this field will store a data encryption key for this particular HFile, encrypted with the current cluster master key using AES.
-For more information see <<hbase.encryption.server,hbase.encryption.server>>.
+For more information see <<hbase.encryption.server,hbase.encryption.server>>.
:numbered: