You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/03/01 21:05:54 UTC

[GitHub] [hbase] busbey opened a new pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

busbey opened a new pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232
 
 
   * add design doc for original MOB changes as they were when HBase 2.0 came out
   * add design doc for distributed MOB compaction
   * remove configuration and commands no longer relevant after distributed MOB compaction
   * add in discussion of configuration options
   * allow asciimath formulas since we use them in the discussion

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387339374

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -36,22 +36,15 @@ read and write paths are optimized for values smaller than 100KB in size. When
HBase deals with large numbers of objects over this threshold, referred to here
as medium objects, or MOBs, performance is degraded due to write amplification
caused by splits and compactions. When using MOBs, ideally your objects will be between
-100KB and 10MB (see the <<faq>>). HBase ***FIX_VERSION_NUMBER*** adds support
-for better managing large numbers of MOBs while maintaining performance,
-consistency, and low operational overhead. MOB support is provided by the work
-done in link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. To
-take advantage of MOB, you need to use <<hfilev3,HFile version 3>>. Optionally,
+100KB and 10MB (see the <<faq>>). HBase 2 added special internal handling of MOBs
+to maintain performance, consistency, and low operational overhead. MOB support is
+provided by the work done in link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339].
+To take advantage of MOB, you need to use <<hfilev3,HFile version 3>>. Optionally,

Review comment:
Maybe point out that was the original work and evolved by means of HBASE-22749?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387635970

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -468,3 +585,54 @@ $ hdfs dfs -count /hbase/mobdir/data/default/some_table
+
This data is spurious and may be reclaimed. You should sideline it, verify your application’s view
of the table, and then delete it.
+
+=== MOB Upgrade Considerations
+
+Generally, data stored using the MOB feature should transparently continue to work correctly across
+HBase upgrades.
+
+==== Upgrading to a version with the "distributed MOB compaction" feature
+
+Prior to the work in HBASE-22749, "Distributed MOB compactions", HBase had the Master coordinate all
+compaction maintenance of the MOB hfiles. Centralizing management of the MOB data allowed for space
+optimizations but safely coordinating that managemet with Region Servers resulted in edge cases that
+caused data loss (ref link:https://issues.apache.org/jira/browse/HBASE-22075[HBASE-22075]).
+
+Users of the MOB feature upgrading to a version of HBase that includes HBASE-22749 should be aware
+of the following changes:
+
+* The MOB system no longer allows setting "MOB Compaction Policies"
+* The MOB system no longer attempts to group MOB values by the date of the original cell's timestamp
+ according to said compaction policies, daily or otherwise
+* The MOB system no longer needs to track individual cell deletes through the use of special
+ files in the MOB storage area with the suffix `_del`. After upgrading you should sideline these
+ files.
+* Under default configuration the MOB system should take much less time to perform a compaction of
+ MOB stored values. This is a direct consequence of the fact that HBase will place a much larger
+ load on the underlying filesystem when doing compactions of MOB stored values; the additional load
+ should be a multiple on the order of magnitude of number of region servers. I.e. for a cluster
+ with three region servers and two masters the default configuration should have HBase put three
+ times the load on HDFS during major compactions that rewrite MOB data when compared to Master
+ handled MOB compaction; it should also be approximately three times as fast.
+* When the MOB system detects that a table has hfiles with references to MOB data but the reference
+ hfiles do not yet have the needed file level metadata (i.e. from use of the MOB feature prior to
+ HBASE-22749) then it will refuse to archive _any_ MOB hfiles from that table. The normal course of
+ periodic compactions done by Region Servers will update existing hfiles with MOB references, but
+ until a given table has been through the needed compactions operators should expect to see an
+ increased amount of storage used by the MOB feature.
+* Performing a compaction with type "MOB" no longer has special handling to compact specifically the
+ MOB hfiles. Instead it will issue a warning and do a major compaction of the table. Similarly,
+ manually performing a major compaction on a table or region will also handle compacting the MOB
+ stored values for that table or region respectively.

Review comment:
So there's no way to trigger mob compaction only via shell/API? It's always a full region/table compaction?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387339342

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

-=== Configure MOB Compaction Policy
+=== Testing MOB
+
+The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
+the MOB feature. The utility is run as follows:
+[source,bash]
+----
+$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
+ -threshold 1024 \
+ -minMobDataSize 512 \
+ -maxMobDataSize 5120
+----
+
+* `*threshold*` is the threshold at which cells are considered to be MOBs.
+ The default is 1 kB, expressed in bytes.
+* `*minMobDataSize*` is the minimum value for the size of MOB data.
+ The default is 512 B, expressed in bytes.
+* `*maxMobDataSize*` is the maximum value for the size of MOB data.
+ The default is 5 kB, expressed in bytes.
+
+=== MOB architecture
+
+==== Overview
+This section is derived from information found in
+link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. For more information see
+the last version of the design doc created during that work:
+"link:https://github.com/apache/hbase/blob/master/dev-support/design-docs/HBASE-11339%20MOB%20GA%20design.pdf[HBASE-11339 MOB GA design.pdf]".
+
+The MOB feature reduces the overall IO load for configured column families by storing values that
+are larger than the configured threshold outside of the normal regions to avoid splits, merges, and
+most importantly normal compactions.
+
+When a cell is first written to a region it is stored in the WAL and memstore regardless of value
+size. When memstores from a column family configured to use MOB are eventually flushed two hfiles
+are written simultaneously. Cells with a value smaller than the threshold size are written to a
+normal region hfile. Cells with a value larger than the threshold are written into a special MOB
+hfile and also have a MOB reference cell written into the normal region HFile. As the Region Server
+flushes a MOB enabled memstore and closes a given normal region HFile it appends metadata that lists
+each of the special MOB hfiles referenced by the cells within.
+
+MOB reference cells have the same key as the cell they are based on. The value of the reference cell
+is made up of two pieces of metadata: the size of the actual value and the MOB hfile that contains
+the original cell. In addition to any tags originally written to HBase, the reference cell prepends
+two additional tags. The first is a marker tag that says the cell is a MOB reference. This can be
+used later to scan specifically just for reference cells. The second stores the namespace and table
+at the time the MOB hfile is written out. This tag is used to optimize how the MOB system finds
+the underlying value in MOB hfiles after a series of HBase snapshot operations (ref HBASE-12332).
+Note that tags are only available within HBase servers and by default are not sent over RPCs.
+
+All MOB hfiles for a given table are managed within a logical region that does not directly serve
+requests. When these MOB hfiles are created from a flush or MOB compaction they are placed in a
+dedicated mob data area under the hbase root directory specific to the namespace, table, mob
+logical region, and column family. In general that means a path structured like:

-By default, MOB files for one specific day are compacted into one large MOB file.
-To reduce MOB file count more, there are other MOB Compaction policies supported.
+----
+%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
+----

-daily policy - compact MOB Files for one day into one large MOB file (default policy)
-weekly policy - compact MOB Files for one week into one large MOB file
-montly policy - compact MOB Files for one month into one large MOB File
+With default configs, an example table named 'some_table' in the
+default namespace with a MOB enabled column family named 'foo' this HDFS directory would be

-.Configure MOB compaction policy Using HBase Shell
----
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
+----
+
+These MOB hfiles are maintained by special chores in the HBase Master and across the individual
+Region Servers. Specifically those chores take care of enforcing TTLs and compacting them. Note that
+this compaction is primarily a matter of controlling the total number of files in HDFS because our
+operational assumptions for MOB data is that it will seldom update or delete.
+
+When a given MOB hfile is no longer needed as a result of our compaction process then a chore in
+the Master will take care of moving it to the archive just
+like any normal hfile. Because the table's mob region is independent of all the normal regions it
+can coexist with them in the regular archive storage area:

Review comment:
What about if the MOB table is part of a snapshot?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387808067

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+----
+/hbase/archive/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
----

-=== Configure MOB Compaction mergeable threshold
+The same hfile cleaning chores that take care of eventually deleting unneeded archived files from
+normal regions thus also will take care of these MOB hfiles.

-If the size of a mob file is less than this value, it's regarded as a small file and needs to
-be merged in mob compaction. The default value is 1280MB.
+==== MOB compaction
+
+Each time the memstore for a MOB enabled column family performs a flush HBase will write values over
+the MOB threshold into MOB specific hfiles. When normal region compaction occurs the Region Server
+rewrites the normal data files while maintaining references to these MOB files without rewriting
+them. Normal client lookups for MOB values transparently will receive the original values because
+the Region Server internals take care of using the reference data to then pull the value out of a
+specific MOB file. This indirection means that building up a large number of MOB hfiles doesn't
+impact the overall time to retrieve any specific MOB cell. Thus, we need not perform compactions of
+the MOB hfiles nearly as often as normal hfiles. As a result, HBase saves IO by not rewriting MOB
+hfiles as a part of the periodic compactions a Region Server does on its own.
+
+However, if deletes and updates of MOB cells are frequent then this indirection will begin to waste
+space. The only way to stop using the space of a particular MOB hfile is to ensure no cells still
+hold references to it. To do that we need to ensure we have written the current values into a new
+MOB hfile. If our backing filesystem has a limitation on the number of files that can be present, as
+HDFS does, then even if we do not have deletes or updates of MOB cells eventually there will be a
+sufficient number of MOB hfiles that we will need to coallesce them.
+
+Periodically a chore in the master coordinates having the region servers
+perform a special major compaction that also handles rewritting new MOB files. Because this
+rewriting has the advantage of looking across all active cells for the region our several small MOB
+files should end up as a single MOB file per region. The chore defaults to running weekly and can be
+configured by setting `hbase.mob.compaction.chore.period` to the desired period in seconds.

====
[source,xml]
----
<property>
- <name>hbase.mob.compaction.mergeable.threshold</name>
- <value>10000000000</value>
+ <name>hbase.mob.compaction.chore.period</name>
+ <value>2592000</value>
+ <description>Example of changing the chore period from a week to a month.</description>
</property>
----
====

-=== Testing MOB
+By default, the periodic MOB compaction coordination chore will attempt to keep every region
+busy doing compactions in parallel in order to maximize the amount of work done on the cluster.
+If you need to tune the amount of IO this compaction generates on the underlying filesystem, you
+can control how many concurrent region-level compaction requests are allowed by setting
+`hbase.mob.major.compaction.region.batch.size` to an integer number greater than zero. If you set
+the configuration to 0 then you will get the default behavior of attempting to do all regions in
+parallel.

-The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
-the MOB feature. The utility is run as follows:
-[source,bash]
+====
+[source,xml]
----
-$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
- -threshold 1024 \
- -minMobDataSize 512 \
- -maxMobDataSize 5120
+<property>
+ <name>hbase.mob.major.compaction.region.batch.size</name>
+ <value>1</value>
+ <description>Example of switching from "as parallel as possible" to "serially"</description>
+</property>
----
+====

Review comment:
sure. I think that's an issue with master periodic chores generally though, not something specific to this one.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387969408

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
briefly noted this in the compaction section

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] Apache-HBase commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

Apache-HBase commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#issuecomment-594905786
 
 
   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 31s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 33s |  master passed  |
   | +0 :ok: |  refguide  |   4m 55s |  branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m  4s |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  There were no new shellcheck issues.  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +0 :ok: |  refguide  |   4m 51s |  patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 19s |  The patch does not generate ASF License warnings.  |
   |  |   |  22m 26s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/3/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/1232 |
   | Optional Tests | dupname asflicense shellcheck shelldocs refguide |
   | uname | Linux be23edabb736 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1232/out/precommit/personality/provided.sh |
   | git revision | master / b6eefcaeb7 |
   | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/3/artifact/out/branch-site/book.html |
   | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/3/artifact/out/patch-site/book.html |
   | Max. process+thread count | 96 (vs. ulimit of 10000) |
   | modules | C: . U: . |
   | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/3/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387623516

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
> It defaults to running daily.

As long as masters are not restarted every day :)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#issuecomment-593145356
 
 
   take a look when you can @VladRodionov 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387341578

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +323,59 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

To manually compact MOB files, rather than waiting for the
-<<mob.cache.configure,configuration>> to trigger compaction, use the
-`compact` or `major_compact` HBase shell commands. These commands
+periodic chore to trigger compaction, use the
+`major_compact` HBase shell commands. These commands
require the first argument to be the table name, and take a column
-family as the second argument. and take a compaction type as the third argument.
+family as the second argument. If used with a column family that includes MOB data, then
+these operator requests will result in the MOB data being compacted.

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’

Review comment:
Would scripts that have the 'MOB' parameter still work? Just from this doc is not clear.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387809615

Review comment:
yes that's correct. There's no notion of compacting mob hfiles independent of compacting the hfiles that contain the references to those mob values. that's why we no longer have the dataloss issue from HBASE-22075.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387809022

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +323,59 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-When a cell is first written to a region it is stored in the WAL and memstore regardless of value
-size. When memstores from a column family configured to use MOB are eventually flushed two hfiles
-are written simultaneously. Cells with a value smaller than the threshold size are written to a
-normal region hfile. Cells with a value larger than the threshold are written into a special MOB
-hfile and also have a MOB reference cell written into the normal region HFile.
-
-MOB reference cells have the same key as the cell they are based on. The value of the reference cell
-is made up of two pieces of metadata: the size of the actual value and the MOB hfile that contains
-the original cell. In addition to any tags originally written to HBase, the reference cell prepends
-two additional tags. The first is a marker tag that says the cell is a MOB reference. This can be
-used later to scan specifically just for reference cells. The second stores the namespace and table
-at the time the MOB hfile is written out. This tag is used to optimize how the MOB system finds
-the underlying value in MOB hfiles after a series of HBase snapshot operations (ref HBASE-12332).
-Note that tags are only available within HBase servers and by default are not sent over RPCs.
-
-All MOB hfiles for a given table are managed within a logical region that does not directly serve
-requests. When these MOB hfiles are created from a flush or MOB compaction they are placed in a
-dedicated mob data area under the hbase root directory specific to the namespace, table, mob
-logical region, and column family. In general that means a path structured like:
+=== MOB Troubleshooting

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

-With default configs, an example table named 'some_table' in the
-default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
+The MOB cleaner chore ignores all MOB hfiles that were created more recently than an hour prior to
+the start of the chore to ensure we don't miss the reference metadata from the corresponding regular
+hfile. Without this safety check it would be possible for the cleaner chore to see a MOB hfile for
+an in progress flush or compaction and prematurely archive the MOB data. This default buffer should
+be sufficient for normal use.

-----
-/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
-----
+You will need to adjust the tolerance if you use write amplification optimized MOB compaction and
+the combination of your underlying filesystem performance and data shape is such that it could take
+more than an hour to complete major compaction of a single region. For example, if your MOB data is
+distributed such that your largest region adds 80GB of MOB data between compactions that include
+rewriting MOB data and your HDFS cluster is only capable of writing 20MB/s for a single file then
+when performing the optimized compaction the Region Server will take about a minute to write the
+first 1GB MOB hfile and then another hour and seven minutes to write the remaining seventy-nine 1GB
+MOB hfiles before finally committing the new reference hfile at the end of the compaction. Given
+this example, you would need a larger tolerance window.

-These MOB hfiles are maintained by special chores in the HBase Master rather than by any individual
-Region Server. Specifically those chores take care of enforcing TTLs and compacting them. Note that
-this compaction is primarily a matter of controlling the total number of files in HDFS because our
-operational assumptions for MOB data is that it will seldom update or delete.
+You will also need to adjust the tolerance if Region Server flush operations take longer than an
+hour for the two HDFS move operations needed to commit both the MOB hfile and the normal hfile that
+references it. Such a delay should not happen with a normally configured and healthy HDFS and HBase.

-When a given MOB hfile is no longer needed as a result of our compaction process it is archived just
-like any normal hfile. Because the table's mob region is independent of all the normal regions it
-can coexist with them in the regular archive storage area:
+The cleaner's window for "too recent" is controlled by setting `hbase.mob.min.age.archive` to a
+positive integer number of milliseconds.

Review comment:
> This sounds like a very important advice. Could we emphasise the importance of this tuning to avoid data loss using a NOTE] tag?

Where would you suggest such a NOTE go in the current documentation? I could maybe put it in the section on enabling write amplification optimized MOB compaction?

> Also, still thinking on potential operation issues we might need to support, is there any idea of an extra safeguard check that could compare mob compaction time taken with the configured value for hbase.mob.min.age.archive and raise alerts/warnings in case it's higher than the threshold?

We could add that in a follow on jira if you like.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387969265

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +323,59 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’

Review comment:
added a call out with example in the upgrade section

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

esteban commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387341038
 
 

 ##########
 File path: src/main/asciidoc/_chapters/hbase_mob.adoc
 ##########
 @@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
 ----
 ====
 
-=== Configure MOB Compaction Policy
+=== Testing MOB
+
+The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
+the MOB feature. The utility is run as follows:
+[source,bash]
+----
+$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
+            -threshold 1024 \
+            -minMobDataSize 512 \
+            -maxMobDataSize 5120
+----
+
+* `*threshold*` is the threshold at which cells are considered to be MOBs.
+   The default is 1 kB, expressed in bytes.
+* `*minMobDataSize*` is the minimum value for the size of MOB data.
+   The default is 512 B, expressed in bytes.
+* `*maxMobDataSize*` is the maximum value for the size of MOB data.
+   The default is 5 kB, expressed in bytes.
+
+=== MOB architecture
+
+==== Overview
+This section is derived from information found in
+link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. For more information see
+the last version of the design doc created during that work:
+"link:https://github.com/apache/hbase/blob/master/dev-support/design-docs/HBASE-11339%20MOB%20GA%20design.pdf[HBASE-11339 MOB GA design.pdf]".
+
+The MOB feature reduces the overall IO load for configured column families by storing values that
+are larger than the configured threshold outside of the normal regions to avoid splits, merges, and
+most importantly normal compactions.
+
+When a cell is first written to a region it is stored in the WAL and memstore regardless of value
+size. When memstores from a column family configured to use MOB are eventually flushed two hfiles
+are written simultaneously. Cells with a value smaller than the threshold size are written to a
+normal region hfile. Cells with a value larger than the threshold are written into a special MOB
+hfile and also have a MOB reference cell written into the normal region HFile. As the Region Server
+flushes a MOB enabled memstore and closes a given normal region HFile it appends metadata that lists
+each of the special MOB hfiles referenced by the cells within.
+
+MOB reference cells have the same key as the cell they are based on. The value of the reference cell
+is made up of two pieces of metadata: the size of the actual value and the MOB hfile that contains
+the original cell. In addition to any tags originally written to HBase, the reference cell prepends
+two additional tags. The first is a marker tag that says the cell is a MOB reference. This can be
+used later to scan specifically just for reference cells. The second stores the namespace and table
+at the time the MOB hfile is written out. This tag is used to optimize how the MOB system finds
+the underlying value in MOB hfiles after a series of HBase snapshot operations (ref HBASE-12332).
+Note that tags are only available within HBase servers and by default are not sent over RPCs.
+
+All MOB hfiles for a given table are managed within a logical region that does not directly serve
+requests. When these MOB hfiles are created from a flush or MOB compaction they are placed in a
+dedicated mob data area under the hbase root directory specific to the namespace, table, mob
+logical region, and column family. In general that means a path structured like:
 
-By default, MOB files for one specific day are compacted into one large MOB file.
-To reduce MOB file count more, there are other MOB Compaction policies supported.
+----
+%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
+----
 
-daily policy  - compact MOB Files for one day into one large MOB file (default policy)
-weekly policy - compact MOB Files for one week into one large MOB file
-montly policy - compact MOB Files for one  month into one large MOB File
+With default configs, an example table named 'some_table' in the
+default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
 
-.Configure MOB compaction policy Using HBase Shell
 ----
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
+----
+
+These MOB hfiles are maintained by special chores in the HBase Master and across the individual
+Region Servers. Specifically those chores take care of enforcing TTLs and compacting them. Note that
+this compaction is primarily a matter of controlling the total number of files in HDFS because our
+operational assumptions for MOB data is that it will seldom update or delete.
+
+When a given MOB hfile is no longer needed as a result of our compaction process then a chore in
+the Master will take care of moving it to the archive just
+like any normal hfile. Because the table's mob region is independent of all the normal regions it
+can coexist with them in the regular archive storage area:
 
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+----
+/hbase/archive/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
 ----
 
-=== Configure MOB Compaction mergeable threshold
+The same hfile cleaning chores that take care of eventually deleting unneeded archived files from
+normal regions thus also will take care of these MOB hfiles.
 
-If the size of a mob file is less than this value, it's regarded as a small file and needs to
-be merged in mob compaction. The default value is 1280MB.
+==== MOB compaction
+
+Each time the memstore for a MOB enabled column family performs a flush HBase will write values over
+the MOB threshold into MOB specific hfiles. When normal region compaction occurs the Region Server
+rewrites the normal data files while maintaining references to these MOB files without rewriting
+them. Normal client lookups for MOB values transparently will receive the original values because
+the Region Server internals take care of using the reference data to then pull the value out of a
+specific MOB file. This indirection means that building up a large number of MOB hfiles doesn't
+impact the overall time to retrieve any specific MOB cell. Thus, we need not perform compactions of
+the MOB hfiles nearly as often as normal hfiles. As a result, HBase saves IO by not rewriting MOB
+hfiles as a part of the periodic compactions a Region Server does on its own.
+
+However, if deletes and updates of MOB cells are frequent then this indirection will begin to waste
+space. The only way to stop using the space of a particular MOB hfile is to ensure no cells still
+hold references to it. To do that we need to ensure we have written the current values into a new
+MOB hfile. If our backing filesystem has a limitation on the number of files that can be present, as
+HDFS does, then even if we do not have deletes or updates of MOB cells eventually there will be a
+sufficient number of MOB hfiles that we will need to coallesce them.
+
+Periodically a chore in the master coordinates having the region servers
+perform a special major compaction that also handles rewritting new MOB files. Because this
+rewriting has the advantage of looking across all active cells for the region our several small MOB
+files should end up as a single MOB file per region. The chore defaults to running weekly and can be
+configured by setting `hbase.mob.compaction.chore.period` to the desired period in seconds.
 
 ====
 [source,xml]
 ----
 <property>
-    <name>hbase.mob.compaction.mergeable.threshold</name>
-    <value>10000000000</value>
+    <name>hbase.mob.compaction.chore.period</name>
+    <value>2592000</value>
+    <description>Example of changing the chore period from a week to a month.</description>
 </property>
 ----
 ====
 
-=== Testing MOB
+By default, the periodic MOB compaction coordination chore will attempt to keep every region
+busy doing compactions in parallel in order to maximize the amount of work done on the cluster.
+If you need to tune the amount of IO this compaction generates on the underlying filesystem, you
+can control how many concurrent region-level compaction requests are allowed by setting
+`hbase.mob.major.compaction.region.batch.size` to an integer number greater than zero. If you set
+the configuration to 0 then you will get the default behavior of attempting to do all regions in
+parallel.
 
-The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
-the MOB feature. The utility is run as follows:
-[source,bash]
+====
+[source,xml]
 ----
-$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
-            -threshold 1024 \
-            -minMobDataSize 512 \
-            -maxMobDataSize 5120
+<property>
+    <name>hbase.mob.major.compaction.region.batch.size</name>
+    <value>1</value>
+    <description>Example of switching from "as parallel as possible" to "serially"</description>
+</property>
 ----
+====
 
-* `*threshold*` is the threshold at which cells are considered to be MOBs.
-   The default is 1 kB, expressed in bytes.
-* `*minMobDataSize*` is the minimum value for the size of MOB data.
-   The default is 512 B, expressed in bytes.
-* `*maxMobDataSize*` is the maximum value for the size of MOB data.
-   The default is 5 kB, expressed in bytes.
+==== MOB file archiving
+
+Eventually we will have MOB hfiles that are no longer needed. Either clients will overwrite the
+value or a MOB-rewriting compaction will store a reference to a newer larger MOB hfile. Because any
+given MOB cell could have originally been written either in the current region or in a parent region
+that existed at some prior point in time, individual Region Servers do not decide when it is time
+to archive MOB hfiles. Instead a periodic chore in the Master evaluates MOB hfiles for archiving.
+
+A MOB HFile will be subject to archiving under any of the following conditions:
+
+* Any MOB HFile older than the column family's TTL
+* Any MOB HFile older than a "too recent" threshold with no references to it from the regular hfiles
+  for all regions in a column family
+
+To determine if a MOB HFile meets the second criteria the chore extracts metadata from the regular
+HFiles for each MOB enabled column family for a given table. That metadata enumerates the complete
+set of MOB HFiles needed to satisfy the references stored in the normal HFile area.
+
+The period of the cleaner chore can be configued by setting `hbase.master.mob.cleaner.period` to a
+positive integer number of seconds. It defaults to running daily. You should not need to tune it
+unless you have a very aggressive TTL or a very high rate of MOB updates with a correspondingly
+high rate of non-MOB compactions.
+
+=== MOB Optimization Tasks
+
+==== Further limiting write amplification
+
+If your MOB workload has few to no updates or deletes then you can opt-in to MOB compactions that
+optimize for limiting the amount of write amplification. It acheives this by setting a
+size threshold to ignore MOB files during the compaction process. When a given region goes
+through MOB compaction it will evaluate the size of the MOB file that currently holds the actual
+value and skip rewriting the value if that file is over threshold.
+
+The bound of write amplification in this mode can be approximated as
+stem:["Write Amplification" = log_K(M/S)] where *K* is the number of files in compaction
+selection, *M* is the configurable threshold for MOB files size, and *S* is the minmum size of
+memstore flushes that create MOB files in the first place. For example given 5 files picked up per
+compaction, a threshold of 1 GB, and a flush size of 10MB the write amplification will be
+stem:[log_5((1GB)/(10MB)) = log_5(100) = 2.86].
+
+If we are using an underlying filesystem with a limitation on the number of files, such as HDFS,
+and we know our expected data set size we can choose our maximum file size in order to approach
+this limit but stay within it in order to minimize write amplification. For example, if we expect to
+store a petabyte and we have a conservative limitation of a million files in our HDFS instance, then
+stem:[(1PB)/(1M) = 1GB] gives us a target limitation of a gigabyte per MOB file.
+
 
 Review comment:
   Can distributed MOB compaction use PressureAwareCompactionThroughputController settings to even further limit this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387807870

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
design doc I think.

the reference cells are written into the normal hfile storage area as a part of the compaction process.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387271665

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +320,51 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-These commands are also available via `Admin.compact` and
-`Admin.majorCompact` methods.
-
-=== MOB architecture
-
-This section is derived from information found in
-link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. For more information see
-the attachment on that issue
-"link:https://issues.apache.org/jira/secure/attachment/12724468/HBase%20MOB%20Design-v5.pdf[Base MOB Design-v5.pdf]".
-
-==== Overview
-The MOB feature reduces the overall IO load for configured column families by storing values that
-are larger than the configured threshold outside of the normal regions to avoid splits, merges, and
-most importantly normal compactions.
-
-When a cell is first written to a region it is stored in the WAL and memstore regardless of value
-size. When memstores from a column family configured to use MOB are eventually flushed two hfiles
-are written simultaneously. Cells with a value smaller than the threshold size are written to a
-normal region hfile. Cells with a value larger than the threshold are written into a special MOB
-hfile and also have a MOB reference cell written into the normal region HFile.
-
-MOB reference cells have the same key as the cell they are based on. The value of the reference cell
-is made up of two pieces of metadata: the size of the actual value and the MOB hfile that contains
-the original cell. In addition to any tags originally written to HBase, the reference cell prepends
-two additional tags. The first is a marker tag that says the cell is a MOB reference. This can be
-used later to scan specifically just for reference cells. The second stores the namespace and table
-at the time the MOB hfile is written out. This tag is used to optimize how the MOB system finds
-the underlying value in MOB hfiles after a series of HBase snapshot operations (ref HBASE-12332).
-Note that tags are only available within HBase servers and by default are not sent over RPCs.
+This same request can be made via the `Admin.majorCompact` Java API.

-All MOB hfiles for a given table are managed within a logical region that does not directly serve
-requests. When these MOB hfiles are created from a flush or MOB compaction they are placed in a
-dedicated mob data area under the hbase root directory specific to the namespace, table, mob
-logical region, and column family. In general that means a path structured like:
+=== MOB Troubleshooting

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

-With default configs, an example table named 'some_table' in the
-default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
+The MOB cleaner chore ignores all MOB hfiles that were created more recently than an hour prior to
+the start of the shore to ensure we don't miss the reference metadata from teh corresponding regular
+hfile. Without this safety check it would be possible for the cleaner chore to see a MOB hfile for
+an in progress flush or compaction and prematurely archive the MOB data. This default buffer should
+be sufficient for normal use.

-----
-/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
-----
-
-These MOB hfiles are maintained by special chores in the HBase Master rather than by any individual
-Region Server. Specifically those chores take care of enforcing TTLs and compacting them. Note that
-this compaction is primarily a matter of controlling the total number of files in HDFS because our
-operational assumptions for MOB data is that it will seldom update or delete.
+You will need to adjust the tolerance if it takes longer than an hour for the two HDFS move

Review comment:
ah. reading this paragraph again it's not clear that's what I'm talking about. let me try an update.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387807250

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
this was a reformatting of those lines because I needed to add the bit about putting metadata in the reference hfiles on flush.

I think the original lines are unclear. We put the mob hfiles in the archive area specifically under the same shadow region name that gets used for the mob area. that's why it can be "next to" the other hfiles in the archive area; there's a special region name that no other region ends up with.

I would rather not rewrite the architecture section explanation of why mob hfiles are in a different part of HDFS in this issue because it's a deep rabbit hole that's only tangentially related. Depending on free time and some additional testing I'd like to poke at I might have a follow on that would justify explaining those intricacies.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387279480

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +320,51 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

-With default configs, an example table named 'some_table' in the
-default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
+The MOB cleaner chore ignores all MOB hfiles that were created more recently than an hour prior to
+the start of the shore to ensure we don't miss the reference metadata from teh corresponding regular
+hfile. Without this safety check it would be possible for the cleaner chore to see a MOB hfile for
+an in progress flush or compaction and prematurely archive the MOB data. This default buffer should
+be sufficient for normal use.

Review comment:
Take a look at the section with the rewrite and let me know if it's clearer, please.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387969120

Review comment:
added a note about this in the architecture header

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387620651

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
Does it update the "mob ref cells" (say, performing a new put for the ref cell) with the new mobfile value? Or it does not need to, because ref cells just point to the related mobdir? Would such extra info worth adding on this section, or just point folks to the design doc enough?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387262356

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +320,51 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

-With default configs, an example table named 'some_table' in the
-default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
+The MOB cleaner chore ignores all MOB hfiles that were created more recently than an hour prior to
+the start of the shore to ensure we don't miss the reference metadata from teh corresponding regular
+hfile. Without this safety check it would be possible for the cleaner chore to see a MOB hfile for
+an in progress flush or compaction and prematurely archive the MOB data. This default buffer should
+be sufficient for normal use.

Review comment:
the paragraph doesn't talk about the time to do things on the master in the cleaning chore. It's talking about the amount of time it takes the region server that's doing a flush or compaction.

for example, consider if something is *very* wrong with HDFS such that the RS doing a flush commits the mob file and then pauses for > 1 hour before committing the reference hfile. if the master did it's "min age" calculation an hour after the mob file commit and it finished iterating over the reference files prior to the commit of the reference hfile, then the cleaner chore would believe it is fine to delete the mob hfile.

similarly, if the compaction process commits a mob hfile (due to the size limitation) and then spends another 3 hours writing other mob hfiles before finally closing out and committing the reference hfile, then the master cleaner chore could have the timing work out so that some of the early mob hfiles are flagged as fine to delete.

It's definitely edge case stuff. that's why it's in the troubleshooting section.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387271665

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +320,51 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

-With default configs, an example table named 'some_table' in the
-default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
+The MOB cleaner chore ignores all MOB hfiles that were created more recently than an hour prior to
+the start of the shore to ensure we don't miss the reference metadata from teh corresponding regular
+hfile. Without this safety check it would be possible for the cleaner chore to see a MOB hfile for
+an in progress flush or compaction and prematurely archive the MOB data. This default buffer should
+be sufficient for normal use.

Review comment:
ah. reading this paragraph against it's not clear that's what I'm talking about. let me try an update.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387921388

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +323,59 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’

Review comment:
All the shell, java, and rpc stuff is still in place for talking about 'MOB' type compactions is still present. so it'll work just do something different.

I'll try to make that clear in the upgrade consideration section.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387969532

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +323,59 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

Review comment:
added a note to the optimized mode

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387350034
 
 

 ##########
 File path: src/main/asciidoc/_chapters/hbase_mob.adoc
 ##########
 @@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
 ----
 ====
 
-=== Configure MOB Compaction Policy
+=== Testing MOB
+
+The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
+the MOB feature. The utility is run as follows:
+[source,bash]
+----
+$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
+            -threshold 1024 \
+            -minMobDataSize 512 \
+            -maxMobDataSize 5120
+----
+
+* `*threshold*` is the threshold at which cells are considered to be MOBs.
+   The default is 1 kB, expressed in bytes.
+* `*minMobDataSize*` is the minimum value for the size of MOB data.
+   The default is 512 B, expressed in bytes.
+* `*maxMobDataSize*` is the maximum value for the size of MOB data.
+   The default is 5 kB, expressed in bytes.
+
+=== MOB architecture
+
+==== Overview
+This section is derived from information found in
+link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. For more information see
+the last version of the design doc created during that work:
+"link:https://github.com/apache/hbase/blob/master/dev-support/design-docs/HBASE-11339%20MOB%20GA%20design.pdf[HBASE-11339 MOB GA design.pdf]".
+
+The MOB feature reduces the overall IO load for configured column families by storing values that
+are larger than the configured threshold outside of the normal regions to avoid splits, merges, and
+most importantly normal compactions.
+
+When a cell is first written to a region it is stored in the WAL and memstore regardless of value
+size. When memstores from a column family configured to use MOB are eventually flushed two hfiles
+are written simultaneously. Cells with a value smaller than the threshold size are written to a
+normal region hfile. Cells with a value larger than the threshold are written into a special MOB
+hfile and also have a MOB reference cell written into the normal region HFile. As the Region Server
+flushes a MOB enabled memstore and closes a given normal region HFile it appends metadata that lists
+each of the special MOB hfiles referenced by the cells within.
+
+MOB reference cells have the same key as the cell they are based on. The value of the reference cell
+is made up of two pieces of metadata: the size of the actual value and the MOB hfile that contains
+the original cell. In addition to any tags originally written to HBase, the reference cell prepends
+two additional tags. The first is a marker tag that says the cell is a MOB reference. This can be
+used later to scan specifically just for reference cells. The second stores the namespace and table
+at the time the MOB hfile is written out. This tag is used to optimize how the MOB system finds
+the underlying value in MOB hfiles after a series of HBase snapshot operations (ref HBASE-12332).
+Note that tags are only available within HBase servers and by default are not sent over RPCs.
+
+All MOB hfiles for a given table are managed within a logical region that does not directly serve
+requests. When these MOB hfiles are created from a flush or MOB compaction they are placed in a
+dedicated mob data area under the hbase root directory specific to the namespace, table, mob
+logical region, and column family. In general that means a path structured like:
 
-By default, MOB files for one specific day are compacted into one large MOB file.
-To reduce MOB file count more, there are other MOB Compaction policies supported.
+----
+%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
+----
 
-daily policy  - compact MOB Files for one day into one large MOB file (default policy)
-weekly policy - compact MOB Files for one week into one large MOB file
-montly policy - compact MOB Files for one  month into one large MOB File
+With default configs, an example table named 'some_table' in the
+default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
 
-.Configure MOB compaction policy Using HBase Shell
 ----
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
+----
+
+These MOB hfiles are maintained by special chores in the HBase Master and across the individual
+Region Servers. Specifically those chores take care of enforcing TTLs and compacting them. Note that
+this compaction is primarily a matter of controlling the total number of files in HDFS because our
+operational assumptions for MOB data is that it will seldom update or delete.
+
+When a given MOB hfile is no longer needed as a result of our compaction process then a chore in
+the Master will take care of moving it to the archive just
+like any normal hfile. Because the table's mob region is independent of all the normal regions it
+can coexist with them in the regular archive storage area:
 
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+----
+/hbase/archive/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
 ----
 
-=== Configure MOB Compaction mergeable threshold
+The same hfile cleaning chores that take care of eventually deleting unneeded archived files from
+normal regions thus also will take care of these MOB hfiles.
 
-If the size of a mob file is less than this value, it's regarded as a small file and needs to
-be merged in mob compaction. The default value is 1280MB.
+==== MOB compaction
+
+Each time the memstore for a MOB enabled column family performs a flush HBase will write values over
+the MOB threshold into MOB specific hfiles. When normal region compaction occurs the Region Server
+rewrites the normal data files while maintaining references to these MOB files without rewriting
+them. Normal client lookups for MOB values transparently will receive the original values because
+the Region Server internals take care of using the reference data to then pull the value out of a
+specific MOB file. This indirection means that building up a large number of MOB hfiles doesn't
+impact the overall time to retrieve any specific MOB cell. Thus, we need not perform compactions of
+the MOB hfiles nearly as often as normal hfiles. As a result, HBase saves IO by not rewriting MOB
+hfiles as a part of the periodic compactions a Region Server does on its own.
+
+However, if deletes and updates of MOB cells are frequent then this indirection will begin to waste
+space. The only way to stop using the space of a particular MOB hfile is to ensure no cells still
+hold references to it. To do that we need to ensure we have written the current values into a new
+MOB hfile. If our backing filesystem has a limitation on the number of files that can be present, as
+HDFS does, then even if we do not have deletes or updates of MOB cells eventually there will be a
+sufficient number of MOB hfiles that we will need to coallesce them.
+
+Periodically a chore in the master coordinates having the region servers
+perform a special major compaction that also handles rewritting new MOB files. Because this
+rewriting has the advantage of looking across all active cells for the region our several small MOB
+files should end up as a single MOB file per region. The chore defaults to running weekly and can be
+configured by setting `hbase.mob.compaction.chore.period` to the desired period in seconds.
 
 ====
 [source,xml]
 ----
 <property>
-    <name>hbase.mob.compaction.mergeable.threshold</name>
-    <value>10000000000</value>
+    <name>hbase.mob.compaction.chore.period</name>
+    <value>2592000</value>
+    <description>Example of changing the chore period from a week to a month.</description>
 </property>
 ----
 ====
 
-=== Testing MOB
+By default, the periodic MOB compaction coordination chore will attempt to keep every region
+busy doing compactions in parallel in order to maximize the amount of work done on the cluster.
+If you need to tune the amount of IO this compaction generates on the underlying filesystem, you
+can control how many concurrent region-level compaction requests are allowed by setting
+`hbase.mob.major.compaction.region.batch.size` to an integer number greater than zero. If you set
+the configuration to 0 then you will get the default behavior of attempting to do all regions in
+parallel.
 
-The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
-the MOB feature. The utility is run as follows:
-[source,bash]
+====
+[source,xml]
 ----
-$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
-            -threshold 1024 \
-            -minMobDataSize 512 \
-            -maxMobDataSize 5120
+<property>
+    <name>hbase.mob.major.compaction.region.batch.size</name>
+    <value>1</value>
+    <description>Example of switching from "as parallel as possible" to "serially"</description>
+</property>
 ----
+====
 
-* `*threshold*` is the threshold at which cells are considered to be MOBs.
-   The default is 1 kB, expressed in bytes.
-* `*minMobDataSize*` is the minimum value for the size of MOB data.
-   The default is 512 B, expressed in bytes.
-* `*maxMobDataSize*` is the maximum value for the size of MOB data.
-   The default is 5 kB, expressed in bytes.
+==== MOB file archiving
+
+Eventually we will have MOB hfiles that are no longer needed. Either clients will overwrite the
+value or a MOB-rewriting compaction will store a reference to a newer larger MOB hfile. Because any
+given MOB cell could have originally been written either in the current region or in a parent region
+that existed at some prior point in time, individual Region Servers do not decide when it is time
+to archive MOB hfiles. Instead a periodic chore in the Master evaluates MOB hfiles for archiving.
+
+A MOB HFile will be subject to archiving under any of the following conditions:
+
+* Any MOB HFile older than the column family's TTL
+* Any MOB HFile older than a "too recent" threshold with no references to it from the regular hfiles
+  for all regions in a column family
+
+To determine if a MOB HFile meets the second criteria the chore extracts metadata from the regular
+HFiles for each MOB enabled column family for a given table. That metadata enumerates the complete
+set of MOB HFiles needed to satisfy the references stored in the normal HFile area.
+
+The period of the cleaner chore can be configued by setting `hbase.master.mob.cleaner.period` to a
+positive integer number of seconds. It defaults to running daily. You should not need to tune it
+unless you have a very aggressive TTL or a very high rate of MOB updates with a correspondingly
+high rate of non-MOB compactions.
+
+=== MOB Optimization Tasks
+
+==== Further limiting write amplification
+
+If your MOB workload has few to no updates or deletes then you can opt-in to MOB compactions that
+optimize for limiting the amount of write amplification. It acheives this by setting a
+size threshold to ignore MOB files during the compaction process. When a given region goes
+through MOB compaction it will evaluate the size of the MOB file that currently holds the actual
+value and skip rewriting the value if that file is over threshold.
+
+The bound of write amplification in this mode can be approximated as
+stem:["Write Amplification" = log_K(M/S)] where *K* is the number of files in compaction
+selection, *M* is the configurable threshold for MOB files size, and *S* is the minmum size of
+memstore flushes that create MOB files in the first place. For example given 5 files picked up per
+compaction, a threshold of 1 GB, and a flush size of 10MB the write amplification will be
+stem:[log_5((1GB)/(10MB)) = log_5(100) = 2.86].
+
+If we are using an underlying filesystem with a limitation on the number of files, such as HDFS,
+and we know our expected data set size we can choose our maximum file size in order to approach
+this limit but stay within it in order to minimize write amplification. For example, if we expect to
+store a petabyte and we have a conservative limitation of a million files in our HDFS instance, then
+stem:[(1PB)/(1M) = 1GB] gives us a target limitation of a gigabyte per MOB file.
+
 
 Review comment:
   The mob compaction happens as a part of region compaction in the region server. Let me check on this specific interaction.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] Apache-HBase commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

Apache-HBase commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#issuecomment-593148751
 
 
   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  5s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files found.  |
   | +0 :ok: |  shelldocs  |   0m  1s |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   6m  8s |  master passed  |
   | +0 :ok: |  refguide  |   5m 36s |  branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 34s |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  There were no new shellcheck issues.  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +0 :ok: |  refguide  |   5m 28s |  patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 18s |  The patch does not generate ASF License warnings.  |
   |  |   |  25m 20s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/1232 |
   | Optional Tests | dupname asflicense shellcheck shelldocs refguide |
   | uname | Linux e7ec58e1a4d3 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1232/out/precommit/personality/provided.sh |
   | git revision | master / 48a3ccf523 |
   | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/1/artifact/out/branch-site/book.html |
   | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/1/artifact/out/patch-site/book.html |
   | Max. process+thread count | 86 (vs. ulimit of 10000) |
   | modules | C: . U: . |
   | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/1/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387350344

Review comment:
How about a link to the design doc in the architecture section on compaction?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387232649

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,209 @@ hcd.setMobThreshold(102400L);
----
====

-The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
-the MOB feature. The utility is run as follows:
-[source,bash]
+====
+[source,xml]
----
-$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
- -threshold 1024 \
- -minMobDataSize 512 \
- -maxMobDataSize 5120
+<property>
+ <name>hbase.mob.major.compcation.region.batch.size</name>
+ <value>1</value>
+ <description>Example of switching from "as parallel as possible" to "serially"</description>
+</property>
----
+====

Review comment:
Value is inn secs. Default is 86400. Needs to be added.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387349796

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +323,59 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’

Review comment:
Let me confirm if it's ignored or if it errors.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387231459

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,209 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
0 - means no limits, should be mentioned.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387632027

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +323,59 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

Review comment:
This sounds like a very important advice. Could we emphasise the importance of this tuning to avoid data loss using a NOTE] tag?

Also, still thinking on potential operation issues we might need to support, is there any idea of an extra safeguard check that could compare mob compaction time taken with the configured value for _hbase.mob.min.age.archive_ and raise alerts/warnings in case it's higher than the threshold?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] Apache-HBase commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

Apache-HBase commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#issuecomment-594167105
 
 
   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 35s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 41s |  master passed  |
   | +0 :ok: |  refguide  |   4m 56s |  branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m  5s |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  1s |  There were no new shellcheck issues.  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +0 :ok: |  refguide  |   5m  1s |  patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 20s |  The patch does not generate ASF License warnings.  |
   |  |   |  22m 53s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/1232 |
   | Optional Tests | dupname asflicense shellcheck shelldocs refguide |
   | uname | Linux 4b374076f7d7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1232/out/precommit/personality/provided.sh |
   | git revision | master / f411e395d3 |
   | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/2/artifact/out/branch-site/book.html |
   | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/2/artifact/out/patch-site/book.html |
   | Max. process+thread count | 76 (vs. ulimit of 10000) |
   | modules | C: . U: . |
   | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1232/2/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387237173

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +320,51 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

-With default configs, an example table named 'some_table' in the
-default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
+The MOB cleaner chore ignores all MOB hfiles that were created more recently than an hour prior to
+the start of the shore to ensure we don't miss the reference metadata from teh corresponding regular
+hfile. Without this safety check it would be possible for the cleaner chore to see a MOB hfile for
+an in progress flush or compaction and prematurely archive the MOB data. This default buffer should
+be sufficient for normal use.

Review comment:
The algorithm is currently slightly different. When compaction chore starts I record start time = startTime, then calculate

maxCreationTimeToArchive = startTime - minAgeToArchive

so, obviously, 1h lagging is more than enough to avoid any possible race conditions and now we do not depend on compaction chore running time.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387230719
 
 

 ##########
 File path: src/main/asciidoc/_chapters/hbase_mob.adoc
 ##########
 @@ -79,63 +72,209 @@ hcd.setMobThreshold(102400L);
 ----
 ====
 
-=== Configure MOB Compaction Policy
+=== Testing MOB
+
+The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
+the MOB feature. The utility is run as follows:
+[source,bash]
+----
+$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
+            -threshold 1024 \
+            -minMobDataSize 512 \
+            -maxMobDataSize 5120
+----
+
+* `*threshold*` is the threshold at which cells are considered to be MOBs.
+   The default is 1 kB, expressed in bytes.
+* `*minMobDataSize*` is the minimum value for the size of MOB data.
+   The default is 512 B, expressed in bytes.
+* `*maxMobDataSize*` is the maximum value for the size of MOB data.
+   The default is 5 kB, expressed in bytes.
+
+=== MOB architecture
+
+==== Overview
+This section is derived from information found in
+link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. For more information see
+the last version of the design doc created during that work:
+"link:https://github.com/apache/hbase/blob/master/dev-support/design-docs/HBASE-11339%20MOB%20GA%20design.pdf[HBASE-11339 MOB GA design.pdf]".
+
+The MOB feature reduces the overall IO load for configured column families by storing values that
+are larger than the configured threshold outside of the normal regions to avoid splits, merges, and
+most importantly normal compactions.
+
+When a cell is first written to a region it is stored in the WAL and memstore regardless of value
+size. When memstores from a column family configured to use MOB are eventually flushed two hfiles
+are written simultaneously. Cells with a value smaller than the threshold size are written to a
+normal region hfile. Cells with a value larger than the threshold are written into a special MOB
+hfile and also have a MOB reference cell written into the normal region HFile. As the Region Server
+flushes a MOB enabled memstore and closes a given normal region HFile it appends metadata that lists
+each of the special MOB hfiles referenced by the cells within.
+
+MOB reference cells have the same key as the cell they are based on. The value of the reference cell
+is made up of two pieces of metadata: the size of the actual value and the MOB hfile that contains
+the original cell. In addition to any tags originally written to HBase, the reference cell prepends
+two additional tags. The first is a marker tag that says the cell is a MOB reference. This can be
+used later to scan specifically just for reference cells. The second stores the namespace and table
+at the time the MOB hfile is written out. This tag is used to optimize how the MOB system finds
+the underlying value in MOB hfiles after a series of HBase snapshot operations (ref HBASE-12332).
+Note that tags are only available within HBase servers and by default are not sent over RPCs.
+
+All MOB hfiles for a given table are managed within a logical region that does not directly serve
+requests. When these MOB hfiles are created from a flush or MOB compaction they are placed in a
+dedicated mob data area under the hbase root directory specific to the namespace, table, mob
+logical region, and column family. In general that means a path structured like:
 
-By default, MOB files for one specific day are compacted into one large MOB file.
-To reduce MOB file count more, there are other MOB Compaction policies supported.
+----
+%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
+----
 
-daily policy  - compact MOB Files for one day into one large MOB file (default policy)
-weekly policy - compact MOB Files for one week into one large MOB file
-montly policy - compact MOB Files for one  month into one large MOB File
+With default configs, an example table named 'some_table' in the
+default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
 
-.Configure MOB compaction policy Using HBase Shell
 ----
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
+----
+
+These MOB hfiles are maintained by special chores in the HBase Master and across the individual
+Region Servers. Specifically those chores take care of enforcing TTLs and compacting them. Note that
+this compaction is primarily a matter of controlling the total number of files in HDFS because our
+operational assumptions for MOB data is that it will seldom update or delete.
+
+When a given MOB hfile is no longer needed as a result of our compaction process then a chore in
+the Master will take care of moving it to the archive just
+like any normal hfile. Because the table's mob region is independent of all the normal regions it
+can coexist with them in the regular archive storage area:
 
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
-hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
+----
+/hbase/archive/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
 ----
 
-=== Configure MOB Compaction mergeable threshold
+The same hfile cleaning chores that take care of eventually deleting unneeded archived files from
+normal regions thus also will take care of these MOB hfiles.
 
-If the size of a mob file is less than this value, it's regarded as a small file and needs to
-be merged in mob compaction. The default value is 1280MB.
+==== MOB compaction
+
+Each time the memstore for a MOB enabled column family performs a flush HBase will write values over
+the MOB threshold into MOB specific hfiles. When normal region compaction occurs the Region Server
+rewrites the normal data files while maintaining references to these MOB files without rewriting
+them. Normal client lookups for MOB values transparently will receive the original values because
+the Region Server internals take care of using the reference data to then pull the value out of a
+specific MOB file. This indirection means that building up a large number of MOB hfiles doesn't
+impact the overall time to retrieve any specific MOB cell. Thus, we need not perform compactions of
+the MOB hfiles nearly as often as normal hfiles. As a result, HBase saves IO by not rewriting MOB
+hfiles as a part of the periodic compactions a Region Server does on its own.
+
+However, if deletes and updates of MOB cells are frequent then this indirection will begin to waste
+space. The only way to stop using the space of a particular MOB hfile is to ensure no cells still
+hold references to it. To do that we need to ensure we have written the current values into a new
+MOB hfile. If our backing filesystem has a limitation on the number of files that can be present, as
+HDFS does, then even if we do not have deletes or updates of MOB cells eventually there will be a
+sufficient number of MOB hfiles that we will need to coallesce them.
+
+Periodically a chore in the master coordinates having the region servers
+perform a special major compaction that also handles rewritting new MOB files. Because this
+rewriting has the advantage of looking across all active cells for the region our several small MOB
+files should end up as a single MOB file per region. The chore defaults to running weekly and can be
+configured by setting `hbase.mob.compaction.chore.period` to the desired period in seconds.
 
 ====
 [source,xml]
 ----
 <property>
-    <name>hbase.mob.compaction.mergeable.threshold</name>
-    <value>10000000000</value>
+    <name>hbase.mob.compaction.chore.period</name>
+    <value>2592000</value>
+    <description>Example of changing the chore period from a week to a month.</description>
 </property>
 ----
 ====
 
-=== Testing MOB
+By default, the periodic MOB compaction coordination chore will attempt to keep every region
+busy doing compactions in parallel in order to maximize the amount of work done on the cluster.
+If you need to tune the amount of IO this compaction generates on the underlying filesystem, you
+can control how many concurrent region-level compaction requests are allowed by setting
+`hbase.mob.major.compaction.region.batch.size` to an integer number greater than zero.
 
-The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
-the MOB feature. The utility is run as follows:
-[source,bash]
+====
+[source,xml]
 ----
-$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
-            -threshold 1024 \
-            -minMobDataSize 512 \
-            -maxMobDataSize 5120
+<property>
+    <name>hbase.mob.major.compcation.region.batch.size</name>
 
 Review comment:
   Fix misspelling please. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] asfgit closed pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

asfgit closed pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

wchevreuil commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387612953

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
We say that mobfiles need to go to their own dir under "mobir", when those are still referenced by "refcells" in normal files, then we say we can put mobfiles together with normal files in the archive dir. Should we briefly explain why we can't place them together in the normal region/cf dir? My assumption is that it would complicate further current region house keeping logic such as splits/merges, minor compactions, normalizer, to name a few.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387969000

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
added a note about this

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387919494

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

-* `*threshold*` is the threshold at which cells are considered to be MOBs.
- The default is 1 kB, expressed in bytes.
-* `*minMobDataSize*` is the minimum value for the size of MOB data.
- The default is 512 B, expressed in bytes.
-* `*maxMobDataSize*` is the maximum value for the size of MOB data.
- The default is 5 kB, expressed in bytes.
+==== MOB file archiving
+
+Eventually we will have MOB hfiles that are no longer needed. Either clients will overwrite the
+value or a MOB-rewriting compaction will store a reference to a newer larger MOB hfile. Because any
+given MOB cell could have originally been written either in the current region or in a parent region
+that existed at some prior point in time, individual Region Servers do not decide when it is time
+to archive MOB hfiles. Instead a periodic chore in the Master evaluates MOB hfiles for archiving.
+
+A MOB HFile will be subject to archiving under any of the following conditions:
+
+* Any MOB HFile older than the column family's TTL
+* Any MOB HFile older than a "too recent" threshold with no references to it from the regular hfiles
+ for all regions in a column family
+
+To determine if a MOB HFile meets the second criteria the chore extracts metadata from the regular
+HFiles for each MOB enabled column family for a given table. That metadata enumerates the complete
+set of MOB HFiles needed to satisfy the references stored in the normal HFile area.
+
+The period of the cleaner chore can be configued by setting `hbase.master.mob.cleaner.period` to a
+positive integer number of seconds. It defaults to running daily. You should not need to tune it
+unless you have a very aggressive TTL or a very high rate of MOB updates with a correspondingly
+high rate of non-MOB compactions.
+
+=== MOB Optimization Tasks
+
+==== Further limiting write amplification
+
+If your MOB workload has few to no updates or deletes then you can opt-in to MOB compactions that
+optimize for limiting the amount of write amplification. It acheives this by setting a
+size threshold to ignore MOB files during the compaction process. When a given region goes
+through MOB compaction it will evaluate the size of the MOB file that currently holds the actual
+value and skip rewriting the value if that file is over threshold.
+
+The bound of write amplification in this mode can be approximated as
+stem:["Write Amplification" = log_K(M/S)] where *K* is the number of files in compaction
+selection, *M* is the configurable threshold for MOB files size, and *S* is the minmum size of
+memstore flushes that create MOB files in the first place. For example given 5 files picked up per
+compaction, a threshold of 1 GB, and a flush size of 10MB the write amplification will be
+stem:[log_5((1GB)/(10MB)) = log_5(100) = 2.86].
+
+If we are using an underlying filesystem with a limitation on the number of files, such as HDFS,
+and we know our expected data set size we can choose our maximum file size in order to approach
+this limit but stay within it in order to minimize write amplification. For example, if we expect to
+store a petabyte and we have a conservative limitation of a million files in our HDFS instance, then
+stem:[(1PB)/(1M) = 1GB] gives us a target limitation of a gigabyte per MOB file.
+

Review comment:
the compactor for MOB enabled column families still uses the throughput controllers, so yes the PressureAwareCompactionThroughputController could be used. However, for cells over the mob size threshold the compactor for MOB enabled column families only tells the throughput controller about the size of the reference cell. is that worth calling out in the section on mob compaction?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on issue #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#issuecomment-594217924
 
 
   > I don't see if there is any special consideration for existing HBase deployments with MOB and how HBASE-22749 will impact those deployments.
   
   The section on "upgrade considerations" should cover this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

busbey commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387350691

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -79,63 +72,212 @@ hcd.setMobThreshold(102400L);
----
====

Review comment:
It should be maintained in the archive area by the snapshot subsystem.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387234197

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +320,51 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

Review comment:
chore

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [hbase] VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.

Posted by GitBox <gi...@apache.org>.

VladRodionov commented on a change in pull request #1232: HBASE-23198 Update ref guide for distributed MOB compaction.
URL: https://github.com/apache/hbase/pull/1232#discussion_r387292709

##########
File path: src/main/asciidoc/_chapters/hbase_mob.adoc
##########
@@ -181,84 +320,51 @@ suit your environment, and restart or rolling restart the RegionServer.
----
====

-=== MOB Optimization Tasks
-
==== Manually Compacting MOB Files

----
-hbase> compact 't1', 'c1’, ‘MOB’
-hbase> major_compact 't1', 'c1’, ‘MOB’
+hbase> major_compact 't1'
+hbase> major_compact 't2', 'c1’
----

-----
-%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
-----
+==== Adjusting the MOB cleaner's tolerance for new hfiles

-With default configs, an example table named 'some_table' in the
-default namespace with a MOB enabled column family named 'foo' this HDFS directory would be
+The MOB cleaner chore ignores all MOB hfiles that were created more recently than an hour prior to
+the start of the shore to ensure we don't miss the reference metadata from teh corresponding regular
+hfile. Without this safety check it would be possible for the cleaner chore to see a MOB hfile for
+an in progress flush or compaction and prematurely archive the MOB data. This default buffer should
+be sufficient for normal use.

Review comment:
We first commit MOB followed by store file. If commit time of a MOB is less than choreStartTime - minAgeToArchive it can be safely archived if the following conditions are met:

1. Time skew across the cluster is less than minAgeToArchive (default is 1h).
2. Under no circumstances time difference between committing MOB and store file can exceed minAgeToArchive. Again - 1h.

That is it, now we do not depend on Chore procedure duration as in v14 patch, b/c the algo was changed.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services