You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafodion.apache.org by db...@apache.org on 2018/01/04 19:14:35 UTC
[1/4] trafodion git commit: Add HBase Configuration Changes for
*Trafodion Provisioning Guide*
Repository: trafodion
Updated Branches:
refs/heads/master d08a0819f -> 925e44d73
Add HBase Configuration Changes for *Trafodion Provisioning Guide*
Project: http://git-wip-us.apache.org/repos/asf/trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafodion/commit/1bf36fe8
Tree: http://git-wip-us.apache.org/repos/asf/trafodion/tree/1bf36fe8
Diff: http://git-wip-us.apache.org/repos/asf/trafodion/diff/1bf36fe8
Branch: refs/heads/master
Commit: 1bf36fe8297866dfd8071d25f753e92a8c50c33f
Parents: 51935dc
Author: liu.yu <yu...@esgyn.cn>
Authored: Fri Dec 29 12:06:53 2017 +0800
Committer: liu.yu <yu...@esgyn.cn>
Committed: Fri Dec 29 12:06:53 2017 +0800
----------------------------------------------------------------------
.../src/asciidoc/_chapters/requirements.adoc | 37 ++++++++++++++++----
1 file changed, 31 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafodion/blob/1bf36fe8/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
----------------------------------------------------------------------
diff --git a/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc b/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
index d49d8d5..520d100 100644
--- a/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
+++ b/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
@@ -319,7 +319,7 @@ These settings are configured in the `hadoop-env.sh` file.
[[requirements-recommended-hbase-configuration-changes]]
=== Recommended HBase Configuration Changes
-[cols="30%l,20%,50%a",options="header"]
+[options="header"]
|===
| Configuration Property | Recommended Setting | Guidance
| hbase.rpc.timeout | 10 minutes | This setting depends on the tables' size. Sixty (60) seconds is the default.
@@ -331,13 +331,38 @@ default. Depending on the size of a user table, we have experienced timeout fail
from this setting. The underlying issue is the length of the execution of the coprocessor within HBase.
+
NOTE: HBase uses the smaller of `hbase.rpc.timeout` and `hbase.client.scanner.timeout.period` to calculate the scanner timeout.
-| hbase.snapshot.master.timeoutMillis and hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
+| hbase.snapshot.master.timeoutMillis
+
+and
+
+hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
If you experience timeout issues with HBase snapshots when you use the {project-name} Bulk Loader or other statements,
you can set the value for these two HBase properties to 10 minutes (600,000 milliseconds).
-| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 (10 GB). We have increased the setting to
-107374182400 (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
+| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 bytes (10 GB). You can increased the setting to
+107374182400 bytes (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
region splitting.
-| hbase.hstore.blockingStoreFiles | 10 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+| hbase.hregion.memstore.block.multiplier | 7
+
+When you have enough memory, you can increase this value to 7 so that more data can be temporarily accepted before flushing to disk instead of blocking writes.
+|This property blocks any further writes from clients to memstores if the memstores exceed the value of `multiplier * flush size`.
+
+Default value: 2
+| hbase.hregion.memstore.flush.size | 536870912 bytes | HBase uses memstore to buffer data before writing it to disk. Once the data in memstore has outgrown this size, it is flushed as an HFile to disk.
+
+Default value: 134217728 bytes (128M)
+| hbase.hstore.blockingStoreFiles | 200 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+
+This property blocks any further writes from memstores to regions after HFile number hitting this limit until compactions are completed.
+
+Default value: 7
+
| hbase.regionserver.handler.count | <num> | This setting should match the number of concurrent sessions (mxosrvr). The default is 10.
-|===
+| hbase.regionserver.region.split.policy |
+`org.apache.hadoop.hbase.regionserver.
+ConstantSizeRegionSplitPolicy`
+
+Once the regions reach the maximum size configured by `hbase.hregion.max.filesize` property, they are split into halves.
+| Specify the split policy of HBase.
+Default value: `IncreasingToUpperBoundRegionSplitPolicy`
+|===
\ No newline at end of file
[2/4] trafodion git commit: Incorporate Comment 1
Posted by db...@apache.org.
Incorporate Comment 1
Project: http://git-wip-us.apache.org/repos/asf/trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafodion/commit/dfcfc5e7
Tree: http://git-wip-us.apache.org/repos/asf/trafodion/tree/dfcfc5e7
Diff: http://git-wip-us.apache.org/repos/asf/trafodion/diff/dfcfc5e7
Branch: refs/heads/master
Commit: dfcfc5e73f9e5ddb6827d88424b40f3b438ede14
Parents: 1bf36fe
Author: liu.yu <yu...@esgyn.cn>
Authored: Wed Jan 3 10:21:42 2018 +0800
Committer: liu.yu <yu...@esgyn.cn>
Committed: Wed Jan 3 10:21:42 2018 +0800
----------------------------------------------------------------------
docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafodion/blob/dfcfc5e7/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
----------------------------------------------------------------------
diff --git a/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc b/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
index 520d100..7d687dc 100644
--- a/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
+++ b/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
@@ -352,7 +352,7 @@ Default value: 2
Default value: 134217728 bytes (128M)
| hbase.hstore.blockingStoreFiles | 200 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
-This property blocks any further writes from memstores to regions after HFile number hitting this limit until compactions are completed.
+This property blocks any further writes from memstores to HFile, after existing HFile number hitting this limit until compactions are completed.
Default value: 7
[4/4] trafodion git commit: Merge [TRAFODION-2866] PR 1361 Add HBase
config changes to provisioning guide
Posted by db...@apache.org.
Merge [TRAFODION-2866] PR 1361 Add HBase config changes to provisioning guide
Project: http://git-wip-us.apache.org/repos/asf/trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafodion/commit/925e44d7
Tree: http://git-wip-us.apache.org/repos/asf/trafodion/tree/925e44d7
Diff: http://git-wip-us.apache.org/repos/asf/trafodion/diff/925e44d7
Branch: refs/heads/master
Commit: 925e44d7344e9a6a3cc9f21c0e81572a9b6b2812
Parents: d08a081 57ab65b
Author: Dave Birdsall <db...@apache.org>
Authored: Thu Jan 4 19:13:40 2018 +0000
Committer: Dave Birdsall <db...@apache.org>
Committed: Thu Jan 4 19:13:40 2018 +0000
----------------------------------------------------------------------
.../src/asciidoc/_chapters/requirements.adoc | 37 ++++++++++++++++----
1 file changed, 31 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
[3/4] trafodion git commit: Incorporate Comments 1
Posted by db...@apache.org.
Incorporate Comments 1
Project: http://git-wip-us.apache.org/repos/asf/trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafodion/commit/57ab65b6
Tree: http://git-wip-us.apache.org/repos/asf/trafodion/tree/57ab65b6
Diff: http://git-wip-us.apache.org/repos/asf/trafodion/diff/57ab65b6
Branch: refs/heads/master
Commit: 57ab65b65962868101bfc016b9e5a3aabc1d6570
Parents: dfcfc5e
Author: liu.yu <yu...@esgyn.cn>
Authored: Thu Jan 4 20:33:59 2018 +0800
Committer: liu.yu <yu...@esgyn.cn>
Committed: Thu Jan 4 20:33:59 2018 +0800
----------------------------------------------------------------------
docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafodion/blob/57ab65b6/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
----------------------------------------------------------------------
diff --git a/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc b/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
index 7d687dc..64501fd 100644
--- a/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
+++ b/docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc
@@ -338,7 +338,7 @@ and
hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
If you experience timeout issues with HBase snapshots when you use the {project-name} Bulk Loader or other statements,
you can set the value for these two HBase properties to 10 minutes (600,000 milliseconds).
-| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 bytes (10 GB). You can increased the setting to
+| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 bytes (10 GB). You can increase the setting to
107374182400 bytes (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
region splitting.
| hbase.hregion.memstore.block.multiplier | 7
@@ -352,7 +352,7 @@ Default value: 2
Default value: 134217728 bytes (128M)
| hbase.hstore.blockingStoreFiles | 200 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
-This property blocks any further writes from memstores to HFile, after existing HFile number hitting this limit until compactions are completed.
+This property blocks any further writes from memstores to HFile, after the number of existing HFile hits this limit, until compactions are completed.
Default value: 7