You are viewing a plain text version of this content. The canonical link for it is here.

Posted to codereview@trafodion.apache.org by liuyu000 <gi...@git.apache.org> on 2017/12/29 04:08:50 UTC

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

GitHub user liuyu000 opened a pull request:

    https://github.com/apache/trafodion/pull/1361

    [TRAFODION-2866] Add HBase Configuration Changes for *Trafodion Provisioning Guide*

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/liuyu000/incubator-trafodion HBaseConfigurationChanges

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/trafodion/pull/1361.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1361
    
----
commit 1bf36fe8297866dfd8071d25f753e92a8c50c33f
Author: liu.yu <yu...@...>
Date:   2017-12-29T04:06:53Z

    Add HBase Configuration Changes for *Trafodion Provisioning Guide*

----


---

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

Posted by liuyu000 <gi...@git.apache.org>.

Github user liuyu000 commented on a diff in the pull request:

https://github.com/apache/trafodion/pull/1361#discussion_r159642410

--- Diff: docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc ---
@@ -331,13 +331,38 @@ default. Depending on the size of a user table, we have experienced timeout fail
from this setting. The underlying issue is the length of the execution of the coprocessor within HBase.
+
NOTE: HBase uses the smaller of `hbase.rpc.timeout` and `hbase.client.scanner.timeout.period` to calculate the scanner timeout.
-| hbase.snapshot.master.timeoutMillis and hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
+| hbase.snapshot.master.timeoutMillis
+
+and
+
+hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
If you experience timeout issues with HBase snapshots when you use the {project-name} Bulk Loader or other statements,
you can set the value for these two HBase properties to 10 minutes (600,000 milliseconds).
-| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 (10 GB). We have increased the setting to
-107374182400 (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
+| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 bytes (10 GB). You can increased the setting to
+107374182400 bytes (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
region splitting.
-| hbase.hstore.blockingStoreFiles | 10 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+| hbase.hregion.memstore.block.multiplier | 7
+
+When you have enough memory, you can increase this value to 7 so that more data can be temporarily accepted before flushing to disk instead of blocking writes.
+|This property blocks any further writes from clients to memstores if the memstores exceed the value of `multiplier * flush size`.
+
+Default value: 2
+| hbase.hregion.memstore.flush.size | 536870912 bytes | HBase uses memstore to buffer data before writing it to disk. Once the data in memstore has outgrown this size, it is flushed as an HFile to disk.
+
+Default value: 134217728 bytes (128M)
+| hbase.hstore.blockingStoreFiles | 200 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+
+This property blocks any further writes from memstores to HFile, after existing HFile number hitting this limit until compactions are completed.
--- End diff --

Thanks Dave, your comment has been incorporated :)

---

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

Posted by DaveBirdsall <gi...@git.apache.org>.

Github user DaveBirdsall commented on a diff in the pull request:

https://github.com/apache/trafodion/pull/1361#discussion_r159526555

Suggested wordsmith: "This property blocks any further writes from memstore to HFiles, after the number of existing HFiles hits this limit, until compactions are completed."

---

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

Posted by DaveBirdsall <gi...@git.apache.org>.

Github user DaveBirdsall commented on a diff in the pull request:

https://github.com/apache/trafodion/pull/1361#discussion_r159526124

Grammar typo: "You can increased" should be "You can increase"

---

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

Posted by liuyu000 <gi...@git.apache.org>.

Github user liuyu000 commented on a diff in the pull request:

https://github.com/apache/trafodion/pull/1361#discussion_r159353975

--- Diff: docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc ---
@@ -331,13 +331,38 @@ default. Depending on the size of a user table, we have experienced timeout fail
from this setting. The underlying issue is the length of the execution of the coprocessor within HBase.
+
NOTE: HBase uses the smaller of `hbase.rpc.timeout` and `hbase.client.scanner.timeout.period` to calculate the scanner timeout.
-| hbase.snapshot.master.timeoutMillis and hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
+| hbase.snapshot.master.timeoutMillis
+
+and
+
+hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
If you experience timeout issues with HBase snapshots when you use the {project-name} Bulk Loader or other statements,
you can set the value for these two HBase properties to 10 minutes (600,000 milliseconds).
-| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 (10 GB). We have increased the setting to
-107374182400 (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
+| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 bytes (10 GB). You can increased the setting to
+107374182400 bytes (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
region splitting.
-| hbase.hstore.blockingStoreFiles | 10 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+| hbase.hregion.memstore.block.multiplier | 7
+
+When you have enough memory, you can increase this value to 7 so that more data can be temporarily accepted before flushing to disk instead of blocking writes.
+|This property blocks any further writes from clients to memstores if the memstores exceed the value of `multiplier * flush size`.
+
+Default value: 2
+| hbase.hregion.memstore.flush.size | 536870912 bytes | HBase uses memstore to buffer data before writing it to disk. Once the data in memstore has outgrown this size, it is flushed as an HFile to disk.
+
+Default value: 134217728 bytes (128M)
+| hbase.hstore.blockingStoreFiles | 200 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+
+This property blocks any further writes from memstores to regions after HFile number hitting this limit until compactions are completed.
--- End diff --

Thanks Ming, your comment has been incorporated. :)

---

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/trafodion/pull/1361


---

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

Posted by traflm <gi...@git.apache.org>.

Github user traflm commented on a diff in the pull request:

https://github.com/apache/trafodion/pull/1361#discussion_r159170848

--- Diff: docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc ---
@@ -331,13 +331,38 @@ default. Depending on the size of a user table, we have experienced timeout fail
from this setting. The underlying issue is the length of the execution of the coprocessor within HBase.
+
NOTE: HBase uses the smaller of `hbase.rpc.timeout` and `hbase.client.scanner.timeout.period` to calculate the scanner timeout.
-| hbase.snapshot.master.timeoutMillis and hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
+| hbase.snapshot.master.timeoutMillis
+
+and
+
+hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds.
If you experience timeout issues with HBase snapshots when you use the {project-name} Bulk Loader or other statements,
you can set the value for these two HBase properties to 10 minutes (600,000 milliseconds).
-| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 (10 GB). We have increased the setting to
-107374182400 (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
+| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 bytes (10 GB). You can increased the setting to
+107374182400 bytes (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from
region splitting.
-| hbase.hstore.blockingStoreFiles | 10 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+| hbase.hregion.memstore.block.multiplier | 7
+
+When you have enough memory, you can increase this value to 7 so that more data can be temporarily accepted before flushing to disk instead of blocking writes.
+|This property blocks any further writes from clients to memstores if the memstores exceed the value of `multiplier * flush size`.
+
+Default value: 2
+| hbase.hregion.memstore.flush.size | 536870912 bytes | HBase uses memstore to buffer data before writing it to disk. Once the data in memstore has outgrown this size, it is flushed as an HFile to disk.
+
+Default value: 134217728 bytes (128M)
+| hbase.hstore.blockingStoreFiles | 200 | http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
+
+This property blocks any further writes from memstores to regions after HFile number hitting this limit until compactions are completed.
--- End diff --

This property blocks any further writes from memstores to regions after HFile number hitting this limit until compactions are completed.
More precisely, may be:
This property blocks any further writes from memstores to HFile, after existing HFile number hitting this limit until compactions are completed.

---

[GitHub] trafodion pull request #1361: [TRAFODION-2866] Add HBase Configuration Chang...

Posted by liuyu000 <gi...@git.apache.org>.

Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/trafodion/pull/1361#discussion_r159642419
  
    --- Diff: docs/provisioning_guide/src/asciidoc/_chapters/requirements.adoc ---
    @@ -331,13 +331,38 @@ default. Depending on the size of a user table, we have experienced timeout fail
     from this setting. The underlying issue is the length of the execution of the coprocessor within HBase.
      +
     NOTE: HBase uses the smaller of `hbase.rpc.timeout` and `hbase.client.scanner.timeout.period` to calculate the scanner timeout. 
    -| hbase.snapshot.master.timeoutMillis and hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds. 
    +| hbase.snapshot.master.timeoutMillis 
    +
    +and 
    +
    +hbase.snapshot.region.timeout | 10 minutes | HBase's default setting is 60000 milliseconds. 
     If you experience timeout issues with HBase snapshots when you use the {project-name} Bulk Loader or other statements, 
     you can set the value for these two HBase properties to 10 minutes (600,000 milliseconds).
    -| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 (10 GB). We have increased the setting to 
    -107374182400 (100 GB), which reduces the number of HStoreFiles per table and appears to reduce disruptions to active transactions from 
    +| hbase.hregion.max.filesize | 107374182400 bytes | HBase's default setting is 10737418240 bytes (10 GB). You can increased the setting to 
    --- End diff --
    
    Oops, sorry, I've corrected, thanks Dave :)


---