You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Liyin Tang (Created) (JIRA)" <ji...@apache.org> on 2012/04/12 23:43:19 UTC
[jira] [Created] (HBASE-5776) HTableMultiplexer
HTableMultiplexer
------------------
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5776:
-------------------------------
Attachment: D2775.3.patch
Liyin updated the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Reviewers: Kannan
Address Kannan's comments.
REVISION DETAIL
https://reviews.facebook.net/D2775
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnection.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/client/HTable.java
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
src/main/java/org/apache/hadoop/hbase/client/MultiplexablePut.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
To: Kannan, Liyin
Cc: JIRA, tedyu
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Kannan Muthukkaruppan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253939#comment-13253939 ]
Kannan Muthukkaruppan commented on HBASE-5776:
----------------------------------------------
Todd:
This is API is user-facing.
The HTables themselves aren't thread safe, so applications end up using a separate HTable per thread, and the per-RS batching happens only within the context of a single HTable request. The HTableMultiplexer was intended as a simple wrapper helper/utility that allows the app to get "deferred put" functionality, but with batching even the puts coming from different application threads in most effective manner. It also supports puts to multiple tables to be grouped in the same batch. Extending HTables which are not thread safe for this use case didn't make sense, and also painful from backward compat perspective. In this approach, a single instance of HTableMultiplexer can be use by all the app threads to shove puts, and let the multiplexer group commits per RS, and forward them along in batches. This non-blocking multiplexer API could get more sophisticated in future, e.g., supporting async gets (with callbacks and so on) and different config knobs (e.g., how much buffering memory to allow for all queues, retry/failure handling logic via callbacks, etc.). And maybe, in future, we can gradually deprecate the deferred put functionality in HTable, and recommend all the non-blocking use cases to go via the multiplexer.
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Kannan Muthukkaruppan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254901#comment-13254901 ]
Kannan Muthukkaruppan commented on HBASE-5776:
----------------------------------------------
Ted: I think keeping "Table" in the name does not take way from the fact that it can support multiple tables. The multiplexer supports table operations-- so imo the name is ok as is.
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279084#comment-13279084 ]
Phabricator commented on HBASE-5776:
------------------------------------
Liyin has closed the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Close this 89-fb revision and will port to apache trunk soon.
REVISION DETAIL
https://reviews.facebook.net/D2775
To: Kannan, Liyin
Cc: JIRA, tedyu
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch, D2775.5.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252965#comment-13252965 ]
Phabricator commented on HBASE-5776:
------------------------------------
tedyu has commented on the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' -> 'and there is no successful put'
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared -> Prepare
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and'
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' -> caught
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk.
src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ?
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount -> successfulCount
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ?
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ?
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping.
Should we introduce some counter for the above case?
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please.
REVISION DETAIL
https://reviews.facebook.net/D2775
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270734#comment-13270734 ]
Liyin Tang commented on HBASE-5776:
-----------------------------------
@Otis,
Good questions:)
The motivation of HTableMultipler is NOT to replace the existing HTable but to provide a new alternative solutions for some high-throughput realtime applications. Usually, these applications can tolerate some data-loss during the region failover or other region-unavailable time instead of caching them in memory.
In these cases, HTableMultiplexer tries its best to ONLY drop the data for these unavailable regions, instead of slowing down the overall multiput throughput which may cause more data dropping since the client fails to buffer any more incoming data across all the regions.
Also, there is no call-back function if the put is finally failed after N attempts. And if client uses the multiput api in HTable, there is no such call back as well. However, we could have a way to config the HTableMultiplex to intelligently call back for some put operations.
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5776:
-------------------------------
Attachment: D2775.2.patch
Liyin updated the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Reviewers: Kannan
As discussed with Kannan offline, HTableMultiplexer will keep a pool htable instance. So user doesn't need to pass the htable as parameter for the put request. Also keep each put retrying several times before failing and clear the cache location after failing.
Add more java doc and license.
REVISION DETAIL
https://reviews.facebook.net/D2775
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnection.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/client/HTable.java
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5776:
-------------------------------
Attachment: D2775.4.patch
Liyin updated the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Reviewers: Kannan
Addressed some offline comments from Kannan about improving the failed put processing.
REVISION DETAIL
https://reviews.facebook.net/D2775
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnection.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/client/HTable.java
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
To: Kannan, Liyin
Cc: JIRA, tedyu
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253122#comment-13253122 ]
Todd Lipcon commented on HBASE-5776:
------------------------------------
Can this be done without adding a new interface? eg can we make the existing queueing/flushing behavior of HTable use a threadpool?
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252966#comment-13252966 ]
Phabricator commented on HBASE-5776:
------------------------------------
tedyu has commented on the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' -> 'and there is no successful put'
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared -> Prepare
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and'
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760.
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' -> caught
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk.
src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only.
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ?
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount -> successfulCount
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ?
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ?
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping.
Should we introduce some counter for the above case?
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please.
REVISION DETAIL
https://reviews.facebook.net/D2775
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Liyin Tang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253768#comment-13253768 ]
Liyin Tang commented on HBASE-5776:
-----------------------------------
@Todd, The HTableMultiplexer is designed to process the puts requests across different tables.
All the puts across the tables will be sharded into each different queues based on their destination region server. It will help to batch more puts for each region server before sending out the rpc request.
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274411#comment-13274411 ]
Otis Gospodnetic commented on HBASE-5776:
-----------------------------------------
I think asynchbase does the same thing - has a queue for each RS.
Compatibility - don't know off hand, check its repo on github.
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273652#comment-13273652 ]
Liyin Tang commented on HBASE-5776:
-----------------------------------
Interesting :) Thanks @Otis for pointing the asynchbase out and I didn't know that before.
I haven't read the asynchbase closely and it looks like both HTableMultiplexer and asynchbase are attempting to provide async (deferred) api for applications.
However, I believe the HTableMulplixer mainly focuses on solving the slow region server problem by partition the put requests based on its destination region server {color:red} before deferring it {color} . So slow region server will only hammer one shard of the requests.
BTW, is the asynchbase compatible with the current HBASE rpc protocol ?
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253976#comment-13253976 ]
Zhihong Yu commented on HBASE-5776:
-----------------------------------
Thanks for the explanation, Kannan.
Since the Multiplexer isn't tied to any single table and it may support get's in the future, shall we remove the 'Table' in the class name ?
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270533#comment-13270533 ]
Otis Gospodnetic commented on HBASE-5776:
-----------------------------------------
What happens when some of the puts fail even after N attempts? Does the caller get notified that a failure happened and which puts failed? If not, how should one deal with such situations?
What happens with puts that are in memory, not yet written to RS, and the app dies/stops for whatever reason. Are those puts lost?
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5776:
-------------------------------
Attachment: D2775.1.patch
Liyin requested code review of "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Reviewers: Kannan
There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
TEST PLAN
Unit tests
REVISION DETAIL
https://reviews.facebook.net/D2775
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnection.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/client/HTable.java
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/6303/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253829#comment-13253829 ]
Todd Lipcon commented on HBASE-5776:
------------------------------------
Gotcha. But is this a user-facing API? or would the HTables themselves write into an HTable multiplexer? It seems to me like this kind of behavior should happen automatically for any writes going into HBase, without expanding our API footprint. Is there some reason that that's impossible?
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493477#comment-13493477 ]
Otis Gospodnetic commented on HBASE-5776:
-----------------------------------------
[~liangly] Any plans/ETA for getting this in trunk?
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5776:
-------------------------------
Attachment: D2775.1.patch
Liyin requested code review of "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Reviewers: Kannan
There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
TEST PLAN
Unit tests
REVISION DETAIL
https://reviews.facebook.net/D2775
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnection.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/client/HTable.java
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/6303/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278600#comment-13278600 ]
Phabricator commented on HBASE-5776:
------------------------------------
Kannan has accepted the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
looks good Liyin. Remaining are cosmetic comments, hence accepting!
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1731 space after "to"
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:174 failed is unused
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:159 failed is unused
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:359 "-1" -> add space between - and 1.
REVISION DETAIL
https://reviews.facebook.net/D2775
BRANCH
HBASE-5776
To: Kannan, Liyin
Cc: JIRA, tedyu
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5776) HTableMultiplexer
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5776:
-------------------------------
Attachment: D2775.5.patch
Liyin updated the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Reviewers: Kannan
Addressed Kannan's comments
REVISION DETAIL
https://reviews.facebook.net/D2775
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnection.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/client/HTable.java
src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
To: Kannan, Liyin
Cc: JIRA, tedyu
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch, D2775.5.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273631#comment-13273631 ]
Otis Gospodnetic commented on HBASE-5776:
-----------------------------------------
I read up on asynchbase yesterday. Doesn't asynchbase already solve the problem this issue is aimed at?
See:
http://search-hadoop.com/m/J6olJ11Idb
http://search-hadoop.com/m/4fogb27wKWC
> HTableMultiplexer
> ------------------
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch
>
>
> There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.
>
> To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
> The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.
> Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira