[jira] [Comment Edited] (HBASE-22634) Improve performance of BufferedMutator

["Sebastien Barnoud (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873509#comment-16873509 ] 

Sebastien Barnoud edited comment on HBASE-22634 at 6/26/19 4:44 PM:
--------------------------------------------------------------------

For the need to synchronize sendMultiAction, well this method may be called in concurrency by the period flush and by mutate when the write buffer is full.
 This method just takes action from the queue and is called by
{code:java}
private <CResult> AsyncRequestFuture submit(AsyncProcessTask<CResult> task,
  boolean atLeastOne) throws InterruptedIOException {
{code}
But in case of backpressure (too many tasks for a region server for exemple), this method may wait a lot of time. That's why i choose to synchronize as late as possible for the shortest time possible.


was (Author: sebastien barnoud):
For the need to synchronize sendMultiAction, well this method may be called in concurrency by the period flush and by mutate when the write buffer is full.
 This method just table action from the queue and is called by
{code:java}
private <CResult> AsyncRequestFuture submit(AsyncProcessTask<CResult> task,
  boolean atLeastOne) throws InterruptedIOException {
{code}
But in case of backpressure (too many tasks for a region server for exemple), this method may wait a lot of time. That's why i choose to synchronize as late as possible for the shortest time possible.

> Improve performance of BufferedMutator
> --------------------------------------
>
>                 Key: HBASE-22634
>                 URL: https://issues.apache.org/jira/browse/HBASE-22634
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client
>    Affects Versions: 2.1.5
>         Environment: HDP 2.6.5
> Linux RedHat
>            Reporter: Sebastien Barnoud
>            Priority: Major
>         Attachments: HBASE-22634.001.branch-2.patch
>
>
> The default ThreadPoolExecutor uses a default size of 1 (property hbase.htable.threads.max). When using a size > 1, we still encountered poor performance and exception while submitting to the pool (pool exceed its capacity).
> This patch propose a fix on different issues encountered when the pool size is > 1:
>  * thread safety issue
>  * concurrent cleanup by Netty and the "legacy" code
>  * errors in the backpressure
>  * Netty memory leak
> And propose a BufferedMutatorThreadPoolExecutor which:
>  * uses hbase.client.max.total.tasks as the default size (instead of 1)
>  * some usefull metrics
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)