[jira] [Commented] (HBASE-22634) Improve performance of BufferedMutator

["Sebastien Barnoud (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873340#comment-16873340 ] 

Sebastien Barnoud commented on HBASE-22634:
-------------------------------------------

With this patch, a sample Spark job using:
 * hbase.htable.threads.max and hbase.client.max.total.tasks 32
 * hbase.client.max.perserver.tasks 8
 * hbase.client.max.perregion.tasks 1 
 * hbase.client.rpc.compressor GzipCodec.class.getCanonicalName()

has a throughput much higher than the default implementation (with hbase.htable.threads.max 1 or 32) and avoid some retries because of bug in the current backpressure management (where we ensure only that 1 single task is available, while submitting N).

In addition, this patch fix several issues, the main is a leak in Netty.

 

> Improve performance of BufferedMutator
> --------------------------------------
>
>                 Key: HBASE-22634
>                 URL: https://issues.apache.org/jira/browse/HBASE-22634
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client
>    Affects Versions: 2.1.5
>         Environment: HDP 2.6.5
> Linux RedHat
>            Reporter: Sebastien Barnoud
>            Priority: Major
>         Attachments: HBASE-22634.001.branch-2.patch
>
>
> The default ThreadPoolExecutor uses a default size of 1 (property hbase.htable.threads.max). When using a size > 1, we still encountered poor performance and exception while submitting to the pool (pool exceed its capacity).
> This patch propose a fix on different issues encountered when the pool size is > 1:
>  * thread safety issue
>  * concurrent cleanup by Netty and the "legacy" code
>  * errors in the backpressure
>  * Netty memory leak
> And propose a BufferedMutatorThreadPoolExecutor which:
>  * uses hbase.client.max.total.tasks as the default size (instead of 1)
>  * some usefull metrics
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)