Re: Async RpcClient

[Jurriaan Mous <ju...@jurmo.us>]

The Async RPC Client has been committed. Thanks to those who helped me with this complex task, especially Stack! :) The existing tests where great to find any hidden issues. If you encounter any issue related to it please report it and mention me in the issue so I can help. (Jurriaan Mous)

To quote the Release Note:

Retrofit a new, netty-based rpc transport on the client. This client is slightly slower if little contention given the extra tier or so that netty adds and that we block on a Future waiting on the call to finish.  This client opens the way for HBase having a native Async API.

This client is on by default in master branch (2.0 hbase). It is off in branch-1.0 (hbase-1.1.x).  To enable it, set "hbase.rpc.client.impl" to "org.apache.hadoop.hbase.ipc.AsyncRpcClient"

Lets wait for a while to see if there are any problems found while being the default in master. When proven stable for a while I suggest to remove the old sync-only client on master/trunk first and implement some first additional native async calls in the Table and Admin interface. Then adapt AsyncProcess to native async calls and move scanners over to an optional async call path.



> Op 18 dec. 2014, om 22:22 heeft Stack <st...@duboce.net> het volgende geschreven:
> 
> On Wed, Dec 17, 2014 at 9:44 AM, Jurriaan Mous <ju...@jurmo.us> wrote:
>> 
>> Hi,
>> 
>> I have been working on a Netty 4 based async HBase client to fit better
>> within the event driven server I have been developing. -
>> https://github.com/jurmous/async-hbase-client/tree/HBase-0.99 <
>> https://github.com/jurmous/async-hbase-client/tree/HBase-0.99>
>> 
>> Recently I have been submitting some patches to make it easier to switch
>> out the RpcClient of HBase. This to enable HBase to use the client itself
>> in all communication. I wanted to do this to use the tests on HBase to
>> check if the client was solid on all edge cases but also to enable HBase to
>> possibly migrate to an async client. These were committed on master and
>> branch-1
>> https://issues.apache.org/jira/browse/HBASE-12597 <
>> https://issues.apache.org/jira/browse/HBASE-12597>
>> https://issues.apache.org/jira/browse/HBASE-12684 <
>> https://issues.apache.org/jira/browse/HBASE-12684>
>> 
>> Now I am at the next step where I want to contribute back the
>> AsyncRpcClient itself.
>> 
>> 
> Sweet.
> 
> 
> 
>> I have opened this issue to add AsyncRpcClient:
>> https://issues.apache.org/jira/browse/HBASE-12684 <
>> https://issues.apache.org/jira/browse/HBASE-12684>
>> In the current patch the new async client is the default.
>> 
>> 3 questions:
>> Can anyone with a proper Kerberos setup test if the async client works?
>> SASL Digest auth works but I haven’t tested Kerberos yet.
>> 
>> Can anyone with know-how on benchmarking test what the performance of this
>> client is compared to the current client? The performance should of course
>> be great in all relevant metrics will it ever be the main client.
>> 
>> 
> I can have a go at the above Jurriaan, maybe next week?
> 
> 
> 
>> What will we do with the old RpcClient if the async RpcClient is
>> introduced? It would be great to remove it so hbase can internally base
>> anything async (like AsyncProcess) on the async RPC client and this would
>> not be possible with an also supported sync RPC client. A possible route is
>> to make AsyncRpcClient an option on 1.x and a default on 2.0 branch where
>> we remove the old client.
>> 
>> 
> Option in 1.1 and default in 2.0 would be the way to go after we get some
> signal it is an improvement.
> 
> 
> 
>> When the new AsyncRpcClient will be the default it is possible to
>> introduce callback variants of the Table, Scanner and Admin methods and
>> possibly deprecate batch and other AsyncProcess based calls to replace it
>> with a more flexible batch callback implementation.
>> 
>> 
> Excellent.  These additions would complement/decorate the new 1.0
> Table/Admin?
> 
> St.Ack