Does HBase combine multiple Puts against the same region server?

[yun peng <pe...@gmail.com>]

Hi, I have question on how the multiple Puts() are executed when they are
issued against the same region server.

For example, in the case of asynchronous executing Put() using
setAutoFlush(true), there will be multiple Puts() in the writeBuffer. Or
use HTbale API put(List puts) which directly issues multiple Puts. In
either case, let's say, would two Puts in the list, which are issued
against the same HRegionServer, be combined in a single RPC before sending
to that RegionServer? I appreciate it if I can get pointer to the code in
HBase. Thanks...

Regards,
Yun

Re: Does HBase combine multiple Puts against the same region server?

["daidongly@gmail.com" <da...@gmail.com>]

Hi, Yun

Code in HConnectionManager.java, method: "submit(List<Action<R>> actionList, ...)". 

It will combine all actions towards the same rs, and send them together...

- Dong
在 2012-12-7，上午2:34， yun peng 写道：

> Hi, I have question on how the multiple Puts() are executed when they are
> issued against the same region server.
> 
> For example, in the case of asynchronous executing Put() using
> setAutoFlush(true), there will be multiple Puts() in the writeBuffer. Or
> use HTbale API put(List puts) which directly issues multiple Puts. In
> either case, let's say, would two Puts in the list, which are issued
> against the same HRegionServer, be combined in a single RPC before sending
> to that RegionServer? I appreciate it if I can get pointer to the code in
> HBase. Thanks...
> 
> Regards,
> Yun

代栋 (Dong Dai)
Phd Student major in Cloud Computing
Dept. C.S. 
University of Science and Technology of China
Suzhou Advanced Institute of USTC
E-mail: daidong@mail.ustc.edu.cn
URL: home.ustc.edu.cn/~daidong

Re: Does HBase combine multiple Puts against the same region server?

[Varun Sharma <va...@pinterest.com>]

The code does the the following:

1) Add put to buffer
2) If buffer is > size (2MB) - then flush the puts
3) Finally, for the left over puts in the buffer - if auto flush if on
(which is the default), flush the puts

When you flush your puts, the puts are grouped by regionserver and a single
call is made to each region server - so this could be 1 call or n calls if
you have n region servers. If you want to flush a large payload - you can
consider increasing the buffer size.

On Thu, Dec 6, 2012 at 11:17 AM, Jimmy Xiang <jx...@cloudera.com> wrote:

> This has been built in hbase for quite some time, no application change.
>
> On Thu, Dec 6, 2012 at 11:08 AM, yun peng <pe...@gmail.com> wrote:
> > Is that done in current HBase implementation (say, 0.94.2 or more recent
> > one) or it requires applications to handle it?
> > Thanks for your note,
> > Yun
> >
> > On Thu, Dec 6, 2012 at 1:42 PM, Jimmy Xiang <jx...@cloudera.com> wrote:
> >
> >> If auto flush is off, multiple puts could be combined into a batch and
> >> send to the region server in one RPC call if they are for the same
> >> region server.
> >>
> >> Thanks,
> >> Jimmy
> >>
> >> On Thu, Dec 6, 2012 at 10:34 AM, yun peng <pe...@gmail.com>
> wrote:
> >> > Hi, I have question on how the multiple Puts() are executed when they
> are
> >> > issued against the same region server.
> >> >
> >> > For example, in the case of asynchronous executing Put() using
> >> > setAutoFlush(true), there will be multiple Puts() in the writeBuffer.
> Or
> >> > use HTbale API put(List puts) which directly issues multiple Puts. In
> >> > either case, let's say, would two Puts in the list, which are issued
> >> > against the same HRegionServer, be combined in a single RPC before
> >> sending
> >> > to that RegionServer? I appreciate it if I can get pointer to the
> code in
> >> > HBase. Thanks...
> >> >
> >> > Regards,
> >> > Yun
> >>
>

Re: Does HBase combine multiple Puts against the same region server?

[Jimmy Xiang <jx...@cloudera.com>]

This has been built in hbase for quite some time, no application change.

On Thu, Dec 6, 2012 at 11:08 AM, yun peng <pe...@gmail.com> wrote:
> Is that done in current HBase implementation (say, 0.94.2 or more recent
> one) or it requires applications to handle it?
> Thanks for your note,
> Yun
>
> On Thu, Dec 6, 2012 at 1:42 PM, Jimmy Xiang <jx...@cloudera.com> wrote:
>
>> If auto flush is off, multiple puts could be combined into a batch and
>> send to the region server in one RPC call if they are for the same
>> region server.
>>
>> Thanks,
>> Jimmy
>>
>> On Thu, Dec 6, 2012 at 10:34 AM, yun peng <pe...@gmail.com> wrote:
>> > Hi, I have question on how the multiple Puts() are executed when they are
>> > issued against the same region server.
>> >
>> > For example, in the case of asynchronous executing Put() using
>> > setAutoFlush(true), there will be multiple Puts() in the writeBuffer. Or
>> > use HTbale API put(List puts) which directly issues multiple Puts. In
>> > either case, let's say, would two Puts in the list, which are issued
>> > against the same HRegionServer, be combined in a single RPC before
>> sending
>> > to that RegionServer? I appreciate it if I can get pointer to the code in
>> > HBase. Thanks...
>> >
>> > Regards,
>> > Yun
>>

Re: Does HBase combine multiple Puts against the same region server?

[yun peng <pe...@gmail.com>]

Is that done in current HBase implementation (say, 0.94.2 or more recent
one) or it requires applications to handle it?
Thanks for your note,
Yun

On Thu, Dec 6, 2012 at 1:42 PM, Jimmy Xiang <jx...@cloudera.com> wrote:

> If auto flush is off, multiple puts could be combined into a batch and
> send to the region server in one RPC call if they are for the same
> region server.
>
> Thanks,
> Jimmy
>
> On Thu, Dec 6, 2012 at 10:34 AM, yun peng <pe...@gmail.com> wrote:
> > Hi, I have question on how the multiple Puts() are executed when they are
> > issued against the same region server.
> >
> > For example, in the case of asynchronous executing Put() using
> > setAutoFlush(true), there will be multiple Puts() in the writeBuffer. Or
> > use HTbale API put(List puts) which directly issues multiple Puts. In
> > either case, let's say, would two Puts in the list, which are issued
> > against the same HRegionServer, be combined in a single RPC before
> sending
> > to that RegionServer? I appreciate it if I can get pointer to the code in
> > HBase. Thanks...
> >
> > Regards,
> > Yun
>

Re: Does HBase combine multiple Puts against the same region server?

[Jimmy Xiang <jx...@cloudera.com>]

If auto flush is off, multiple puts could be combined into a batch and
send to the region server in one RPC call if they are for the same
region server.

Thanks,
Jimmy

On Thu, Dec 6, 2012 at 10:34 AM, yun peng <pe...@gmail.com> wrote:
> Hi, I have question on how the multiple Puts() are executed when they are
> issued against the same region server.
>
> For example, in the case of asynchronous executing Put() using
> setAutoFlush(true), there will be multiple Puts() in the writeBuffer. Or
> use HTbale API put(List puts) which directly issues multiple Puts. In
> either case, let's say, would two Puts in the list, which are issued
> against the same HRegionServer, be combined in a single RPC before sending
> to that RegionServer? I appreciate it if I can get pointer to the code in
> HBase. Thanks...
>
> Regards,
> Yun