short scans in one RPC

[Dave Latham <la...@davelink.net>]

I was recently dealing with a situation of the META table being slammed
with queries for a long time.  While evaluating the problem I saw the IPCs
being in a cycle like the following:

HRegionInterface#getClosestRowBefore queueTime=200 processingTime=1 ... 444
bytes.
HRegionInterface#openScanner queueTime=191 processingTime=0 ... 22 bytes.
HRegionInterface#next queueTime=154 processingTime=1 ...  4551 bytes.
HRegionInterface#close queueTime=216 processingTime=0 ... 16 bytes.
(rinse, repeat)

I'm bumping our prefetching from 10 rows to 50 rows to help cut down on the
number of cycles, but it also occurs to me that it's a bit painful to
require 3 RPCs to do a short scan.

What would people think about adding method for short scans which
encapsulates (openScanner / next / close) in a single RPC?

Dave

Re: short scans in one RPC

[Dave Latham <la...@davelink.net>]

>
> Does trunk not do this?
>
> It at least has all the mechanisms to achieve this where open scanner
> returns results and you can set close on the scanner so you do not have to
> go back again to do the close explicitly.
>
> A Get is a scan; it for sure does this.
>

Cool. I didn't realize the scan interface had changed so much as part of
switching to PB in 0.96.  For anyone else who is curious you can find the
trail in the review comments on HBASE-5443.  From what I can tell the
Client code still doesn't attempt to detect an already completed scanner
and skip the extra RPC for close, but I haven't gone through carefully
enough to say for sure.  For META prefetch scans (for example) there's no
field in the Scan object to indicate the max number of rows to return and
it's not using any filters so it must be still doing at least 2 RPCs from
what I can tell.

Re: short scans in one RPC

[Stack <st...@duboce.net>]

On Thu, Aug 22, 2013 at 9:53 AM, Dave Latham <la...@davelink.net> wrote:

> I was recently dealing with a situation of the META table being slammed
> with queries for a long time.  While evaluating the problem I saw the IPCs
> being in a cycle like the following:
>
> HRegionInterface#getClosestRowBefore queueTime=200 processingTime=1 ... 444
> bytes.
> HRegionInterface#openScanner queueTime=191 processingTime=0 ... 22 bytes.
> HRegionInterface#next queueTime=154 processingTime=1 ...  4551 bytes.
> HRegionInterface#close queueTime=216 processingTime=0 ... 16 bytes.
> (rinse, repeat)
>
> I'm bumping our prefetching from 10 rows to 50 rows to help cut down on the
> number of cycles, but it also occurs to me that it's a bit painful to
> require 3 RPCs to do a short scan.
>
> What would people think about adding method for short scans which
> encapsulates (openScanner / next / close) in a single RPC?
>

Does trunk not do this?

It at least has all the mechanisms to achieve this where open scanner
returns results and you can set close on the scanner so you do not have to
go back again to do the close explicitly.

A Get is a scan; it for sure does this.

St.Ack

Re: short scans in one RPC

[Ted Yu <yu...@gmail.com>]

Interesting idea.


On Thu, Aug 22, 2013 at 9:53 AM, Dave Latham <la...@davelink.net> wrote:

> I was recently dealing with a situation of the META table being slammed
> with queries for a long time.  While evaluating the problem I saw the IPCs
> being in a cycle like the following:
>
> HRegionInterface#getClosestRowBefore queueTime=200 processingTime=1 ... 444
> bytes.
> HRegionInterface#openScanner queueTime=191 processingTime=0 ... 22 bytes.
> HRegionInterface#next queueTime=154 processingTime=1 ...  4551 bytes.
> HRegionInterface#close queueTime=216 processingTime=0 ... 16 bytes.
> (rinse, repeat)
>
> I'm bumping our prefetching from 10 rows to 50 rows to help cut down on the
> number of cycles, but it also occurs to me that it's a bit painful to
> require 3 RPCs to do a short scan.
>
> What would people think about adding method for short scans which
> encapsulates (openScanner / next / close) in a single RPC?
>
> Dave
>