[jira] [Commented] (PHOENIX-3188) Making renew leases of scanners more resilient

["James Taylor (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423740#comment-15423740 ] 

James Taylor commented on PHOENIX-3188:
---------------------------------------

Looking for advice from some of you HBasers, [~apurtell] and [~lhofhansl]. This is primarily an issue for aggregate queries which do the following:
# client creates scan that will run over a bunch of rows (say from 300MB to at most an entire region's worth of data, depending on how Phoenix is configured)
# server runs scan to completion in coprocessor
# lease times out waiting for response back from server

Ideally, we'd like to renew the lease from our coprocessor on the server side, at some interval of time as we're doing our server-side scan. Is there anyway to do that today? I can't figure out a way to get my hands on the scanner ID to renew it.

We have code now that attempts to renew the lease from the client side, but the problem is that the ClientScanner next and renewLease calls need to be synchronized, so while next is waiting, we can't call renewLease.

A hacky alternative is to
- get the scanner ID on the client side (we can do this in a hacky way because the member variables are protected).
- do a separate RPC (at a higher priority as Samarth mentioned) from a new endpoint coprocessor that passes through the scannerID. Our endpoint coprocessor can then call renewLease directly (in which case the client would need to continue making these calls at some regular time interval) or it could just populate a map of scanUUID->scannerID that our existing server-side coprocessor doing the scan could look for to enable the renewLease to be done there.

In the meantime, we can figure out the best way in HBase and get supported APIs (or use existing ones we don't know about).


> Making renew leases of scanners more resilient
> ----------------------------------------------
>
>                 Key: PHOENIX-3188
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3188
>             Project: Phoenix
>          Issue Type: Improvement
>            Reporter: Samarth Jain
>
> For renewing scanner leases, we need to perform an RPC per lease renewal. It could happen that under load, these calls won't be picked up by region handler threads soon enough which could cause queries to fail. There are a couple of ways to solve this problem:
> 1) Issue renew lease calls at a higher priority. This essentially causes a different thread pool to be used on the region servers. One such example is the metadata and index updates calls we make. This would scale well unless these special thread pools themselves get saturated with requests.
> 2) Batch up the the renewLease rpc calls. If HBase is able to provide us scanner names and ids, then we can potentially renew leases for multiple scanners by batching them up in one rpc. This would entail changes in both client and server side of HBase. Client side - to expose scanner name. Server side - to expose renewLease call on a scanner given a scanner name/id. 
> We still need to fix renewing leases for non-aggregate queries though. See PHOENIX-1751



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)