Regarding QoS for Handlers

[Varun Sharma <va...@pinterest.com>]

Hi,

I am wondering if QoS on handler threads is being looked at by someone ? We
have found huge gains by doing this for asymmetric workloads which are
extremely write heavy but latency on reads matters the most. Basically we
forked out a separate thread pool of X threads for handling reads to
prevent starvation of read requests which were far outnumbered by the # of
writes.

As of now, we have been making adhoc changes (like introduce higher
priority levels for certain types of requests like gets and scans) but
having a framework to control this would be a really nice thing.

One simple way to achieve this would be to have a group of "high priority
handler threads" and be able to mark each table to prioritize
"reads/writes". On receiving an RPC, the request is directly dispatched to
this other pool of threads and the remaining RPCs go to the regular pool.
This would be a naive implementation.

The issue with this could be that what happens when the user ends up
directing most of their load to the high priority pool which has fewer
threads. We could do something simple like having a tight upper bound on
the call queue length and if a new high priority call is rejected from this
pool, just enqueue it to the regular pool of requests.

Thoughts ?

Thanks !
Varun

Re: Regarding QoS for Handlers

[Devaraj Das <dd...@hortonworks.com>]

Varun, it'll be good to have your inputs on HBASE-8836 which I think is
related to what you are talking about.


On Sun, Oct 13, 2013 at 1:54 PM, Varun Sharma <va...@pinterest.com> wrote:

> Or a slightly alternative approach would be to have a CallQueue
> implementation which is more of a priorityQueue.
>
> The Queue wrapper has a number of different queues [TableName:RpcName]
> sorted in descending order by priority level and finally the default quue.
> Dequeue on this queue shall go in descending order of priority.
>
> Varun
>
>
> On Sun, Oct 13, 2013 at 1:36 PM, Varun Sharma <va...@pinterest.com> wrote:
>
> > Hi,
> >
> > I am wondering if QoS on handler threads is being looked at by someone ?
> > We have found huge gains by doing this for asymmetric workloads which are
> > extremely write heavy but latency on reads matters the most. Basically we
> > forked out a separate thread pool of X threads for handling reads to
> > prevent starvation of read requests which were far outnumbered by the #
> of
> > writes.
> >
> > As of now, we have been making adhoc changes (like introduce higher
> > priority levels for certain types of requests like gets and scans) but
> > having a framework to control this would be a really nice thing.
> >
> > One simple way to achieve this would be to have a group of "high priority
> > handler threads" and be able to mark each table to prioritize
> > "reads/writes". On receiving an RPC, the request is directly dispatched
> to
> > this other pool of threads and the remaining RPCs go to the regular pool.
> > This would be a naive implementation.
> >
> > The issue with this could be that what happens when the user ends up
> > directing most of their load to the high priority pool which has fewer
> > threads. We could do something simple like having a tight upper bound on
> > the call queue length and if a new high priority call is rejected from
> this
> > pool, just enqueue it to the regular pool of requests.
> >
> > Thoughts ?
> >
> > Thanks !
> > Varun
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Regarding QoS for Handlers

[Varun Sharma <va...@pinterest.com>]

Or a slightly alternative approach would be to have a CallQueue
implementation which is more of a priorityQueue.

The Queue wrapper has a number of different queues [TableName:RpcName]
sorted in descending order by priority level and finally the default quue.
Dequeue on this queue shall go in descending order of priority.

Varun


On Sun, Oct 13, 2013 at 1:36 PM, Varun Sharma <va...@pinterest.com> wrote:

> Hi,
>
> I am wondering if QoS on handler threads is being looked at by someone ?
> We have found huge gains by doing this for asymmetric workloads which are
> extremely write heavy but latency on reads matters the most. Basically we
> forked out a separate thread pool of X threads for handling reads to
> prevent starvation of read requests which were far outnumbered by the # of
> writes.
>
> As of now, we have been making adhoc changes (like introduce higher
> priority levels for certain types of requests like gets and scans) but
> having a framework to control this would be a really nice thing.
>
> One simple way to achieve this would be to have a group of "high priority
> handler threads" and be able to mark each table to prioritize
> "reads/writes". On receiving an RPC, the request is directly dispatched to
> this other pool of threads and the remaining RPCs go to the regular pool.
> This would be a naive implementation.
>
> The issue with this could be that what happens when the user ends up
> directing most of their load to the high priority pool which has fewer
> threads. We could do something simple like having a tight upper bound on
> the call queue length and if a new high priority call is rejected from this
> pool, just enqueue it to the regular pool of requests.
>
> Thoughts ?
>
> Thanks !
> Varun
>

Re: Regarding QoS for Handlers

[Stack <st...@duboce.net>]

On Sun, Oct 13, 2013 at 1:36 PM, Varun Sharma <va...@pinterest.com> wrote:

> Hi,
>
> I am wondering if QoS on handler threads is being looked at by someone ?



What Devaraj said plus, trunk has HBASE-8884 "Pluggable RpcScheduler"
committed w/ two implementations currently -- our current coarse QoS that
works off method annotations and target region name and a new FIFO schedule
for the master RPCs.  The authors, like yourselves, were motivated by
desire for a treatment that was other than our default prioritization
wanting to demark on whether the incoming RPC was a read or a write
(Beware, backport would be messy -- there was a few follow-on patches after
the initial commit).

...



> The issue with this could be that what happens when the user ends up
> directing most of their load to the high priority pool which has fewer
> threads. We could do something simple like having a tight upper bound on
> the call queue length and if a new high priority call is rejected from this
> pool, just enqueue it to the regular pool of requests.
>
> Thoughts ?
>


All sounds good to me Varun.

+ At a minimum we need a means of second-classing mapreduce workload.
+ Naive is fine.  Beyond this, the problem gets complicated fast (LarsG and
AndrewWang talk at HBaseCon)
+ We just added 'priority' to the RPC header so client can say what
priority they want (Server can do what it likes with it).

St.Ack