You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by rahul gidwani <ra...@gmail.com> on 2016/04/20 22:47:58 UTC
Allowing clients to set priorities
I was wondering if people would be interested in allowing the client to
specify priorities? I really think we are good responsible adults and wont
abuse this feature. :)
This would not just be for one particular operation but all operations.
I'll make it feature complete.
As for batch operations prioirites would be at batch level.
I know the phoenix guys would really like this feature as it would really
help with their indexing work.
Eventually I think it would be nice to get to a point where we can have
some sort of configurable reservation system. Where regionservers could
have handler groups and we could send a little bit more info with the rpc
call to specify the reserved set of handlers they would like to utilize.
thanks
rahul
Re:Re: Allowing clients to set priorities
Posted by WangYQ <wa...@163.com>.
using this feature, we can set priority for different request, sounds useful
At 2016-04-21 14:21:15, "Stack" <st...@duboce.net> wrote:
>On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <ra...@gmail.com>
>wrote:
>
>> I was wondering if people would be interested in allowing the client to
>> specify priorities? I really think we are good responsible adults and wont
>> abuse this feature. :)
>>
>> This would not just be for one particular operation but all operations.
>> I'll make it feature complete.
>>
>>
>Sounds sweet.
>
>RPC passes priority in the header already IIRC.
>
>We could then purge our ugly decompose of the request just to figure what
>it is so we can prioritize based off annotation.
>
>St.Ack
>
>
>
>> As for batch operations prioirites would be at batch level.
>>
>> I know the phoenix guys would really like this feature as it would really
>> help with their indexing work.
>>
>
>
>> Eventually I think it would be nice to get to a point where we can have
>> some sort of configurable reservation system. Where regionservers could
>> have handler groups and we could send a little bit more info with the rpc
>> call to specify the reserved set of handlers they would like to utilize.
>>
>> thanks
>> rahul
>>
Re: Allowing clients to set priorities
Posted by Andrew Purtell <an...@gmail.com>.
I think option one is interesting if we are able to make our theadpools work more like Java executors: there is a reservation of core threads always available, and another limit where a pool can dynamically create more handlers up to some max and terminate when unused. This addresses the utilization question. In the aggregate all the handlers will be serving current workload subject to their absolute limits and the work routing logic. It would not address overcommit issues, though, if and when a user configures too many handler pools with too large max limits.
> On Apr 25, 2016, at 12:47 PM, Mikhail Antonov <ol...@gmail.com> wrote:
>
> An interesting discussion indeed.
>
> "It is related to PHOENIX-938, but also had this desire back in the
> days at Flurry, I recall us always wishing for a way that we could
> reserve a group of handlers for specific users or tables."
>
> Could you tell more about what you wished you had at that moment if
> possible?
> Did you have some requests to a table not meeting desired QoS because all
> handlers were
> occupied? Is that the lack of quotas at RPC threadpool layer that was a
> pain for you?
> Did you use the quotas that are available?
>
> So conceptually I guess I see two approaches (not really mutually
> exclusive) to any latency guaranties in multi-tenant environment as far as
> RPC handling goes:
>
> 1) we can have more theadpools. More buckets for the client to choose
> from, can fine-tune the priorities and noone can eat up more capacity from
> someone else's buffer.
> I assume to be flexible we would need to start/stop new threadpools
> without restarting the server (that goes back the point that we should have
> more settings to be dynamically configurable,
> otherwise optimizing things on prod cluster becomes a pain).
>
> Potential issues I see with that are - we already have 3 threadpools for
> RPC threads, so how much is too much?
> How do we avoid ovecommitting handler threads if we have many
> (dynamically configurable) threadpools?
> How do we make sure they're fully utilized? Do we let them steal the
> Call's from each other if there's spare capacity?
>
> 2) Use bigger and fewer threadpools, but use more advanced logic in
> dequeuing/scheduling, like moving Calls around based on their priorities,
> how clogged the queues are, current load etc?
> Having many queues per threadpool for different calls? In general,
> decouple call queues from threadpools to handle them?
>
> Thoughts?
>
>> On Mon, Apr 25, 2016 at 10:02 AM, Stack <st...@duboce.net> wrote:
>>
>> On Fri, Apr 22, 2016 at 3:41 PM, rahul gidwani <ra...@gmail.com>
>> wrote:
>>
>>> ...
>>> For the exposing priority to the client I was thinking this work goes
>>> into OperationWithAttributes or Operation we can add a setPriority()
>>> and getPriority() and by default they are set to normal priority
>>> unless explicitly overridden. For things like Meta table I am
>>> assuming there will be some resolution system we come up with that
>>> wont be exposed but documented. Maybe the logic can go into
>>> Operation.getPriority() and then you would simply call
>>> controller.setProperty(operation.getPriority). The batch operations
>>> could do something like take the highest priority individual operation
>>> and that become the priority at the entire batch level.
>> I think we need to wire up client setting priority as you suggest above.
>> How it is implemented is another matter. I think it important to cast
>> client 'priority' as an advisory that may or not be respected.
>>
>>
>>> Another option is to piggy back on Matteo's great work on Quotas and
>>> we could add a reservation type system. Maybe similar functionality
>>> and behavior as quotas but phase 1 is to only be able to reserve
>>> handlers for (tables, users, namespaces) - If we go that route, no
>>> need to expose priorities to the clients, they will just refer to some
>>> reservation cache and route them to the appropriate group of handlers.
>>> Because this information would probably be stored in the Quota table
>>> and we need this prior to starting up handlers - this might require
>>> discussions or a design discussion.
>>>
>>> I think going the reservation route might be more elegant (but I need
>>> to hash out a few things before getting a design ready) but I would
>>> love to hear what you guys think.
>> We have to do this too (smile).
>>
>> To my mind, clients having to know about 'reservation caches' or 'handlers'
>> is TMI and another dimension on multitenancy that an operator would control
>> under the wraps rather than user.
>>
>> Start small (smile).
>>
>> Good stuff Rahul,
>> St.Ack
>>
>>
>>
>>> Thanks for discussing this,
>>> rahul
>>>
>>>
>>>
>>>
>>>> On Apr 22, 2016, at 10:35 AM, Stack <st...@duboce.net> wrote:
>>>>
>>>> A few things:
>>>>
>>>> + Our Matteo reminded me of the HBASE-11048 "Support setting custom
>>>> priority per client RPC"/PHOENIX-938 "Use higher priority queue for
>> index
>>>> updates to prevent deadlock" work which adds in a factory so clients
>> can
>>> do
>>>> their rpccontroller. You'd build on this or controllers will always
>> pass
>>>> priority regardless of the rpccontroller implementation? How would your
>>>> work and it relate? You'd surface priority on each method?
>>>> + What are you thinking regards priority on meta? How to not preempt
>> the
>>>> internal workings of hbase itself?
>>>>
>>>> Answers to above can wait till design time, np.
>>>>
>>>> Is there an associated phoenix issue other than PHOENIX-938 that goes
>> w/
>>>> this work?
>>>>
>>>> Thanks Rahul,
>>>> St.Ack
>>>>
>>>> On Fri, Apr 22, 2016 at 9:53 AM, rahul gidwani <
>> rahul.gidwani@gmail.com>
>>>> wrote:
>>>>
>>>>> Sure sorry didn't provide a good example.
>>>>>
>>>>> There are two situations where I have thought this feature might be
>>>>> useful. Maybe more...
>>>>>
>>>>> 1. For something like Phoenix, there are situations where you want
>>>>> certain operations for tables / users to always have handlers
>>>>> available to do work. For example any write to an index table should
>>>>> never block. One way to almost guarantee this is to give it its own
>>>>> special set of handlers and allow from a client call to denote that
>>>>> this call is meant for this specific handler pool.
>>>>>
>>>>> 2. Suppose you have a large cluster with 1 really large table. You
>>>>> can't really use regionserver groups and there are clients doing all
>>>>> sorts of operations on this cluster. Map/Reduce jobs (long scans,
>>>>> heavy reduces), processing pipeline, random reads, etc. There are
>>>>> features already to de-prioritize long running scanners and there is
>>>>> rpc throttling and we can split up the handlers into read, write.
>>>>> Currently there is no easy way to say I want to reserve 10 handlers on
>>>>> each regionserver for the processing pipeline and from the client you
>>>>> can pass something along that would tell the server to use this
>>>>> special handler pool.
>>>>>
>>>>> Also Stack, I saw your TODO and I believe we could get rid of the
>>>>> AnnotatinoReadingPriorityFunction.
>>>>>
>>>>> We can talk about design if folks are interested.
>>>>>
>>>>> Thanks
>>>>> rahul
>>>>>
>>>>>
>>>>>>> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <olorinbant@gmail.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>> This is interesting idea. Sorry if I missed some context - what's the
>>>>>> primary incentive here? What's examples of those categorized thread
>>>>> pools?
>>>>>>
>>>>>> Sounds intersecting a bit with HBASE-15136
>>>>>> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline
>>> scheduling
>>>>>> for RPC requests) in the area of rpc prioritizing.
>>>>>>
>>>>>> -Mikhail
>>>>>>
>>>>>>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <
>>> rahul.gidwani@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I was wondering if people would be interested in allowing the
>> client
>>> to
>>>>>>>> specify priorities? I really think we are good responsible adults
>>> and
>>>>>>> wont
>>>>>>>> abuse this feature. :)
>>>>>>>>
>>>>>>>> This would not just be for one particular operation but all
>>> operations.
>>>>>>>> I'll make it feature complete.
>>>>>>> Sounds sweet.
>>>>>>>
>>>>>>> RPC passes priority in the header already IIRC.
>>>>>>>
>>>>>>> We could then purge our ugly decompose of the request just to figure
>>>>> what
>>>>>>> it is so we can prioritize based off annotation.
>>>>>>>
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> As for batch operations prioirites would be at batch level.
>>>>>>>>
>>>>>>>> I know the phoenix guys would really like this feature as it would
>>>>> really
>>>>>>>> help with their indexing work.
>>>>>>>
>>>>>>>
>>>>>>>> Eventually I think it would be nice to get to a point where we can
>>> have
>>>>>>>> some sort of configurable reservation system. Where regionservers
>>>>> could
>>>>>>>> have handler groups and we could send a little bit more info with
>> the
>>>>> rpc
>>>>>>>> call to specify the reserved set of handlers they would like to
>>>>> utilize.
>>>>>>>>
>>>>>>>> thanks
>>>>>>>> rahul
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Michael Antonov
>
>
>
> --
> Thanks,
> Michael Antonov
Re: Allowing clients to set priorities
Posted by Mikhail Antonov <ol...@gmail.com>.
An interesting discussion indeed.
"It is related to PHOENIX-938, but also had this desire back in the
days at Flurry, I recall us always wishing for a way that we could
reserve a group of handlers for specific users or tables."
Could you tell more about what you wished you had at that moment if
possible?
Did you have some requests to a table not meeting desired QoS because all
handlers were
occupied? Is that the lack of quotas at RPC threadpool layer that was a
pain for you?
Did you use the quotas that are available?
So conceptually I guess I see two approaches (not really mutually
exclusive) to any latency guaranties in multi-tenant environment as far as
RPC handling goes:
1) we can have more theadpools. More buckets for the client to choose
from, can fine-tune the priorities and noone can eat up more capacity from
someone else's buffer.
I assume to be flexible we would need to start/stop new threadpools
without restarting the server (that goes back the point that we should have
more settings to be dynamically configurable,
otherwise optimizing things on prod cluster becomes a pain).
Potential issues I see with that are - we already have 3 threadpools for
RPC threads, so how much is too much?
How do we avoid ovecommitting handler threads if we have many
(dynamically configurable) threadpools?
How do we make sure they're fully utilized? Do we let them steal the
Call's from each other if there's spare capacity?
2) Use bigger and fewer threadpools, but use more advanced logic in
dequeuing/scheduling, like moving Calls around based on their priorities,
how clogged the queues are, current load etc?
Having many queues per threadpool for different calls? In general,
decouple call queues from threadpools to handle them?
Thoughts?
On Mon, Apr 25, 2016 at 10:02 AM, Stack <st...@duboce.net> wrote:
> On Fri, Apr 22, 2016 at 3:41 PM, rahul gidwani <ra...@gmail.com>
> wrote:
>
> > ...
> > For the exposing priority to the client I was thinking this work goes
> > into OperationWithAttributes or Operation we can add a setPriority()
> > and getPriority() and by default they are set to normal priority
> > unless explicitly overridden. For things like Meta table I am
> > assuming there will be some resolution system we come up with that
> > wont be exposed but documented. Maybe the logic can go into
> > Operation.getPriority() and then you would simply call
> > controller.setProperty(operation.getPriority). The batch operations
> > could do something like take the highest priority individual operation
> > and that become the priority at the entire batch level.
> >
> >
> I think we need to wire up client setting priority as you suggest above.
> How it is implemented is another matter. I think it important to cast
> client 'priority' as an advisory that may or not be respected.
>
>
> > Another option is to piggy back on Matteo's great work on Quotas and
> > we could add a reservation type system. Maybe similar functionality
> > and behavior as quotas but phase 1 is to only be able to reserve
> > handlers for (tables, users, namespaces) - If we go that route, no
> > need to expose priorities to the clients, they will just refer to some
> > reservation cache and route them to the appropriate group of handlers.
> > Because this information would probably be stored in the Quota table
> > and we need this prior to starting up handlers - this might require
> > discussions or a design discussion.
> >
> > I think going the reservation route might be more elegant (but I need
> > to hash out a few things before getting a design ready) but I would
> > love to hear what you guys think.
> >
> >
> We have to do this too (smile).
>
> To my mind, clients having to know about 'reservation caches' or 'handlers'
> is TMI and another dimension on multitenancy that an operator would control
> under the wraps rather than user.
>
> Start small (smile).
>
> Good stuff Rahul,
> St.Ack
>
>
>
> > Thanks for discussing this,
> > rahul
> >
> >
> >
> >
> > > On Apr 22, 2016, at 10:35 AM, Stack <st...@duboce.net> wrote:
> > >
> > > A few things:
> > >
> > > + Our Matteo reminded me of the HBASE-11048 "Support setting custom
> > > priority per client RPC"/PHOENIX-938 "Use higher priority queue for
> index
> > > updates to prevent deadlock" work which adds in a factory so clients
> can
> > do
> > > their rpccontroller. You'd build on this or controllers will always
> pass
> > > priority regardless of the rpccontroller implementation? How would your
> > > work and it relate? You'd surface priority on each method?
> > > + What are you thinking regards priority on meta? How to not preempt
> the
> > > internal workings of hbase itself?
> > >
> > > Answers to above can wait till design time, np.
> > >
> > > Is there an associated phoenix issue other than PHOENIX-938 that goes
> w/
> > > this work?
> > >
> > > Thanks Rahul,
> > > St.Ack
> > >
> > > On Fri, Apr 22, 2016 at 9:53 AM, rahul gidwani <
> rahul.gidwani@gmail.com>
> > > wrote:
> > >
> > >> Sure sorry didn't provide a good example.
> > >>
> > >> There are two situations where I have thought this feature might be
> > >> useful. Maybe more...
> > >>
> > >> 1. For something like Phoenix, there are situations where you want
> > >> certain operations for tables / users to always have handlers
> > >> available to do work. For example any write to an index table should
> > >> never block. One way to almost guarantee this is to give it its own
> > >> special set of handlers and allow from a client call to denote that
> > >> this call is meant for this specific handler pool.
> > >>
> > >> 2. Suppose you have a large cluster with 1 really large table. You
> > >> can't really use regionserver groups and there are clients doing all
> > >> sorts of operations on this cluster. Map/Reduce jobs (long scans,
> > >> heavy reduces), processing pipeline, random reads, etc. There are
> > >> features already to de-prioritize long running scanners and there is
> > >> rpc throttling and we can split up the handlers into read, write.
> > >> Currently there is no easy way to say I want to reserve 10 handlers on
> > >> each regionserver for the processing pipeline and from the client you
> > >> can pass something along that would tell the server to use this
> > >> special handler pool.
> > >>
> > >> Also Stack, I saw your TODO and I believe we could get rid of the
> > >> AnnotatinoReadingPriorityFunction.
> > >>
> > >> We can talk about design if folks are interested.
> > >>
> > >> Thanks
> > >> rahul
> > >>
> > >>
> > >>>> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <olorinbant@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>> This is interesting idea. Sorry if I missed some context - what's the
> > >>> primary incentive here? What's examples of those categorized thread
> > >> pools?
> > >>>
> > >>> Sounds intersecting a bit with HBASE-15136
> > >>> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline
> > scheduling
> > >>> for RPC requests) in the area of rpc prioritizing.
> > >>>
> > >>> -Mikhail
> > >>>
> > >>>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <
> > rahul.gidwani@gmail.com
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> I was wondering if people would be interested in allowing the
> client
> > to
> > >>>>> specify priorities? I really think we are good responsible adults
> > and
> > >>>> wont
> > >>>>> abuse this feature. :)
> > >>>>>
> > >>>>> This would not just be for one particular operation but all
> > operations.
> > >>>>> I'll make it feature complete.
> > >>>> Sounds sweet.
> > >>>>
> > >>>> RPC passes priority in the header already IIRC.
> > >>>>
> > >>>> We could then purge our ugly decompose of the request just to figure
> > >> what
> > >>>> it is so we can prioritize based off annotation.
> > >>>>
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>>
> > >>>>> As for batch operations prioirites would be at batch level.
> > >>>>>
> > >>>>> I know the phoenix guys would really like this feature as it would
> > >> really
> > >>>>> help with their indexing work.
> > >>>>
> > >>>>
> > >>>>> Eventually I think it would be nice to get to a point where we can
> > have
> > >>>>> some sort of configurable reservation system. Where regionservers
> > >> could
> > >>>>> have handler groups and we could send a little bit more info with
> the
> > >> rpc
> > >>>>> call to specify the reserved set of handlers they would like to
> > >> utilize.
> > >>>>>
> > >>>>> thanks
> > >>>>> rahul
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Thanks,
> > >>> Michael Antonov
> > >>
> >
>
--
Thanks,
Michael Antonov
Re: Allowing clients to set priorities
Posted by Stack <st...@duboce.net>.
On Fri, Apr 22, 2016 at 3:41 PM, rahul gidwani <ra...@gmail.com>
wrote:
> ...
> For the exposing priority to the client I was thinking this work goes
> into OperationWithAttributes or Operation we can add a setPriority()
> and getPriority() and by default they are set to normal priority
> unless explicitly overridden. For things like Meta table I am
> assuming there will be some resolution system we come up with that
> wont be exposed but documented. Maybe the logic can go into
> Operation.getPriority() and then you would simply call
> controller.setProperty(operation.getPriority). The batch operations
> could do something like take the highest priority individual operation
> and that become the priority at the entire batch level.
>
>
I think we need to wire up client setting priority as you suggest above.
How it is implemented is another matter. I think it important to cast
client 'priority' as an advisory that may or not be respected.
> Another option is to piggy back on Matteo's great work on Quotas and
> we could add a reservation type system. Maybe similar functionality
> and behavior as quotas but phase 1 is to only be able to reserve
> handlers for (tables, users, namespaces) - If we go that route, no
> need to expose priorities to the clients, they will just refer to some
> reservation cache and route them to the appropriate group of handlers.
> Because this information would probably be stored in the Quota table
> and we need this prior to starting up handlers - this might require
> discussions or a design discussion.
>
> I think going the reservation route might be more elegant (but I need
> to hash out a few things before getting a design ready) but I would
> love to hear what you guys think.
>
>
We have to do this too (smile).
To my mind, clients having to know about 'reservation caches' or 'handlers'
is TMI and another dimension on multitenancy that an operator would control
under the wraps rather than user.
Start small (smile).
Good stuff Rahul,
St.Ack
> Thanks for discussing this,
> rahul
>
>
>
>
> > On Apr 22, 2016, at 10:35 AM, Stack <st...@duboce.net> wrote:
> >
> > A few things:
> >
> > + Our Matteo reminded me of the HBASE-11048 "Support setting custom
> > priority per client RPC"/PHOENIX-938 "Use higher priority queue for index
> > updates to prevent deadlock" work which adds in a factory so clients can
> do
> > their rpccontroller. You'd build on this or controllers will always pass
> > priority regardless of the rpccontroller implementation? How would your
> > work and it relate? You'd surface priority on each method?
> > + What are you thinking regards priority on meta? How to not preempt the
> > internal workings of hbase itself?
> >
> > Answers to above can wait till design time, np.
> >
> > Is there an associated phoenix issue other than PHOENIX-938 that goes w/
> > this work?
> >
> > Thanks Rahul,
> > St.Ack
> >
> > On Fri, Apr 22, 2016 at 9:53 AM, rahul gidwani <ra...@gmail.com>
> > wrote:
> >
> >> Sure sorry didn't provide a good example.
> >>
> >> There are two situations where I have thought this feature might be
> >> useful. Maybe more...
> >>
> >> 1. For something like Phoenix, there are situations where you want
> >> certain operations for tables / users to always have handlers
> >> available to do work. For example any write to an index table should
> >> never block. One way to almost guarantee this is to give it its own
> >> special set of handlers and allow from a client call to denote that
> >> this call is meant for this specific handler pool.
> >>
> >> 2. Suppose you have a large cluster with 1 really large table. You
> >> can't really use regionserver groups and there are clients doing all
> >> sorts of operations on this cluster. Map/Reduce jobs (long scans,
> >> heavy reduces), processing pipeline, random reads, etc. There are
> >> features already to de-prioritize long running scanners and there is
> >> rpc throttling and we can split up the handlers into read, write.
> >> Currently there is no easy way to say I want to reserve 10 handlers on
> >> each regionserver for the processing pipeline and from the client you
> >> can pass something along that would tell the server to use this
> >> special handler pool.
> >>
> >> Also Stack, I saw your TODO and I believe we could get rid of the
> >> AnnotatinoReadingPriorityFunction.
> >>
> >> We can talk about design if folks are interested.
> >>
> >> Thanks
> >> rahul
> >>
> >>
> >>>> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <ol...@gmail.com>
> >>> wrote:
> >>>
> >>> This is interesting idea. Sorry if I missed some context - what's the
> >>> primary incentive here? What's examples of those categorized thread
> >> pools?
> >>>
> >>> Sounds intersecting a bit with HBASE-15136
> >>> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline
> scheduling
> >>> for RPC requests) in the area of rpc prioritizing.
> >>>
> >>> -Mikhail
> >>>
> >>>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
> >>>>
> >>>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <
> rahul.gidwani@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> I was wondering if people would be interested in allowing the client
> to
> >>>>> specify priorities? I really think we are good responsible adults
> and
> >>>> wont
> >>>>> abuse this feature. :)
> >>>>>
> >>>>> This would not just be for one particular operation but all
> operations.
> >>>>> I'll make it feature complete.
> >>>> Sounds sweet.
> >>>>
> >>>> RPC passes priority in the header already IIRC.
> >>>>
> >>>> We could then purge our ugly decompose of the request just to figure
> >> what
> >>>> it is so we can prioritize based off annotation.
> >>>>
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>>> As for batch operations prioirites would be at batch level.
> >>>>>
> >>>>> I know the phoenix guys would really like this feature as it would
> >> really
> >>>>> help with their indexing work.
> >>>>
> >>>>
> >>>>> Eventually I think it would be nice to get to a point where we can
> have
> >>>>> some sort of configurable reservation system. Where regionservers
> >> could
> >>>>> have handler groups and we could send a little bit more info with the
> >> rpc
> >>>>> call to specify the reserved set of handlers they would like to
> >> utilize.
> >>>>>
> >>>>> thanks
> >>>>> rahul
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Michael Antonov
> >>
>
Re: Allowing clients to set priorities
Posted by Andrew Purtell <ap...@apache.org>.
I like the idea of reservations. So in this sort of scheme, there would be
separate handler pools and a (hopefully, dynamic) configuration to route
requests to a specific pool? Or maybe a default configuration set by the
site file as done today, but with a runtime facility for adding more
handler pools? We'd keep the routing in a system table and cache it? In a
Phoenix system, it would do stuff the user wouldn't have to worry about?
Maybe at the same time we can make handler pools more like executors:
there's a core set of threads at the ready and an additional amount that
can be created on the fly if the core pool is fully occupied, to be
terminated if unused for some amount of time afterward? Then it's not so
painful if you set up a bunch of pools for a bunch of reservations,
assuming all pools all the time won't be fully committed. Too radical to
ever get back to branch-1 I wonder. Don't let this derail the conversation
if so.
On Fri, Apr 22, 2016 at 3:41 PM, rahul gidwani <ra...@gmail.com>
wrote:
> Sure:
>
> For the first item. Yes that was implemented but there were a few issues.
>
> It is related to PHOENIX-938, but also had this desire back in the
> days at Flurry, I recall us always wishing for a way that we could
> reserve a group of handlers for specific users or tables.
>
> The other issue with the approach that PHOENIX-938 takes it has a
> custom RpcScheduler implementation, which has a LimitedPrivate API,
> thus only guaranteeing binary and source compatibility between source
> versions. The API changed between hbase 1.1 and 1.2. That was one
> issue the other was a custom RpcController is for the clients, which
> is also bad because that is IA.Private.
>
> Additionally the client might know which handler pool it wants to go
> to, but the current method of doing custom controller implementation
> is the fact it isin't configurable. If I wanted to add a new group of
> regionserver handlers, I would have to update all the client and
> server jars to do this. In addition to that we have to infer what the
> intent of the rpc call is by the time we get to the RpcController
> instead of knowing if we did it at the Operation level. The
> DelegatingPayloadCarryingRpcController doesn't know much about the
> intent of the request either, you end up having to hard code static
> maps for special tables in your custom controller implementation.
>
> For the exposing priority to the client I was thinking this work goes
> into OperationWithAttributes or Operation we can add a setPriority()
> and getPriority() and by default they are set to normal priority
> unless explicitly overridden. For things like Meta table I am
> assuming there will be some resolution system we come up with that
> wont be exposed but documented. Maybe the logic can go into
> Operation.getPriority() and then you would simply call
> controller.setProperty(operation.getPriority). The batch operations
> could do something like take the highest priority individual operation
> and that become the priority at the entire batch level.
>
> Another option is to piggy back on Matteo's great work on Quotas and
> we could add a reservation type system. Maybe similar functionality
> and behavior as quotas but phase 1 is to only be able to reserve
> handlers for (tables, users, namespaces) - If we go that route, no
> need to expose priorities to the clients, they will just refer to some
> reservation cache and route them to the appropriate group of handlers.
> Because this information would probably be stored in the Quota table
> and we need this prior to starting up handlers - this might require
> discussions or a design discussion.
>
> I think going the reservation route might be more elegant (but I need
> to hash out a few things before getting a design ready) but I would
> love to hear what you guys think.
>
> Thanks for discussing this,
> rahul
>
>
>
>
> > On Apr 22, 2016, at 10:35 AM, Stack <st...@duboce.net> wrote:
> >
> > A few things:
> >
> > + Our Matteo reminded me of the HBASE-11048 "Support setting custom
> > priority per client RPC"/PHOENIX-938 "Use higher priority queue for index
> > updates to prevent deadlock" work which adds in a factory so clients can
> do
> > their rpccontroller. You'd build on this or controllers will always pass
> > priority regardless of the rpccontroller implementation? How would your
> > work and it relate? You'd surface priority on each method?
> > + What are you thinking regards priority on meta? How to not preempt the
> > internal workings of hbase itself?
> >
> > Answers to above can wait till design time, np.
> >
> > Is there an associated phoenix issue other than PHOENIX-938 that goes w/
> > this work?
> >
> > Thanks Rahul,
> > St.Ack
> >
> > On Fri, Apr 22, 2016 at 9:53 AM, rahul gidwani <ra...@gmail.com>
> > wrote:
> >
> >> Sure sorry didn't provide a good example.
> >>
> >> There are two situations where I have thought this feature might be
> >> useful. Maybe more...
> >>
> >> 1. For something like Phoenix, there are situations where you want
> >> certain operations for tables / users to always have handlers
> >> available to do work. For example any write to an index table should
> >> never block. One way to almost guarantee this is to give it its own
> >> special set of handlers and allow from a client call to denote that
> >> this call is meant for this specific handler pool.
> >>
> >> 2. Suppose you have a large cluster with 1 really large table. You
> >> can't really use regionserver groups and there are clients doing all
> >> sorts of operations on this cluster. Map/Reduce jobs (long scans,
> >> heavy reduces), processing pipeline, random reads, etc. There are
> >> features already to de-prioritize long running scanners and there is
> >> rpc throttling and we can split up the handlers into read, write.
> >> Currently there is no easy way to say I want to reserve 10 handlers on
> >> each regionserver for the processing pipeline and from the client you
> >> can pass something along that would tell the server to use this
> >> special handler pool.
> >>
> >> Also Stack, I saw your TODO and I believe we could get rid of the
> >> AnnotatinoReadingPriorityFunction.
> >>
> >> We can talk about design if folks are interested.
> >>
> >> Thanks
> >> rahul
> >>
> >>
> >>>> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <ol...@gmail.com>
> >>> wrote:
> >>>
> >>> This is interesting idea. Sorry if I missed some context - what's the
> >>> primary incentive here? What's examples of those categorized thread
> >> pools?
> >>>
> >>> Sounds intersecting a bit with HBASE-15136
> >>> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline
> scheduling
> >>> for RPC requests) in the area of rpc prioritizing.
> >>>
> >>> -Mikhail
> >>>
> >>>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
> >>>>
> >>>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <
> rahul.gidwani@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> I was wondering if people would be interested in allowing the client
> to
> >>>>> specify priorities? I really think we are good responsible adults
> and
> >>>> wont
> >>>>> abuse this feature. :)
> >>>>>
> >>>>> This would not just be for one particular operation but all
> operations.
> >>>>> I'll make it feature complete.
> >>>> Sounds sweet.
> >>>>
> >>>> RPC passes priority in the header already IIRC.
> >>>>
> >>>> We could then purge our ugly decompose of the request just to figure
> >> what
> >>>> it is so we can prioritize based off annotation.
> >>>>
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>>> As for batch operations prioirites would be at batch level.
> >>>>>
> >>>>> I know the phoenix guys would really like this feature as it would
> >> really
> >>>>> help with their indexing work.
> >>>>
> >>>>
> >>>>> Eventually I think it would be nice to get to a point where we can
> have
> >>>>> some sort of configurable reservation system. Where regionservers
> >> could
> >>>>> have handler groups and we could send a little bit more info with the
> >> rpc
> >>>>> call to specify the reserved set of handlers they would like to
> >> utilize.
> >>>>>
> >>>>> thanks
> >>>>> rahul
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Michael Antonov
> >>
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Allowing clients to set priorities
Posted by rahul gidwani <ra...@gmail.com>.
Sure:
For the first item. Yes that was implemented but there were a few issues.
It is related to PHOENIX-938, but also had this desire back in the
days at Flurry, I recall us always wishing for a way that we could
reserve a group of handlers for specific users or tables.
The other issue with the approach that PHOENIX-938 takes it has a
custom RpcScheduler implementation, which has a LimitedPrivate API,
thus only guaranteeing binary and source compatibility between source
versions. The API changed between hbase 1.1 and 1.2. That was one
issue the other was a custom RpcController is for the clients, which
is also bad because that is IA.Private.
Additionally the client might know which handler pool it wants to go
to, but the current method of doing custom controller implementation
is the fact it isin't configurable. If I wanted to add a new group of
regionserver handlers, I would have to update all the client and
server jars to do this. In addition to that we have to infer what the
intent of the rpc call is by the time we get to the RpcController
instead of knowing if we did it at the Operation level. The
DelegatingPayloadCarryingRpcController doesn't know much about the
intent of the request either, you end up having to hard code static
maps for special tables in your custom controller implementation.
For the exposing priority to the client I was thinking this work goes
into OperationWithAttributes or Operation we can add a setPriority()
and getPriority() and by default they are set to normal priority
unless explicitly overridden. For things like Meta table I am
assuming there will be some resolution system we come up with that
wont be exposed but documented. Maybe the logic can go into
Operation.getPriority() and then you would simply call
controller.setProperty(operation.getPriority). The batch operations
could do something like take the highest priority individual operation
and that become the priority at the entire batch level.
Another option is to piggy back on Matteo's great work on Quotas and
we could add a reservation type system. Maybe similar functionality
and behavior as quotas but phase 1 is to only be able to reserve
handlers for (tables, users, namespaces) - If we go that route, no
need to expose priorities to the clients, they will just refer to some
reservation cache and route them to the appropriate group of handlers.
Because this information would probably be stored in the Quota table
and we need this prior to starting up handlers - this might require
discussions or a design discussion.
I think going the reservation route might be more elegant (but I need
to hash out a few things before getting a design ready) but I would
love to hear what you guys think.
Thanks for discussing this,
rahul
> On Apr 22, 2016, at 10:35 AM, Stack <st...@duboce.net> wrote:
>
> A few things:
>
> + Our Matteo reminded me of the HBASE-11048 "Support setting custom
> priority per client RPC"/PHOENIX-938 "Use higher priority queue for index
> updates to prevent deadlock" work which adds in a factory so clients can do
> their rpccontroller. You'd build on this or controllers will always pass
> priority regardless of the rpccontroller implementation? How would your
> work and it relate? You'd surface priority on each method?
> + What are you thinking regards priority on meta? How to not preempt the
> internal workings of hbase itself?
>
> Answers to above can wait till design time, np.
>
> Is there an associated phoenix issue other than PHOENIX-938 that goes w/
> this work?
>
> Thanks Rahul,
> St.Ack
>
> On Fri, Apr 22, 2016 at 9:53 AM, rahul gidwani <ra...@gmail.com>
> wrote:
>
>> Sure sorry didn't provide a good example.
>>
>> There are two situations where I have thought this feature might be
>> useful. Maybe more...
>>
>> 1. For something like Phoenix, there are situations where you want
>> certain operations for tables / users to always have handlers
>> available to do work. For example any write to an index table should
>> never block. One way to almost guarantee this is to give it its own
>> special set of handlers and allow from a client call to denote that
>> this call is meant for this specific handler pool.
>>
>> 2. Suppose you have a large cluster with 1 really large table. You
>> can't really use regionserver groups and there are clients doing all
>> sorts of operations on this cluster. Map/Reduce jobs (long scans,
>> heavy reduces), processing pipeline, random reads, etc. There are
>> features already to de-prioritize long running scanners and there is
>> rpc throttling and we can split up the handlers into read, write.
>> Currently there is no easy way to say I want to reserve 10 handlers on
>> each regionserver for the processing pipeline and from the client you
>> can pass something along that would tell the server to use this
>> special handler pool.
>>
>> Also Stack, I saw your TODO and I believe we could get rid of the
>> AnnotatinoReadingPriorityFunction.
>>
>> We can talk about design if folks are interested.
>>
>> Thanks
>> rahul
>>
>>
>>>> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <ol...@gmail.com>
>>> wrote:
>>>
>>> This is interesting idea. Sorry if I missed some context - what's the
>>> primary incentive here? What's examples of those categorized thread
>> pools?
>>>
>>> Sounds intersecting a bit with HBASE-15136
>>> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline scheduling
>>> for RPC requests) in the area of rpc prioritizing.
>>>
>>> -Mikhail
>>>
>>>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <rahul.gidwani@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> I was wondering if people would be interested in allowing the client to
>>>>> specify priorities? I really think we are good responsible adults and
>>>> wont
>>>>> abuse this feature. :)
>>>>>
>>>>> This would not just be for one particular operation but all operations.
>>>>> I'll make it feature complete.
>>>> Sounds sweet.
>>>>
>>>> RPC passes priority in the header already IIRC.
>>>>
>>>> We could then purge our ugly decompose of the request just to figure
>> what
>>>> it is so we can prioritize based off annotation.
>>>>
>>>> St.Ack
>>>>
>>>>
>>>>
>>>>> As for batch operations prioirites would be at batch level.
>>>>>
>>>>> I know the phoenix guys would really like this feature as it would
>> really
>>>>> help with their indexing work.
>>>>
>>>>
>>>>> Eventually I think it would be nice to get to a point where we can have
>>>>> some sort of configurable reservation system. Where regionservers
>> could
>>>>> have handler groups and we could send a little bit more info with the
>> rpc
>>>>> call to specify the reserved set of handlers they would like to
>> utilize.
>>>>>
>>>>> thanks
>>>>> rahul
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Michael Antonov
>>
Re: Allowing clients to set priorities
Posted by Stack <st...@duboce.net>.
A few things:
+ Our Matteo reminded me of the HBASE-11048 "Support setting custom
priority per client RPC"/PHOENIX-938 "Use higher priority queue for index
updates to prevent deadlock" work which adds in a factory so clients can do
their rpccontroller. You'd build on this or controllers will always pass
priority regardless of the rpccontroller implementation? How would your
work and it relate? You'd surface priority on each method?
+ What are you thinking regards priority on meta? How to not preempt the
internal workings of hbase itself?
Answers to above can wait till design time, np.
Is there an associated phoenix issue other than PHOENIX-938 that goes w/
this work?
Thanks Rahul,
St.Ack
On Fri, Apr 22, 2016 at 9:53 AM, rahul gidwani <ra...@gmail.com>
wrote:
> Sure sorry didn't provide a good example.
>
> There are two situations where I have thought this feature might be
> useful. Maybe more...
>
> 1. For something like Phoenix, there are situations where you want
> certain operations for tables / users to always have handlers
> available to do work. For example any write to an index table should
> never block. One way to almost guarantee this is to give it its own
> special set of handlers and allow from a client call to denote that
> this call is meant for this specific handler pool.
>
> 2. Suppose you have a large cluster with 1 really large table. You
> can't really use regionserver groups and there are clients doing all
> sorts of operations on this cluster. Map/Reduce jobs (long scans,
> heavy reduces), processing pipeline, random reads, etc. There are
> features already to de-prioritize long running scanners and there is
> rpc throttling and we can split up the handlers into read, write.
> Currently there is no easy way to say I want to reserve 10 handlers on
> each regionserver for the processing pipeline and from the client you
> can pass something along that would tell the server to use this
> special handler pool.
>
> Also Stack, I saw your TODO and I believe we could get rid of the
> AnnotatinoReadingPriorityFunction.
>
> We can talk about design if folks are interested.
>
> Thanks
> rahul
>
>
> > On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <ol...@gmail.com>
> wrote:
> >
> > This is interesting idea. Sorry if I missed some context - what's the
> > primary incentive here? What's examples of those categorized thread
> pools?
> >
> > Sounds intersecting a bit with HBASE-15136
> > <https://issues.apache.org/jira/browse/HBASE-15136> (deadline scheduling
> > for RPC requests) in the area of rpc prioritizing.
> >
> > -Mikhail
> >
> >> On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
> >>
> >> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <rahul.gidwani@gmail.com
> >
> >> wrote:
> >>
> >>> I was wondering if people would be interested in allowing the client to
> >>> specify priorities? I really think we are good responsible adults and
> >> wont
> >>> abuse this feature. :)
> >>>
> >>> This would not just be for one particular operation but all operations.
> >>> I'll make it feature complete.
> >> Sounds sweet.
> >>
> >> RPC passes priority in the header already IIRC.
> >>
> >> We could then purge our ugly decompose of the request just to figure
> what
> >> it is so we can prioritize based off annotation.
> >>
> >> St.Ack
> >>
> >>
> >>
> >>> As for batch operations prioirites would be at batch level.
> >>>
> >>> I know the phoenix guys would really like this feature as it would
> really
> >>> help with their indexing work.
> >>
> >>
> >>> Eventually I think it would be nice to get to a point where we can have
> >>> some sort of configurable reservation system. Where regionservers
> could
> >>> have handler groups and we could send a little bit more info with the
> rpc
> >>> call to specify the reserved set of handlers they would like to
> utilize.
> >>>
> >>> thanks
> >>> rahul
> >
> >
> >
> > --
> > Thanks,
> > Michael Antonov
>
Re: Allowing clients to set priorities
Posted by rahul gidwani <ra...@gmail.com>.
Sure sorry didn't provide a good example.
There are two situations where I have thought this feature might be
useful. Maybe more...
1. For something like Phoenix, there are situations where you want
certain operations for tables / users to always have handlers
available to do work. For example any write to an index table should
never block. One way to almost guarantee this is to give it its own
special set of handlers and allow from a client call to denote that
this call is meant for this specific handler pool.
2. Suppose you have a large cluster with 1 really large table. You
can't really use regionserver groups and there are clients doing all
sorts of operations on this cluster. Map/Reduce jobs (long scans,
heavy reduces), processing pipeline, random reads, etc. There are
features already to de-prioritize long running scanners and there is
rpc throttling and we can split up the handlers into read, write.
Currently there is no easy way to say I want to reserve 10 handlers on
each regionserver for the processing pipeline and from the client you
can pass something along that would tell the server to use this
special handler pool.
Also Stack, I saw your TODO and I believe we could get rid of the
AnnotatinoReadingPriorityFunction.
We can talk about design if folks are interested.
Thanks
rahul
> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <ol...@gmail.com> wrote:
>
> This is interesting idea. Sorry if I missed some context - what's the
> primary incentive here? What's examples of those categorized thread pools?
>
> Sounds intersecting a bit with HBASE-15136
> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline scheduling
> for RPC requests) in the area of rpc prioritizing.
>
> -Mikhail
>
>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
>>
>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <ra...@gmail.com>
>> wrote:
>>
>>> I was wondering if people would be interested in allowing the client to
>>> specify priorities? I really think we are good responsible adults and
>> wont
>>> abuse this feature. :)
>>>
>>> This would not just be for one particular operation but all operations.
>>> I'll make it feature complete.
>> Sounds sweet.
>>
>> RPC passes priority in the header already IIRC.
>>
>> We could then purge our ugly decompose of the request just to figure what
>> it is so we can prioritize based off annotation.
>>
>> St.Ack
>>
>>
>>
>>> As for batch operations prioirites would be at batch level.
>>>
>>> I know the phoenix guys would really like this feature as it would really
>>> help with their indexing work.
>>
>>
>>> Eventually I think it would be nice to get to a point where we can have
>>> some sort of configurable reservation system. Where regionservers could
>>> have handler groups and we could send a little bit more info with the rpc
>>> call to specify the reserved set of handlers they would like to utilize.
>>>
>>> thanks
>>> rahul
>
>
>
> --
> Thanks,
> Michael Antonov
Re: Allowing clients to set priorities
Posted by Mikhail Antonov <ol...@gmail.com>.
This is interesting idea. Sorry if I missed some context - what's the
primary incentive here? What's examples of those categorized thread pools?
Sounds intersecting a bit with HBASE-15136
<https://issues.apache.org/jira/browse/HBASE-15136> (deadline scheduling
for RPC requests) in the area of rpc prioritizing.
-Mikhail
On Wed, Apr 20, 2016 at 11:21 PM, Stack <st...@duboce.net> wrote:
> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <ra...@gmail.com>
> wrote:
>
> > I was wondering if people would be interested in allowing the client to
> > specify priorities? I really think we are good responsible adults and
> wont
> > abuse this feature. :)
> >
> > This would not just be for one particular operation but all operations.
> > I'll make it feature complete.
> >
> >
> Sounds sweet.
>
> RPC passes priority in the header already IIRC.
>
> We could then purge our ugly decompose of the request just to figure what
> it is so we can prioritize based off annotation.
>
> St.Ack
>
>
>
> > As for batch operations prioirites would be at batch level.
> >
> > I know the phoenix guys would really like this feature as it would really
> > help with their indexing work.
> >
>
>
> > Eventually I think it would be nice to get to a point where we can have
> > some sort of configurable reservation system. Where regionservers could
> > have handler groups and we could send a little bit more info with the rpc
> > call to specify the reserved set of handlers they would like to utilize.
> >
> > thanks
> > rahul
> >
>
--
Thanks,
Michael Antonov
Re: Allowing clients to set priorities
Posted by Stack <st...@duboce.net>.
On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <ra...@gmail.com>
wrote:
> I was wondering if people would be interested in allowing the client to
> specify priorities? I really think we are good responsible adults and wont
> abuse this feature. :)
>
> This would not just be for one particular operation but all operations.
> I'll make it feature complete.
>
>
Sounds sweet.
RPC passes priority in the header already IIRC.
We could then purge our ugly decompose of the request just to figure what
it is so we can prioritize based off annotation.
St.Ack
> As for batch operations prioirites would be at batch level.
>
> I know the phoenix guys would really like this feature as it would really
> help with their indexing work.
>
> Eventually I think it would be nice to get to a point where we can have
> some sort of configurable reservation system. Where regionservers could
> have handler groups and we could send a little bit more info with the rpc
> call to specify the reserved set of handlers they would like to utilize.
>
> thanks
> rahul
>