You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by ashish singhi <as...@huawei.com> on 2015/05/28 08:26:39 UTC
Scan priority based on type of request origin
Hello folks.
We had a customer scenario to support scan priority based on type of request origin. Consider the scenario , where in n number of scan requests are generated from different clients. Some of these requests are high priority as they are originated from high privileged users OR from data viewer Web UI etc. Such request should not wait and must finish execution at the earliest compare to other requests.
We went through the code and found that HBase support for de-prioritizing long running scanner(HBASE-10993) and providing super user rpc requests higher priority comparing to other users(HBASE-13375). So this means apart from super user all other user request will be treated as same and with normal QOS.
Let us know if there are any existing solutions already available which may be we are missing.
If implementing this make sense , we can implement this in community.
Regards,
Ashish Singhi
Re: Scan priority based on type of request origin
Posted by Vladimir Rodionov <vl...@gmail.com>.
Scanner prioritization is not sufficient, there are a lot of other IO
activity (compaction, flushing, logging), which can severely affect
high priority scanners.
HBase IO scheduler anyone?
On Wed, May 27, 2015 at 11:26 PM, ashish singhi <as...@huawei.com>
wrote:
> Hello folks.
>
> We had a customer scenario to support scan priority based on type of
> request origin. Consider the scenario , where in n number of scan requests
> are generated from different clients. Some of these requests are high
> priority as they are originated from high privileged users OR from data
> viewer Web UI etc. Such request should not wait and must finish execution
> at the earliest compare to other requests.
>
> We went through the code and found that HBase support for de-prioritizing
> long running scanner(HBASE-10993) and providing super user rpc requests
> higher priority comparing to other users(HBASE-13375). So this means apart
> from super user all other user request will be treated as same and with
> normal QOS.
>
> Let us know if there are any existing solutions already available which
> may be we are missing.
>
> If implementing this make sense , we can implement this in community.
>
> Regards,
> Ashish Singhi
>
RE: Scan priority based on type of request origin
Posted by ashish singhi <as...@huawei.com>.
For both the question answer is same i.e., it will based on the time.
Regards,
Ashish Singhi
-----Original Message-----
From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
Sent: 01 June 2015 15:51
To: dev@hbase.apache.org
Subject: Re: Scan priority based on type of request origin
That is right. So if your implementation is just based on the priority of the user then the queue will be fetching the request coming out of a higher priority user always.
If there are multiple requests from the higher priority user/users - how will you distinguish between those requests? Within users of same priority you will again categorise it based on time?
Regards
Ram
On Mon, Jun 1, 2015 at 3:11 PM, ashish singhi <as...@huawei.com>
wrote:
> @Ram:
> > I would say the deadline based impl would queue up the one with 0
> deadline as the prioritized one rather than the normal one.
> No. The actual deadline also includes the current timestamp. So the
> request which has before timestamp will be first in the queue in this case.
>
> > But if you want to implement any priority based scheduling then you
> > can
> have a new scheduler and make your own queue implementation out of it.
> Yeah.
>
> Regards,
> Ashish Singhi
>
>
> -----Original Message-----
> From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> Sent: 28 May 2015 23:43
> To: dev@hbase.apache.org
> Subject: Re: Scan priority based on type of request origin
>
> bq.Consider a case when there is one normal priority request
> (submitted at
> t1) with deadline X in the queue and one more high priority request
> (Submitted at t2 >= t1 + x ) with zero deadline. In the current
> implementation the normal priority request will be offered from the
> queue
>
> I would say the deadline based impl would queue up the one with 0
> deadline as the prioritized one rather than the normal one. See this
> BoundedPriorityBlockingQueue. Am I missing something here?
>
> But if you want to implement any priority based scheduling then you
> can have a new scheduler and make your own queue implementation out of it.
>
> On Thu, May 28, 2015 at 3:08 PM, ashish singhi
> <as...@huawei.com>
> wrote:
>
> > Thanks for the quick responses.
> >
> > @Anoop:
> > Yes, we were also thinking of implementing it along the same lines.
> >
> > @Ram:
> > High priority request should be always given preference over all
> > other scan request.
> > As per my understanding in the current implementation of deadline
> > the scenario cannot be satisfied. Consider a case when there is one
> > normal priority request (submitted at t1) with deadline X in the
> > queue and one more high priority request (Submitted at t2 >= t1 + x
> > ) with zero
> deadline.
> > In the current implementation the normal priority request will be
> > offered from the queue. But our scenario require high priority
> > request to be offered. So there is a need of some more decision
> > parameters (such as
> > priority) in the Scheduler logic.
> > Please correct me If I am missing something.
> >
> > Vladimir Rodionov wrote
> > > Scanner prioritization is not sufficient, there are a lot of other
> > > IO
> > activity (compaction, flushing, logging), which can severely affect
> > high priority scanners.
> > >
> > > HBase IO scheduler anyone?
> >
> > Yes I agree, Scan priority alone is not sufficient enough but its
> > required at first place. With stuffs like IO scheduler the solution
> > will
> be complete.
> >
> > Regards,
> > Ashish Singhi
> >
> >
> > -----Original Message-----
> > From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> > Sent: 28 May 2015 12:18
> > To: dev@hbase.apache.org
> > Subject: Re: Scan priority based on type of request origin
> >
> > Atleast in trunk it uses SimpleRPCScheduler and internally it tries
> > to use the Priority based queue so that based on the deadline set
> > for every request it can schedule the highest priority RPC calls.
> >
> > I would think that your current requirement may fit with the
> > existing
> one?
> > Except that those calls should be given higher prioirty or rather a
> > deadline such it is scheduled first?
> >
> > On Thu, May 28, 2015 at 12:14 PM, Anoop John <an...@gmail.com>
> > wrote:
> >
> > > We have pluggable RPC scheduler now. See config
> > > 'hbase.region.server.rpc.scheduler.factory.class'
> > >
> > > By default a SimpleRpcScheduler is in use and there as you said,
> > > there are prioritization for super user. Other prioritization is
> > > based on the RS side method prioritization based on annotation.
> > >
> > > You can see how/whether u can have a custom RPC scheduler and
> > > queuing mechanism using which you can fulfill this.
> > >
> > > -Anoop-
> > >
> > > On Thu, May 28, 2015 at 11:56 AM, ashish singhi
> > > <as...@huawei.com>
> > > wrote:
> > >
> > > > Hello folks.
> > > >
> > > > We had a customer scenario to support scan priority based on
> > > > type of request origin. Consider the scenario , where in n
> > > > number of scan
> > > requests
> > > > are generated from different clients. Some of these requests are
> > > > high priority as they are originated from high privileged users
> > > > OR from data viewer Web UI etc. Such request should not wait and
> > > > must finish execution at the earliest compare to other requests.
> > > >
> > > > We went through the code and found that HBase support for
> > > > de-prioritizing long running scanner(HBASE-10993) and providing
> > > > super user rpc requests higher priority comparing to other
> > > > users(HBASE-13375). So this means
> > > apart
> > > > from super user all other user request will be treated as same
> > > > and with normal QOS.
> > > >
> > > > Let us know if there are any existing solutions already
> > > > available which may be we are missing.
> > > >
> > > > If implementing this make sense , we can implement this in community.
> > > >
> > > > Regards,
> > > > Ashish Singhi
> > > >
> > >
> >
>
Re: Scan priority based on type of request origin
Posted by ramkrishna vasudevan <ra...@gmail.com>.
That is right. So if your implementation is just based on the priority of
the user then the queue will be fetching the request coming out of a higher
priority user always.
If there are multiple requests from the higher priority user/users - how
will you distinguish between those requests? Within users of same priority
you will again categorise it based on time?
Regards
Ram
On Mon, Jun 1, 2015 at 3:11 PM, ashish singhi <as...@huawei.com>
wrote:
> @Ram:
> > I would say the deadline based impl would queue up the one with 0
> deadline as the prioritized one rather than the normal one.
> No. The actual deadline also includes the current timestamp. So the
> request which has before timestamp will be first in the queue in this case.
>
> > But if you want to implement any priority based scheduling then you can
> have a new scheduler and make your own queue implementation out of it.
> Yeah.
>
> Regards,
> Ashish Singhi
>
>
> -----Original Message-----
> From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> Sent: 28 May 2015 23:43
> To: dev@hbase.apache.org
> Subject: Re: Scan priority based on type of request origin
>
> bq.Consider a case when there is one normal priority request (submitted at
> t1) with deadline X in the queue and one more high priority request
> (Submitted at t2 >= t1 + x ) with zero deadline. In the current
> implementation the normal priority request will be offered from the queue
>
> I would say the deadline based impl would queue up the one with 0 deadline
> as the prioritized one rather than the normal one. See this
> BoundedPriorityBlockingQueue. Am I missing something here?
>
> But if you want to implement any priority based scheduling then you can
> have a new scheduler and make your own queue implementation out of it.
>
> On Thu, May 28, 2015 at 3:08 PM, ashish singhi <as...@huawei.com>
> wrote:
>
> > Thanks for the quick responses.
> >
> > @Anoop:
> > Yes, we were also thinking of implementing it along the same lines.
> >
> > @Ram:
> > High priority request should be always given preference over all other
> > scan request.
> > As per my understanding in the current implementation of deadline the
> > scenario cannot be satisfied. Consider a case when there is one normal
> > priority request (submitted at t1) with deadline X in the queue and
> > one more high priority request (Submitted at t2 >= t1 + x ) with zero
> deadline.
> > In the current implementation the normal priority request will be
> > offered from the queue. But our scenario require high priority request
> > to be offered. So there is a need of some more decision parameters
> > (such as
> > priority) in the Scheduler logic.
> > Please correct me If I am missing something.
> >
> > Vladimir Rodionov wrote
> > > Scanner prioritization is not sufficient, there are a lot of other
> > > IO
> > activity (compaction, flushing, logging), which can severely affect
> > high priority scanners.
> > >
> > > HBase IO scheduler anyone?
> >
> > Yes I agree, Scan priority alone is not sufficient enough but its
> > required at first place. With stuffs like IO scheduler the solution will
> be complete.
> >
> > Regards,
> > Ashish Singhi
> >
> >
> > -----Original Message-----
> > From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> > Sent: 28 May 2015 12:18
> > To: dev@hbase.apache.org
> > Subject: Re: Scan priority based on type of request origin
> >
> > Atleast in trunk it uses SimpleRPCScheduler and internally it tries to
> > use the Priority based queue so that based on the deadline set for
> > every request it can schedule the highest priority RPC calls.
> >
> > I would think that your current requirement may fit with the existing
> one?
> > Except that those calls should be given higher prioirty or rather a
> > deadline such it is scheduled first?
> >
> > On Thu, May 28, 2015 at 12:14 PM, Anoop John <an...@gmail.com>
> > wrote:
> >
> > > We have pluggable RPC scheduler now. See config
> > > 'hbase.region.server.rpc.scheduler.factory.class'
> > >
> > > By default a SimpleRpcScheduler is in use and there as you said,
> > > there are prioritization for super user. Other prioritization is
> > > based on the RS side method prioritization based on annotation.
> > >
> > > You can see how/whether u can have a custom RPC scheduler and
> > > queuing mechanism using which you can fulfill this.
> > >
> > > -Anoop-
> > >
> > > On Thu, May 28, 2015 at 11:56 AM, ashish singhi
> > > <as...@huawei.com>
> > > wrote:
> > >
> > > > Hello folks.
> > > >
> > > > We had a customer scenario to support scan priority based on type
> > > > of request origin. Consider the scenario , where in n number of
> > > > scan
> > > requests
> > > > are generated from different clients. Some of these requests are
> > > > high priority as they are originated from high privileged users OR
> > > > from data viewer Web UI etc. Such request should not wait and must
> > > > finish execution at the earliest compare to other requests.
> > > >
> > > > We went through the code and found that HBase support for
> > > > de-prioritizing long running scanner(HBASE-10993) and providing
> > > > super user rpc requests higher priority comparing to other
> > > > users(HBASE-13375). So this means
> > > apart
> > > > from super user all other user request will be treated as same and
> > > > with normal QOS.
> > > >
> > > > Let us know if there are any existing solutions already available
> > > > which may be we are missing.
> > > >
> > > > If implementing this make sense , we can implement this in community.
> > > >
> > > > Regards,
> > > > Ashish Singhi
> > > >
> > >
> >
>
RE: Scan priority based on type of request origin
Posted by ashish singhi <as...@huawei.com>.
@Ram:
> I would say the deadline based impl would queue up the one with 0 deadline as the prioritized one rather than the normal one.
No. The actual deadline also includes the current timestamp. So the request which has before timestamp will be first in the queue in this case.
> But if you want to implement any priority based scheduling then you can have a new scheduler and make your own queue implementation out of it.
Yeah.
Regards,
Ashish Singhi
-----Original Message-----
From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
Sent: 28 May 2015 23:43
To: dev@hbase.apache.org
Subject: Re: Scan priority based on type of request origin
bq.Consider a case when there is one normal priority request (submitted at
t1) with deadline X in the queue and one more high priority request (Submitted at t2 >= t1 + x ) with zero deadline. In the current implementation the normal priority request will be offered from the queue
I would say the deadline based impl would queue up the one with 0 deadline as the prioritized one rather than the normal one. See this BoundedPriorityBlockingQueue. Am I missing something here?
But if you want to implement any priority based scheduling then you can have a new scheduler and make your own queue implementation out of it.
On Thu, May 28, 2015 at 3:08 PM, ashish singhi <as...@huawei.com>
wrote:
> Thanks for the quick responses.
>
> @Anoop:
> Yes, we were also thinking of implementing it along the same lines.
>
> @Ram:
> High priority request should be always given preference over all other
> scan request.
> As per my understanding in the current implementation of deadline the
> scenario cannot be satisfied. Consider a case when there is one normal
> priority request (submitted at t1) with deadline X in the queue and
> one more high priority request (Submitted at t2 >= t1 + x ) with zero deadline.
> In the current implementation the normal priority request will be
> offered from the queue. But our scenario require high priority request
> to be offered. So there is a need of some more decision parameters
> (such as
> priority) in the Scheduler logic.
> Please correct me If I am missing something.
>
> Vladimir Rodionov wrote
> > Scanner prioritization is not sufficient, there are a lot of other
> > IO
> activity (compaction, flushing, logging), which can severely affect
> high priority scanners.
> >
> > HBase IO scheduler anyone?
>
> Yes I agree, Scan priority alone is not sufficient enough but its
> required at first place. With stuffs like IO scheduler the solution will be complete.
>
> Regards,
> Ashish Singhi
>
>
> -----Original Message-----
> From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> Sent: 28 May 2015 12:18
> To: dev@hbase.apache.org
> Subject: Re: Scan priority based on type of request origin
>
> Atleast in trunk it uses SimpleRPCScheduler and internally it tries to
> use the Priority based queue so that based on the deadline set for
> every request it can schedule the highest priority RPC calls.
>
> I would think that your current requirement may fit with the existing one?
> Except that those calls should be given higher prioirty or rather a
> deadline such it is scheduled first?
>
> On Thu, May 28, 2015 at 12:14 PM, Anoop John <an...@gmail.com>
> wrote:
>
> > We have pluggable RPC scheduler now. See config
> > 'hbase.region.server.rpc.scheduler.factory.class'
> >
> > By default a SimpleRpcScheduler is in use and there as you said,
> > there are prioritization for super user. Other prioritization is
> > based on the RS side method prioritization based on annotation.
> >
> > You can see how/whether u can have a custom RPC scheduler and
> > queuing mechanism using which you can fulfill this.
> >
> > -Anoop-
> >
> > On Thu, May 28, 2015 at 11:56 AM, ashish singhi
> > <as...@huawei.com>
> > wrote:
> >
> > > Hello folks.
> > >
> > > We had a customer scenario to support scan priority based on type
> > > of request origin. Consider the scenario , where in n number of
> > > scan
> > requests
> > > are generated from different clients. Some of these requests are
> > > high priority as they are originated from high privileged users OR
> > > from data viewer Web UI etc. Such request should not wait and must
> > > finish execution at the earliest compare to other requests.
> > >
> > > We went through the code and found that HBase support for
> > > de-prioritizing long running scanner(HBASE-10993) and providing
> > > super user rpc requests higher priority comparing to other
> > > users(HBASE-13375). So this means
> > apart
> > > from super user all other user request will be treated as same and
> > > with normal QOS.
> > >
> > > Let us know if there are any existing solutions already available
> > > which may be we are missing.
> > >
> > > If implementing this make sense , we can implement this in community.
> > >
> > > Regards,
> > > Ashish Singhi
> > >
> >
>
Re: Scan priority based on type of request origin
Posted by ramkrishna vasudevan <ra...@gmail.com>.
bq.Consider a case when there is one normal priority request (submitted at
t1) with deadline X in the queue and one more high priority request
(Submitted at t2 >= t1 + x ) with zero deadline. In the current
implementation the normal priority request will be offered from the queue
I would say the deadline based impl would queue up the one with 0 deadline
as the prioritized one rather than the normal one. See
this BoundedPriorityBlockingQueue. Am I missing something here?
But if you want to implement any priority based scheduling then you can
have a new scheduler and make your own queue implementation out of it.
On Thu, May 28, 2015 at 3:08 PM, ashish singhi <as...@huawei.com>
wrote:
> Thanks for the quick responses.
>
> @Anoop:
> Yes, we were also thinking of implementing it along the same lines.
>
> @Ram:
> High priority request should be always given preference over all other
> scan request.
> As per my understanding in the current implementation of deadline the
> scenario cannot be satisfied. Consider a case when there is one normal
> priority request (submitted at t1) with deadline X in the queue and one
> more high priority request (Submitted at t2 >= t1 + x ) with zero deadline.
> In the current implementation the normal priority request will be offered
> from the queue. But our scenario require high priority request to be
> offered. So there is a need of some more decision parameters (such as
> priority) in the Scheduler logic.
> Please correct me If I am missing something.
>
> Vladimir Rodionov wrote
> > Scanner prioritization is not sufficient, there are a lot of other IO
> activity (compaction, flushing, logging), which can severely affect high
> priority scanners.
> >
> > HBase IO scheduler anyone?
>
> Yes I agree, Scan priority alone is not sufficient enough but its required
> at first place. With stuffs like IO scheduler the solution will be complete.
>
> Regards,
> Ashish Singhi
>
>
> -----Original Message-----
> From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> Sent: 28 May 2015 12:18
> To: dev@hbase.apache.org
> Subject: Re: Scan priority based on type of request origin
>
> Atleast in trunk it uses SimpleRPCScheduler and internally it tries to use
> the Priority based queue so that based on the deadline set for every
> request it can schedule the highest priority RPC calls.
>
> I would think that your current requirement may fit with the existing one?
> Except that those calls should be given higher prioirty or rather a
> deadline such it is scheduled first?
>
> On Thu, May 28, 2015 at 12:14 PM, Anoop John <an...@gmail.com>
> wrote:
>
> > We have pluggable RPC scheduler now. See config
> > 'hbase.region.server.rpc.scheduler.factory.class'
> >
> > By default a SimpleRpcScheduler is in use and there as you said, there
> > are prioritization for super user. Other prioritization is based on
> > the RS side method prioritization based on annotation.
> >
> > You can see how/whether u can have a custom RPC scheduler and queuing
> > mechanism using which you can fulfill this.
> >
> > -Anoop-
> >
> > On Thu, May 28, 2015 at 11:56 AM, ashish singhi
> > <as...@huawei.com>
> > wrote:
> >
> > > Hello folks.
> > >
> > > We had a customer scenario to support scan priority based on type of
> > > request origin. Consider the scenario , where in n number of scan
> > requests
> > > are generated from different clients. Some of these requests are
> > > high priority as they are originated from high privileged users OR
> > > from data viewer Web UI etc. Such request should not wait and must
> > > finish execution at the earliest compare to other requests.
> > >
> > > We went through the code and found that HBase support for
> > > de-prioritizing long running scanner(HBASE-10993) and providing
> > > super user rpc requests higher priority comparing to other
> > > users(HBASE-13375). So this means
> > apart
> > > from super user all other user request will be treated as same and
> > > with normal QOS.
> > >
> > > Let us know if there are any existing solutions already available
> > > which may be we are missing.
> > >
> > > If implementing this make sense , we can implement this in community.
> > >
> > > Regards,
> > > Ashish Singhi
> > >
> >
>
RE: Scan priority based on type of request origin
Posted by ashish singhi <as...@huawei.com>.
Thanks for the quick responses.
@Anoop:
Yes, we were also thinking of implementing it along the same lines.
@Ram:
High priority request should be always given preference over all other scan request.
As per my understanding in the current implementation of deadline the scenario cannot be satisfied. Consider a case when there is one normal priority request (submitted at t1) with deadline X in the queue and one more high priority request (Submitted at t2 >= t1 + x ) with zero deadline. In the current implementation the normal priority request will be offered from the queue. But our scenario require high priority request to be offered. So there is a need of some more decision parameters (such as priority) in the Scheduler logic.
Please correct me If I am missing something.
Vladimir Rodionov wrote
> Scanner prioritization is not sufficient, there are a lot of other IO activity (compaction, flushing, logging), which can severely affect high priority scanners.
>
> HBase IO scheduler anyone?
Yes I agree, Scan priority alone is not sufficient enough but its required at first place. With stuffs like IO scheduler the solution will be complete.
Regards,
Ashish Singhi
-----Original Message-----
From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
Sent: 28 May 2015 12:18
To: dev@hbase.apache.org
Subject: Re: Scan priority based on type of request origin
Atleast in trunk it uses SimpleRPCScheduler and internally it tries to use the Priority based queue so that based on the deadline set for every request it can schedule the highest priority RPC calls.
I would think that your current requirement may fit with the existing one?
Except that those calls should be given higher prioirty or rather a deadline such it is scheduled first?
On Thu, May 28, 2015 at 12:14 PM, Anoop John <an...@gmail.com> wrote:
> We have pluggable RPC scheduler now. See config
> 'hbase.region.server.rpc.scheduler.factory.class'
>
> By default a SimpleRpcScheduler is in use and there as you said, there
> are prioritization for super user. Other prioritization is based on
> the RS side method prioritization based on annotation.
>
> You can see how/whether u can have a custom RPC scheduler and queuing
> mechanism using which you can fulfill this.
>
> -Anoop-
>
> On Thu, May 28, 2015 at 11:56 AM, ashish singhi
> <as...@huawei.com>
> wrote:
>
> > Hello folks.
> >
> > We had a customer scenario to support scan priority based on type of
> > request origin. Consider the scenario , where in n number of scan
> requests
> > are generated from different clients. Some of these requests are
> > high priority as they are originated from high privileged users OR
> > from data viewer Web UI etc. Such request should not wait and must
> > finish execution at the earliest compare to other requests.
> >
> > We went through the code and found that HBase support for
> > de-prioritizing long running scanner(HBASE-10993) and providing
> > super user rpc requests higher priority comparing to other
> > users(HBASE-13375). So this means
> apart
> > from super user all other user request will be treated as same and
> > with normal QOS.
> >
> > Let us know if there are any existing solutions already available
> > which may be we are missing.
> >
> > If implementing this make sense , we can implement this in community.
> >
> > Regards,
> > Ashish Singhi
> >
>
Re: Scan priority based on type of request origin
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Atleast in trunk it uses SimpleRPCScheduler and internally it tries to use
the Priority based queue so that based on the deadline set for every
request it can schedule the highest priority RPC calls.
I would think that your current requirement may fit with the existing one?
Except that those calls should be given higher prioirty or rather a
deadline such it is scheduled first?
On Thu, May 28, 2015 at 12:14 PM, Anoop John <an...@gmail.com> wrote:
> We have pluggable RPC scheduler now. See config
> 'hbase.region.server.rpc.scheduler.factory.class'
>
> By default a SimpleRpcScheduler is in use and there as you said, there are
> prioritization for super user. Other prioritization is based on the RS
> side method prioritization based on annotation.
>
> You can see how/whether u can have a custom RPC scheduler and queuing
> mechanism using which you can fulfill this.
>
> -Anoop-
>
> On Thu, May 28, 2015 at 11:56 AM, ashish singhi <as...@huawei.com>
> wrote:
>
> > Hello folks.
> >
> > We had a customer scenario to support scan priority based on type of
> > request origin. Consider the scenario , where in n number of scan
> requests
> > are generated from different clients. Some of these requests are high
> > priority as they are originated from high privileged users OR from data
> > viewer Web UI etc. Such request should not wait and must finish execution
> > at the earliest compare to other requests.
> >
> > We went through the code and found that HBase support for de-prioritizing
> > long running scanner(HBASE-10993) and providing super user rpc requests
> > higher priority comparing to other users(HBASE-13375). So this means
> apart
> > from super user all other user request will be treated as same and with
> > normal QOS.
> >
> > Let us know if there are any existing solutions already available which
> > may be we are missing.
> >
> > If implementing this make sense , we can implement this in community.
> >
> > Regards,
> > Ashish Singhi
> >
>
Re: Scan priority based on type of request origin
Posted by Anoop John <an...@gmail.com>.
We have pluggable RPC scheduler now. See config
'hbase.region.server.rpc.scheduler.factory.class'
By default a SimpleRpcScheduler is in use and there as you said, there are
prioritization for super user. Other prioritization is based on the RS
side method prioritization based on annotation.
You can see how/whether u can have a custom RPC scheduler and queuing
mechanism using which you can fulfill this.
-Anoop-
On Thu, May 28, 2015 at 11:56 AM, ashish singhi <as...@huawei.com>
wrote:
> Hello folks.
>
> We had a customer scenario to support scan priority based on type of
> request origin. Consider the scenario , where in n number of scan requests
> are generated from different clients. Some of these requests are high
> priority as they are originated from high privileged users OR from data
> viewer Web UI etc. Such request should not wait and must finish execution
> at the earliest compare to other requests.
>
> We went through the code and found that HBase support for de-prioritizing
> long running scanner(HBASE-10993) and providing super user rpc requests
> higher priority comparing to other users(HBASE-13375). So this means apart
> from super user all other user request will be treated as same and with
> normal QOS.
>
> Let us know if there are any existing solutions already available which
> may be we are missing.
>
> If implementing this make sense , we can implement this in community.
>
> Regards,
> Ashish Singhi
>