You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Rami Mankevich <ra...@amdocs.com> on 2013/04/09 19:51:51 UTC
RE: Hbase question
First of all - thanks for the quick response.
Basically threads I want to open are for my own internal structure updates and I guess have no relations to HBase internal structures.
All I want is initiations for some asynchronous structure updates as part of coprocessor execution in order not to block user reponse.
The only reason I was asking is to be sure Hbase will not kill those threads.
As I understand - shouldn't be any issue with that. Am I correct?
In addition - Is there any Hbase Thread pool I can use?
Thanks
From: Andrew Purtell [mailto:apurtell@apache.org]
Sent: Tuesday, April 09, 2013 6:53 PM
To: Rami Mankevich
Cc: apurtell@apache.org
Subject: Re: Hbase question
Hi Rami,
It is no problem to create threads in a coprocessor as a generic answer. More specifically there could be issues depending on exactly what you want to do, since coprocessor code changes HBase internals. Perhaps you could say a bit more. I also encourage you to ask this question on user@hbase.apache.org<ma...@hbase.apache.org> so other contributors can chime in too.
On Tuesday, April 9, 2013, Rami Mankevich wrote:
Hey
According to the Hbase documentation you are one of contrinuters to the HBase project
I would like to raise some question when nobody can basically advice me:
In context of coprocessors I want to raise some threads.
Do you see any problems with that?
Thanks
This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Hbase question
Posted by Ted Yu <yu...@gmail.com>.
Gary has provided nice summary of things to watch out for.
One more thing I want to mention is that care should be taken w.r.t.
coordinating the progress of the thread pool and normal region operations.
There're already many threads running in the region server JVM. Adding one
more thread pool may make resource consumption more complex.
What if the custom processing cannot keep up with the rate at which
asynchronous requests are queued ?
w.r.t. thread pool, you can refer to the following code in HRegion:
ThreadPoolExecutor storeOpenerThreadPool =
getStoreOpenAndCloseThreadPool(
"StoreOpenerThread-" + this.getRegionNameAsString());
CompletionService<HStore> completionService =
new ExecutorCompletionService<HStore>(storeOpenerThreadPool);
Cheers
On Tue, Apr 9, 2013 at 11:57 AM, Gary Helmling <gh...@gmail.com> wrote:
> Hi Rami,
>
> One thing to note for RegionObservers, is that each table region gets its
> own instance of each configured coprocessor. So if your cluster has N
> regions per region server, with your RegionObserver loaded on all tables,
> then each region server will have N instances of your coprocessor. You
> should just be aware of this in case you, say, create a thread pool in your
> coprocessor constructor. An alternative in this case is to use a singleton
> class per region server (aka per jvm) to manage the resources.
>
> You do want to be sure that all threads are daemon threads, so that they
> don't block region server shutdown. Or else you'll need to ensure you
> properly stop/join all the threads you've spawned on shutdown.
> RegionServerObserver.preStopRegionServer() may help there.
>
> --gh
>
>
>
> On Tue, Apr 9, 2013 at 11:40 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Rami:
> > Can you tell us what coprocessor hook you plan to use ?
> >
> > Thanks
> >
> > On Tue, Apr 9, 2013 at 10:51 AM, Rami Mankevich <ra...@amdocs.com>
> wrote:
> >
> > > First of all - thanks for the quick response.
> > >
> > > Basically threads I want to open are for my own internal structure
> > > updates and I guess have no relations to HBase internal structures.
> > > All I want is initiations for some asynchronous structure updates as
> part
> > > of coprocessor execution in order not to block user reponse.
> > >
> > > The only reason I was asking is to be sure Hbase will not kill those
> > > threads.
> > > As I understand - shouldn't be any issue with that. Am I correct?
> > >
> > > In addition - Is there any Hbase Thread pool I can use?
> > >
> > >
> > > Thanks
> > > From: Andrew Purtell [mailto:apurtell@apache.org]
> > > Sent: Tuesday, April 09, 2013 6:53 PM
> > > To: Rami Mankevich
> > > Cc: apurtell@apache.org
> > > Subject: Re: Hbase question
> > >
> > > Hi Rami,
> > >
> > > It is no problem to create threads in a coprocessor as a generic
> answer.
> > > More specifically there could be issues depending on exactly what you
> > want
> > > to do, since coprocessor code changes HBase internals. Perhaps you
> could
> > > say a bit more. I also encourage you to ask this question on
> > > user@hbase.apache.org<ma...@hbase.apache.org> so other
> > contributors
> > > can chime in too.
> > >
> > > On Tuesday, April 9, 2013, Rami Mankevich wrote:
> > > Hey
> > > According to the Hbase documentation you are one of contrinuters to the
> > > HBase project
> > > I would like to raise some question when nobody can basically advice
> me:
> > >
> > > In context of coprocessors I want to raise some threads.
> > > Do you see any problems with that?
> > >
> > > Thanks
> > > This message and the information contained herein is proprietary and
> > > confidential and subject to the Amdocs policy statement, you may review
> > at
> > > http://www.amdocs.com/email_disclaimer.asp
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
Re: Hbase question
Posted by Gary Helmling <gh...@gmail.com>.
Hi Rami,
One thing to note for RegionObservers, is that each table region gets its
own instance of each configured coprocessor. So if your cluster has N
regions per region server, with your RegionObserver loaded on all tables,
then each region server will have N instances of your coprocessor. You
should just be aware of this in case you, say, create a thread pool in your
coprocessor constructor. An alternative in this case is to use a singleton
class per region server (aka per jvm) to manage the resources.
You do want to be sure that all threads are daemon threads, so that they
don't block region server shutdown. Or else you'll need to ensure you
properly stop/join all the threads you've spawned on shutdown.
RegionServerObserver.preStopRegionServer() may help there.
--gh
On Tue, Apr 9, 2013 at 11:40 AM, Ted Yu <yu...@gmail.com> wrote:
> Rami:
> Can you tell us what coprocessor hook you plan to use ?
>
> Thanks
>
> On Tue, Apr 9, 2013 at 10:51 AM, Rami Mankevich <ra...@amdocs.com> wrote:
>
> > First of all - thanks for the quick response.
> >
> > Basically threads I want to open are for my own internal structure
> > updates and I guess have no relations to HBase internal structures.
> > All I want is initiations for some asynchronous structure updates as part
> > of coprocessor execution in order not to block user reponse.
> >
> > The only reason I was asking is to be sure Hbase will not kill those
> > threads.
> > As I understand - shouldn't be any issue with that. Am I correct?
> >
> > In addition - Is there any Hbase Thread pool I can use?
> >
> >
> > Thanks
> > From: Andrew Purtell [mailto:apurtell@apache.org]
> > Sent: Tuesday, April 09, 2013 6:53 PM
> > To: Rami Mankevich
> > Cc: apurtell@apache.org
> > Subject: Re: Hbase question
> >
> > Hi Rami,
> >
> > It is no problem to create threads in a coprocessor as a generic answer.
> > More specifically there could be issues depending on exactly what you
> want
> > to do, since coprocessor code changes HBase internals. Perhaps you could
> > say a bit more. I also encourage you to ask this question on
> > user@hbase.apache.org<ma...@hbase.apache.org> so other
> contributors
> > can chime in too.
> >
> > On Tuesday, April 9, 2013, Rami Mankevich wrote:
> > Hey
> > According to the Hbase documentation you are one of contrinuters to the
> > HBase project
> > I would like to raise some question when nobody can basically advice me:
> >
> > In context of coprocessors I want to raise some threads.
> > Do you see any problems with that?
> >
> > Thanks
> > This message and the information contained herein is proprietary and
> > confidential and subject to the Amdocs policy statement, you may review
> at
> > http://www.amdocs.com/email_disclaimer.asp
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: Hbase question
Posted by Ted Yu <yu...@gmail.com>.
Rami:
Can you tell us what coprocessor hook you plan to use ?
Thanks
On Tue, Apr 9, 2013 at 10:51 AM, Rami Mankevich <ra...@amdocs.com> wrote:
> First of all - thanks for the quick response.
>
> Basically threads I want to open are for my own internal structure
> updates and I guess have no relations to HBase internal structures.
> All I want is initiations for some asynchronous structure updates as part
> of coprocessor execution in order not to block user reponse.
>
> The only reason I was asking is to be sure Hbase will not kill those
> threads.
> As I understand - shouldn't be any issue with that. Am I correct?
>
> In addition - Is there any Hbase Thread pool I can use?
>
>
> Thanks
> From: Andrew Purtell [mailto:apurtell@apache.org]
> Sent: Tuesday, April 09, 2013 6:53 PM
> To: Rami Mankevich
> Cc: apurtell@apache.org
> Subject: Re: Hbase question
>
> Hi Rami,
>
> It is no problem to create threads in a coprocessor as a generic answer.
> More specifically there could be issues depending on exactly what you want
> to do, since coprocessor code changes HBase internals. Perhaps you could
> say a bit more. I also encourage you to ask this question on
> user@hbase.apache.org<ma...@hbase.apache.org> so other contributors
> can chime in too.
>
> On Tuesday, April 9, 2013, Rami Mankevich wrote:
> Hey
> According to the Hbase documentation you are one of contrinuters to the
> HBase project
> I would like to raise some question when nobody can basically advice me:
>
> In context of coprocessors I want to raise some threads.
> Do you see any problems with that?
>
> Thanks
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement, you may review at
> http://www.amdocs.com/email_disclaimer.asp
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>