Re: Allocating more heap for endpoint coprocessors

[Andrew Purtell <ap...@apache.org>]

+1 HBASE-4047

On Friday, August 31, 2012, Gary Helmling wrote:

> Maybe we need to add a coprocessors section to the ref guide.  I think
> all the current documentation is in javadoc.  And if all the
> potentially destabilizing issues of in-process coprocessor usage are
> not yet called out (memory usage, cpu, etc), we could more explicitly
> detail that.
>
> In we want to really be able to support these cases, though, we
> probably need to push HBASE-4047 (external coprocessor host) forward.
> Without it there's not a lot we can do to prevent coprocessors from
> taking down region servers.
>
>
> On Thu, Aug 30, 2012 at 10:04 PM, lars hofhansl <lhofhansl@yahoo.com<javascript:;>>
> wrote:
> > Maybe we should be better with marking improvement as such in jira, and
> then have a special list in the release announcements highlighting these
> (otherwise they just be in the noise of all the other bug fixes).
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: Ramkrishna.S.Vasudevan <ramkrishna.vasudevan@huawei.com<javascript:;>
> >
> > To: user@hbase.apache.org <javascript:;>; 'lars hofhansl' <
> lhofhansl@yahoo.com <javascript:;>>
> > Sent: Thursday, August 30, 2012 9:59 PM
> > Subject: RE: Allocating more heap for endpoint coprocessors
> >
> > @Lars/@Gary
> >
> > Do we need to document such things.  Recently someone was asking me a
> > question like this, if my endpoint impl is so memory intensive it just
> > affects a running cluster and already the RS
> > has a huge memory heap associated with it.
> >
> > So its better we document saying if your endpoint consumes memory and
> > because it runs along with RS based on your need add that extra amount of
> > memory that will be taken up by the endpoint impl.
> >
> > Regards
> > Ram
> >
> >> -----Original Message-----
> >> From: lars hofhansl [mailto:lhofhansl@yahoo.com <javascript:;>]
> >> Sent: Friday, August 31, 2012 3:04 AM
> >> To: user@hbase.apache.org <javascript:;>
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> >> you to share state between RegionObservers (and Endpoints) within the
> >> same RegionServer.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >>  From: Gary Helmling <ghelmling@gmail.com <javascript:;>>
> >> To: user@hbase.apache.org <javascript:;>
> >> Sent: Thursday, August 30, 2012 1:59 PM
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> Endpoint coprocessors are loaded and run within the HBase RegionServer
> >> process.  Your endpoint coprocessors will be running on the region
> >> servers hosting the regions for the table(s) on which the coprocessor
> >> is configured.
> >>
> >> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> >> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> >> the region server.
> >>
> >> Note that a separate coprocessor instance is loaded for each table
> >> region, so, say you want to allocate 10MB for your coprocessor, but
> >> each region server hosts 20 regions, you would want to increase the
> >> heap size by 200MB (20x10MB).
> >>
> >> --gh
> >>
> >> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <ykim@crunchyroll.com<javascript:;>
> >
> >> wrote:
> >> > Hi,
> >> >
> >> > We have some memory intensive endpoint coprocessors running on our
> >> RegionServers. As a result, we want to allocate more heap for the
> >> coprocessors, but there doesn't seem to be much documentation on which
> >> Hbase processes are directly responsible for coprocessors. Does anyone
> >> happen to know or direct me to some resource that does?
> >> >
> >> > Thanks,
> >> > Young Kim
> >> >
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)