You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Young Kim <yk...@crunchyroll.com> on 2012/08/30 22:45:29 UTC
Allocating more heap for endpoint coprocessors
Hi,
We have some memory intensive endpoint coprocessors running on our RegionServers. As a result, we want to allocate more heap for the coprocessors, but there doesn't seem to be much documentation on which Hbase processes are directly responsible for coprocessors. Does anyone happen to know or direct me to some resource that does?
Thanks,
Young Kim
Re: Allocating more heap for endpoint coprocessors
Posted by Andrew Purtell <ap...@apache.org>.
+1 HBASE-4047
On Friday, August 31, 2012, Gary Helmling wrote:
> Maybe we need to add a coprocessors section to the ref guide. I think
> all the current documentation is in javadoc. And if all the
> potentially destabilizing issues of in-process coprocessor usage are
> not yet called out (memory usage, cpu, etc), we could more explicitly
> detail that.
>
> In we want to really be able to support these cases, though, we
> probably need to push HBASE-4047 (external coprocessor host) forward.
> Without it there's not a lot we can do to prevent coprocessors from
> taking down region servers.
>
>
> On Thu, Aug 30, 2012 at 10:04 PM, lars hofhansl <lhofhansl@yahoo.com<javascript:;>>
> wrote:
> > Maybe we should be better with marking improvement as such in jira, and
> then have a special list in the release announcements highlighting these
> (otherwise they just be in the noise of all the other bug fixes).
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Ramkrishna.S.Vasudevan <ramkrishna.vasudevan@huawei.com<javascript:;>
> >
> > To: user@hbase.apache.org <javascript:;>; 'lars hofhansl' <
> lhofhansl@yahoo.com <javascript:;>>
> > Sent: Thursday, August 30, 2012 9:59 PM
> > Subject: RE: Allocating more heap for endpoint coprocessors
> >
> > @Lars/@Gary
> >
> > Do we need to document such things. Recently someone was asking me a
> > question like this, if my endpoint impl is so memory intensive it just
> > affects a running cluster and already the RS
> > has a huge memory heap associated with it.
> >
> > So its better we document saying if your endpoint consumes memory and
> > because it runs along with RS based on your need add that extra amount of
> > memory that will be taken up by the endpoint impl.
> >
> > Regards
> > Ram
> >
> >> -----Original Message-----
> >> From: lars hofhansl [mailto:lhofhansl@yahoo.com <javascript:;>]
> >> Sent: Friday, August 31, 2012 3:04 AM
> >> To: user@hbase.apache.org <javascript:;>
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> >> you to share state between RegionObservers (and Endpoints) within the
> >> same RegionServer.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >> From: Gary Helmling <ghelmling@gmail.com <javascript:;>>
> >> To: user@hbase.apache.org <javascript:;>
> >> Sent: Thursday, August 30, 2012 1:59 PM
> >> Subject: Re: Allocating more heap for endpoint coprocessors
> >>
> >> Endpoint coprocessors are loaded and run within the HBase RegionServer
> >> process. Your endpoint coprocessors will be running on the region
> >> servers hosting the regions for the table(s) on which the coprocessor
> >> is configured.
> >>
> >> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> >> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> >> the region server.
> >>
> >> Note that a separate coprocessor instance is loaded for each table
> >> region, so, say you want to allocate 10MB for your coprocessor, but
> >> each region server hosts 20 regions, you would want to increase the
> >> heap size by 200MB (20x10MB).
> >>
> >> --gh
> >>
> >> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <ykim@crunchyroll.com<javascript:;>
> >
> >> wrote:
> >> > Hi,
> >> >
> >> > We have some memory intensive endpoint coprocessors running on our
> >> RegionServers. As a result, we want to allocate more heap for the
> >> coprocessors, but there doesn't seem to be much documentation on which
> >> Hbase processes are directly responsible for coprocessors. Does anyone
> >> happen to know or direct me to some resource that does?
> >> >
> >> > Thanks,
> >> > Young Kim
> >> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Allocating more heap for endpoint coprocessors
Posted by Gary Helmling <gh...@gmail.com>.
Maybe we need to add a coprocessors section to the ref guide. I think
all the current documentation is in javadoc. And if all the
potentially destabilizing issues of in-process coprocessor usage are
not yet called out (memory usage, cpu, etc), we could more explicitly
detail that.
In we want to really be able to support these cases, though, we
probably need to push HBASE-4047 (external coprocessor host) forward.
Without it there's not a lot we can do to prevent coprocessors from
taking down region servers.
On Thu, Aug 30, 2012 at 10:04 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Maybe we should be better with marking improvement as such in jira, and then have a special list in the release announcements highlighting these (otherwise they just be in the noise of all the other bug fixes).
>
> -- Lars
>
>
>
> ________________________________
> From: Ramkrishna.S.Vasudevan <ra...@huawei.com>
> To: user@hbase.apache.org; 'lars hofhansl' <lh...@yahoo.com>
> Sent: Thursday, August 30, 2012 9:59 PM
> Subject: RE: Allocating more heap for endpoint coprocessors
>
> @Lars/@Gary
>
> Do we need to document such things. Recently someone was asking me a
> question like this, if my endpoint impl is so memory intensive it just
> affects a running cluster and already the RS
> has a huge memory heap associated with it.
>
> So its better we document saying if your endpoint consumes memory and
> because it runs along with RS based on your need add that extra amount of
> memory that will be taken up by the endpoint impl.
>
> Regards
> Ram
>
>> -----Original Message-----
>> From: lars hofhansl [mailto:lhofhansl@yahoo.com]
>> Sent: Friday, August 31, 2012 3:04 AM
>> To: user@hbase.apache.org
>> Subject: Re: Allocating more heap for endpoint coprocessors
>>
>> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
>> you to share state between RegionObservers (and Endpoints) within the
>> same RegionServer.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Gary Helmling <gh...@gmail.com>
>> To: user@hbase.apache.org
>> Sent: Thursday, August 30, 2012 1:59 PM
>> Subject: Re: Allocating more heap for endpoint coprocessors
>>
>> Endpoint coprocessors are loaded and run within the HBase RegionServer
>> process. Your endpoint coprocessors will be running on the region
>> servers hosting the regions for the table(s) on which the coprocessor
>> is configured.
>>
>> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
>> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
>> the region server.
>>
>> Note that a separate coprocessor instance is loaded for each table
>> region, so, say you want to allocate 10MB for your coprocessor, but
>> each region server hosts 20 regions, you would want to increase the
>> heap size by 200MB (20x10MB).
>>
>> --gh
>>
>> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <yk...@crunchyroll.com>
>> wrote:
>> > Hi,
>> >
>> > We have some memory intensive endpoint coprocessors running on our
>> RegionServers. As a result, we want to allocate more heap for the
>> coprocessors, but there doesn't seem to be much documentation on which
>> Hbase processes are directly responsible for coprocessors. Does anyone
>> happen to know or direct me to some resource that does?
>> >
>> > Thanks,
>> > Young Kim
>> >
Re: Allocating more heap for endpoint coprocessors
Posted by lars hofhansl <lh...@yahoo.com>.
Maybe we should be better with marking improvement as such in jira, and then have a special list in the release announcements highlighting these (otherwise they just be in the noise of all the other bug fixes).
-- Lars
________________________________
From: Ramkrishna.S.Vasudevan <ra...@huawei.com>
To: user@hbase.apache.org; 'lars hofhansl' <lh...@yahoo.com>
Sent: Thursday, August 30, 2012 9:59 PM
Subject: RE: Allocating more heap for endpoint coprocessors
@Lars/@Gary
Do we need to document such things. Recently someone was asking me a
question like this, if my endpoint impl is so memory intensive it just
affects a running cluster and already the RS
has a huge memory heap associated with it.
So its better we document saying if your endpoint consumes memory and
because it runs along with RS based on your need add that extra amount of
memory that will be taken up by the endpoint impl.
Regards
Ram
> -----Original Message-----
> From: lars hofhansl [mailto:lhofhansl@yahoo.com]
> Sent: Friday, August 31, 2012 3:04 AM
> To: user@hbase.apache.org
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> you to share state between RegionObservers (and Endpoints) within the
> same RegionServer.
>
> -- Lars
>
>
>
> ________________________________
> From: Gary Helmling <gh...@gmail.com>
> To: user@hbase.apache.org
> Sent: Thursday, August 30, 2012 1:59 PM
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> Endpoint coprocessors are loaded and run within the HBase RegionServer
> process. Your endpoint coprocessors will be running on the region
> servers hosting the regions for the table(s) on which the coprocessor
> is configured.
>
> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> the region server.
>
> Note that a separate coprocessor instance is loaded for each table
> region, so, say you want to allocate 10MB for your coprocessor, but
> each region server hosts 20 regions, you would want to increase the
> heap size by 200MB (20x10MB).
>
> --gh
>
> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <yk...@crunchyroll.com>
> wrote:
> > Hi,
> >
> > We have some memory intensive endpoint coprocessors running on our
> RegionServers. As a result, we want to allocate more heap for the
> coprocessors, but there doesn't seem to be much documentation on which
> Hbase processes are directly responsible for coprocessors. Does anyone
> happen to know or direct me to some resource that does?
> >
> > Thanks,
> > Young Kim
> >
RE: Allocating more heap for endpoint coprocessors
Posted by "Ramkrishna.S.Vasudevan" <ra...@huawei.com>.
@Lars/@Gary
Do we need to document such things. Recently someone was asking me a
question like this, if my endpoint impl is so memory intensive it just
affects a running cluster and already the RS
has a huge memory heap associated with it.
So its better we document saying if your endpoint consumes memory and
because it runs along with RS based on your need add that extra amount of
memory that will be taken up by the endpoint impl.
Regards
Ram
> -----Original Message-----
> From: lars hofhansl [mailto:lhofhansl@yahoo.com]
> Sent: Friday, August 31, 2012 3:04 AM
> To: user@hbase.apache.org
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> In the upcoming 0.94.2 release will also have HBASE-6505, which allows
> you to share state between RegionObservers (and Endpoints) within the
> same RegionServer.
>
> -- Lars
>
>
>
> ________________________________
> From: Gary Helmling <gh...@gmail.com>
> To: user@hbase.apache.org
> Sent: Thursday, August 30, 2012 1:59 PM
> Subject: Re: Allocating more heap for endpoint coprocessors
>
> Endpoint coprocessors are loaded and run within the HBase RegionServer
> process. Your endpoint coprocessors will be running on the region
> servers hosting the regions for the table(s) on which the coprocessor
> is configured.
>
> So the way to allocate more memory is by setting either HBASE_HEAPSIZE
> or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
> the region server.
>
> Note that a separate coprocessor instance is loaded for each table
> region, so, say you want to allocate 10MB for your coprocessor, but
> each region server hosts 20 regions, you would want to increase the
> heap size by 200MB (20x10MB).
>
> --gh
>
> On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <yk...@crunchyroll.com>
> wrote:
> > Hi,
> >
> > We have some memory intensive endpoint coprocessors running on our
> RegionServers. As a result, we want to allocate more heap for the
> coprocessors, but there doesn't seem to be much documentation on which
> Hbase processes are directly responsible for coprocessors. Does anyone
> happen to know or direct me to some resource that does?
> >
> > Thanks,
> > Young Kim
> >
Re: Allocating more heap for endpoint coprocessors
Posted by lars hofhansl <lh...@yahoo.com>.
In the upcoming 0.94.2 release will also have HBASE-6505, which allows you to share state between RegionObservers (and Endpoints) within the same RegionServer.
-- Lars
________________________________
From: Gary Helmling <gh...@gmail.com>
To: user@hbase.apache.org
Sent: Thursday, August 30, 2012 1:59 PM
Subject: Re: Allocating more heap for endpoint coprocessors
Endpoint coprocessors are loaded and run within the HBase RegionServer
process. Your endpoint coprocessors will be running on the region
servers hosting the regions for the table(s) on which the coprocessor
is configured.
So the way to allocate more memory is by setting either HBASE_HEAPSIZE
or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
the region server.
Note that a separate coprocessor instance is loaded for each table
region, so, say you want to allocate 10MB for your coprocessor, but
each region server hosts 20 regions, you would want to increase the
heap size by 200MB (20x10MB).
--gh
On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <yk...@crunchyroll.com> wrote:
> Hi,
>
> We have some memory intensive endpoint coprocessors running on our RegionServers. As a result, we want to allocate more heap for the coprocessors, but there doesn't seem to be much documentation on which Hbase processes are directly responsible for coprocessors. Does anyone happen to know or direct me to some resource that does?
>
> Thanks,
> Young Kim
>
Re: Allocating more heap for endpoint coprocessors
Posted by Gary Helmling <gh...@gmail.com>.
Endpoint coprocessors are loaded and run within the HBase RegionServer
process. Your endpoint coprocessors will be running on the region
servers hosting the regions for the table(s) on which the coprocessor
is configured.
So the way to allocate more memory is by setting either HBASE_HEAPSIZE
or setting the max heap in HBASE_REGIONSERVER_OPTS in hbase-env.sh on
the region server.
Note that a separate coprocessor instance is loaded for each table
region, so, say you want to allocate 10MB for your coprocessor, but
each region server hosts 20 regions, you would want to increase the
heap size by 200MB (20x10MB).
--gh
On Thu, Aug 30, 2012 at 1:45 PM, Young Kim <yk...@crunchyroll.com> wrote:
> Hi,
>
> We have some memory intensive endpoint coprocessors running on our RegionServers. As a result, we want to allocate more heap for the coprocessors, but there doesn't seem to be much documentation on which Hbase processes are directly responsible for coprocessors. Does anyone happen to know or direct me to some resource that does?
>
> Thanks,
> Young Kim
>