You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2011/12/05 18:51:34 UTC
MaxDirectMemorySize Was: Suspected memory leak
How about a slightly larger value, say 10MB ?
Cheers
On Mon, Dec 5, 2011 at 9:47 AM, Stack <st...@duboce.net> wrote:
> On Sun, Dec 4, 2011 at 10:08 PM, Lars <lh...@yahoo.com> wrote:
> > Lastly, XX:MaxDirectMemorySize should definitely be documented.
> >
>
> I can doc this. Should we set it to 1MB(?) in our default jvm configs
> (seems like default is 64M according to
>
> http://stackoverflow.com/questions/3773775/default-for-xxmaxdirectmemorysize
> )?
>
> Nice work lads,
> St.Ack
>
Re: MaxDirectMemorySize Was: Suspected memory leak
Posted by Doug Meil <do...@explorysmedical.com>.
If you don't mind, please add to Troubleshooting (as opposed to FAQ)
There's a Client section in the Troubleshooting chapter already.
http://hbase.apache.org/book.html#trouble.client
On 12/5/11 3:12 PM, "Stack" <st...@duboce.net> wrote:
>On Mon, Dec 5, 2011 at 10:43 AM, Todd Lipcon <to...@cloudera.com> wrote:
>> If the problem is only the client, then why don't we suggest different
>> settings for client vs server?
>>
>> I imagine most clients aren't even using the HBASE_OPTS parameter,
>> right? Only the shell... in which case these settings in hbase-env
>> won't have any effect.
>>
>> We should probably default the offheapcache to 0% and make people
>> explicitly enable it, IMO.
>>
>
>Let me work up an FAQ for client running out of memory suggesting set
>an upper bound on clients (that can be different from server where
>offheap caching might be going on).
>
>St.Ack
>
Re: MaxDirectMemorySize Was: Suspected memory leak
Posted by Stack <st...@duboce.net>.
On Mon, Dec 5, 2011 at 10:43 AM, Todd Lipcon <to...@cloudera.com> wrote:
> If the problem is only the client, then why don't we suggest different
> settings for client vs server?
>
> I imagine most clients aren't even using the HBASE_OPTS parameter,
> right? Only the shell... in which case these settings in hbase-env
> won't have any effect.
>
> We should probably default the offheapcache to 0% and make people
> explicitly enable it, IMO.
>
Let me work up an FAQ for client running out of memory suggesting set
an upper bound on clients (that can be different from server where
offheap caching might be going on).
St.Ack
Re: MaxDirectMemorySize Was: Suspected memory leak
Posted by Todd Lipcon <to...@cloudera.com>.
If the problem is only the client, then why don't we suggest different
settings for client vs server?
I imagine most clients aren't even using the HBASE_OPTS parameter,
right? Only the shell... in which case these settings in hbase-env
won't have any effect.
We should probably default the offheapcache to 0% and make people
explicitly enable it, IMO.
-Todd
On Mon, Dec 5, 2011 at 10:22 AM, Ted Yu <yu...@gmail.com> wrote:
> Thanks for the reminder Todd.
> Currently we use the following config param to calculate offheap cache size:
> long offHeapCacheSize =
> (long) (conf.getFloat("hbase.offheapcache.percentage", (float) 0.95) *
> DirectMemoryUtils.getDirectMemorySize());
> We should lower the default percentage for offheap cache and document the
> tradeoff between giving larger space from direct memory to either offheap
> cache or client NIO.
>
> Cheers
>
> On Mon, Dec 5, 2011 at 10:04 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> The off-heap cache will also depend on a high value for this setting..
>> need to make sure there isn't some kind of conflict between this
>> feature and whatever we choose as a default.
>>
>> -Todd
>>
>> On Mon, Dec 5, 2011 at 9:53 AM, Stack <st...@duboce.net> wrote:
>> > On Mon, Dec 5, 2011 at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>> >> How about a slightly larger value, say 10MB ?
>> >>
>> >
>> > Sure. Let me put a patch up and include it in next RC.
>> > St.Ack
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
--
Todd Lipcon
Software Engineer, Cloudera
Re: MaxDirectMemorySize Was: Suspected memory leak
Posted by Ted Yu <yu...@gmail.com>.
Thanks for the reminder Todd.
Currently we use the following config param to calculate offheap cache size:
long offHeapCacheSize =
(long) (conf.getFloat("hbase.offheapcache.percentage", (float) 0.95) *
DirectMemoryUtils.getDirectMemorySize());
We should lower the default percentage for offheap cache and document the
tradeoff between giving larger space from direct memory to either offheap
cache or client NIO.
Cheers
On Mon, Dec 5, 2011 at 10:04 AM, Todd Lipcon <to...@cloudera.com> wrote:
> The off-heap cache will also depend on a high value for this setting..
> need to make sure there isn't some kind of conflict between this
> feature and whatever we choose as a default.
>
> -Todd
>
> On Mon, Dec 5, 2011 at 9:53 AM, Stack <st...@duboce.net> wrote:
> > On Mon, Dec 5, 2011 at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
> >> How about a slightly larger value, say 10MB ?
> >>
> >
> > Sure. Let me put a patch up and include it in next RC.
> > St.Ack
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: MaxDirectMemorySize Was: Suspected memory leak
Posted by Todd Lipcon <to...@cloudera.com>.
The off-heap cache will also depend on a high value for this setting..
need to make sure there isn't some kind of conflict between this
feature and whatever we choose as a default.
-Todd
On Mon, Dec 5, 2011 at 9:53 AM, Stack <st...@duboce.net> wrote:
> On Mon, Dec 5, 2011 at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>> How about a slightly larger value, say 10MB ?
>>
>
> Sure. Let me put a patch up and include it in next RC.
> St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
Re: MaxDirectMemorySize Was: Suspected memory leak
Posted by Stack <st...@duboce.net>.
On Mon, Dec 5, 2011 at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
> How about a slightly larger value, say 10MB ?
>
Sure. Let me put a patch up and include it in next RC.
St.Ack