You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Stuart Smith <st...@yahoo.com> on 2010/08/11 00:26:30 UTC
Avoiding OutOfMemory Java heap space in region servers
Hello,
I'm seeing errors like so:
010-08-10 12:58:38,938 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher: Got ZooKeeper event, state: Disconnected, type: None, path: null
2010-08-10 12:58:38,939 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
2010-08-10 12:58:38,941 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError, aborting.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:942)
Then I see:
2010-08-10 12:58:39,408 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 79 on 60020, call close(-2793534857581898004) from 192.168.195.88:41233: error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
And finally:
2010-08-10 12:58:39,514 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stop requested, clearing toDo despite exception
2010-08-10 12:58:39,515 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2010-08-10 12:58:39,515 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting
And the server begins to shut down.
Now, it's very likely these are due to retrieving unusually large cells - in fact, that's my current assumption.. I'm seeing M/R tasks fail with intermittently with the same issue on the read of cell data.
My question is why does this bring the whole regionserver down? I would think the regionserver would just fail the Get(), and move on...
Am I misdiagnosing the error? Or is it the case that if I want different behavior, I should pony up with some code? :)
Take care,
-stu
Re: Stream interface to cell Data? Was -> Re: Avoiding OutOfMemory
Java heap space in region servers
Posted by Ryan Rawson <ry...@gmail.com>.
Hey,
Streaming is one of those kinds of things that would require a major
wholesale change... good ones, but needless to say reworking the
fundamentals of how the RPC system and the storage system and the file
format works is not really an overnight project.
If you are storing extremely large cells the best bet is HDFS. Most
systems end up having to do mixed storage, and it might be difficult
to make HBase useful for 10 byte cells and 10 GB cells. With some
good API layers on your app side it shouldn't be too hard.
-ryan
On Wed, Aug 18, 2010 at 9:02 PM, Stack <st...@duboce.net> wrote:
> On Wed, Aug 18, 2010 at 4:47 PM, Stuart Smith <st...@yahoo.com> wrote:
>>
>> Hello,
>>
>> I was wondering if there are any plans for a stream interface to Cell data. I saw this:
>>
>>> > or they are using large client write buffers so big
>>> payloads are being
>>> > passed to the server in each RPC request. Our
>>> RPC is not streaming.
>>
>> So I'm guessing there's not one now (and I couldn't find one in 0.20.6 either). HDFS does seem to provide a stream interface (I'm about to try it out).
>>
>> So is there a fundamental limitation on hbase that prevents a streaming
>> interface to Cells, is it possible but distasteful for some reason, or is it just a TODO item?
>>
>
>
>
> Our RPC doesn't do streaming.
>
> A streaming/chunking protocol would be nice -- there is even an old
> issue to do it -- but I think general consensus is that its low
> priority (do you think different)?
>
> Also, if your cells are large, you might consider keeping the content
> in hdfs and their location up in hbase. If the cell is 100MB, the
> lookup in hbase pales beside the time to stream from hdfs.
>
>
>> I'm thinking this could help alleviate the Big Cell OOME situation. This would be especially handy if you just have a few outlier cells that are really big, and lots of smaller ones.
>>
>
> Big cell OOME is rare, unless I'm mistaken. Or saying it another way,
> its rare in my experience that hbase is used hosting big cells. We
> should add better cell size checks out on client and like the
> speed-limiter on your hertz ferrari, it'll keep you safe at least
> until you go out of your way to dismantle the check.
>
>> Right now I'm just going with the solution of putting a layer on top of my system that writes filemetadata and most (smaller) files to hbase, and the occasional big file to HDFS. This should work, and is probably best in the long run, but a streaming interface would be handy!
>>
>
> Oh, yeah, this is a bit of a pain having to handle two sources for
> data. Does your dataset fluctuate wildly in its size? Is there a
> way you can separate the big from the small? If so, perhaps you could
> model it so the big was in one column family and the small in another.
> The big column family held the hdfs location where the small-data
> column family actually carried the data?
>
> St.Ack
>
Re: Stream interface to cell Data? Was -> Re: Avoiding OutOfMemory
Java heap space in region servers
Posted by Stack <st...@duboce.net>.
On Wed, Aug 18, 2010 at 4:47 PM, Stuart Smith <st...@yahoo.com> wrote:
>
> Hello,
>
> I was wondering if there are any plans for a stream interface to Cell data. I saw this:
>
>> > or they are using large client write buffers so big
>> payloads are being
>> > passed to the server in each RPC request. Our
>> RPC is not streaming.
>
> So I'm guessing there's not one now (and I couldn't find one in 0.20.6 either). HDFS does seem to provide a stream interface (I'm about to try it out).
>
> So is there a fundamental limitation on hbase that prevents a streaming
> interface to Cells, is it possible but distasteful for some reason, or is it just a TODO item?
>
Our RPC doesn't do streaming.
A streaming/chunking protocol would be nice -- there is even an old
issue to do it -- but I think general consensus is that its low
priority (do you think different)?
Also, if your cells are large, you might consider keeping the content
in hdfs and their location up in hbase. If the cell is 100MB, the
lookup in hbase pales beside the time to stream from hdfs.
> I'm thinking this could help alleviate the Big Cell OOME situation. This would be especially handy if you just have a few outlier cells that are really big, and lots of smaller ones.
>
Big cell OOME is rare, unless I'm mistaken. Or saying it another way,
its rare in my experience that hbase is used hosting big cells. We
should add better cell size checks out on client and like the
speed-limiter on your hertz ferrari, it'll keep you safe at least
until you go out of your way to dismantle the check.
> Right now I'm just going with the solution of putting a layer on top of my system that writes filemetadata and most (smaller) files to hbase, and the occasional big file to HDFS. This should work, and is probably best in the long run, but a streaming interface would be handy!
>
Oh, yeah, this is a bit of a pain having to handle two sources for
data. Does your dataset fluctuate wildly in its size? Is there a
way you can separate the big from the small? If so, perhaps you could
model it so the big was in one column family and the small in another.
The big column family held the hdfs location where the small-data
column family actually carried the data?
St.Ack
Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Stack <st...@duboce.net>.
On Sat, Aug 14, 2010 at 12:21 PM, Sean Bigdatafun
<se...@gmail.com> wrote:
> On Sat, Aug 14, 2010 at 11:36 AM, Stack <st...@duboce.net> wrote:
>> You don't answer my questions above.
>>
> I ran 6GB now, but I really wonder why I should not run 12GB. (People said
> if I have too much heapsize, GC will kill me)
>
The bigger the heap, the longer it takes to sweep.
So, what have you experienced? A long GC pause that caused your
RegionServers to shutdown? You using default GC settings? You played
with them at all? Have you had a look at
http://wiki.apache.org/hadoop/PerformanceTuning or seen what others
have reported on this list lately the GC configs that work for their
hw/loading? Are you swapping at all? Whats your loading like? All
writes? A mix? Realtime? Analytics? Whats your hardware like?
GC is a pain. Its the bane of all java apps. HBase seems to be
particularly taxing of GC algos.
> It would be really helpful if we can get a detail suggestion on the heapsize
> configuration. Theoretically, a) the more heap size I configure, the more
> data I can hold in memory; b) but does GC pause insanely long on a large
> heap size JVM?
Sorry. We (or should I say java) doesn't have a one answer that fits
all deploys/hardward/loadings. Yeah, a.) is true above -- it means
you can cache more so you'll get better read performance. b.) it can
happen, for sure. Thats the JVM for you. See the cited, if a little
stale, link above and what others have posted mitigating b.). Ongoing
there is HBASE-2902 wherein we hope to improve our default shipping GC
config. and a few of the lads are going beyond deep diving on this
stuff. We'll report back if they find any pearls.
Yours,
St.Ack
Stream interface to cell Data? Was -> Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Stuart Smith <st...@yahoo.com>.
Hello,
I was wondering if there are any plans for a stream interface to Cell data. I saw this:
> > or they are using large client write buffers so big
> payloads are being
> > passed to the server in each RPC request. Our
> RPC is not streaming.
So I'm guessing there's not one now (and I couldn't find one in 0.20.6 either). HDFS does seem to provide a stream interface (I'm about to try it out).
So is there a fundamental limitation on hbase that prevents a streaming
interface to Cells, is it possible but distasteful for some reason, or is it just a TODO item?
I'm thinking this could help alleviate the Big Cell OOME situation. This would be especially handy if you just have a few outlier cells that are really big, and lots of smaller ones.
Right now I'm just going with the solution of putting a layer on top of my system that writes filemetadata and most (smaller) files to hbase, and the occasional big file to HDFS. This should work, and is probably best in the long run, but a streaming interface would be handy!
Take care,
-stu
--- On Sat, 8/14/10, Sean Bigdatafun <se...@gmail.com> wrote:
> From: Sean Bigdatafun <se...@gmail.com>
> Subject: Re: Avoiding OutOfMemory Java heap space in region servers
> To: user@hbase.apache.org
> Date: Saturday, August 14, 2010, 3:21 PM
> On Sat, Aug 14, 2010 at 11:36 AM,
> Stack <st...@duboce.net>
> wrote:
>
> > On Sat, Aug 14, 2010 at 1:26 AM, Sean Bigdatafun
> > <se...@gmail.com>
> wrote:
> > > On Tue, Aug 10, 2010 at 3:40 PM, Stack <st...@duboce.net>
> wrote:
> > >
> > >> OOME may manifest in one place but be caused
> by some other behavior
> > >> altogether. Its an Error. You
> can't tell for sure what damage its
> > >> done to the running process (Though, in your
> stack trace, an OOME
> > >> during the array copy could likely be because
> of very large cells).
> > >> Rather than let the damaged server continue,
> HBase is conservative and
> > >> shuts itself down to minimize possible
> dataloss whenever it gets an
> > >> OOME (It has kept aside an emergency memory
> supply that it releases on
> > >> OOME so the shutdown can 'complete'
> successfully).
> > >>
> > > I understand that your above saying meant that
> HBase shut down the
> > service
> > > for data protection. But can't HBase avoid OOME
> at the first place? Or
> > the
> > > OOME situation is a pending bug in HBase?
> > >
> > > It sounds that HBase can give OOME whenever it
> under heavy loads -- I
> > recall
> > > that several people reporting OOME for unknown
> reasons.
> > >
> >
> > There is always a reason for an OOME.
> >
> > In our experience, the only remaining cause of OOME in
> hbase is
> > because clients are trying to load up many megabyte
> cells concurrently
> > or they are using large client write buffers so big
> payloads are being
> > passed to the server in each RPC request. Our
> RPC is not streaming.
> > It passes byte arrays. If lots of handlers in
> the server and all are
> > being passed big payloads, then its possible that at
> that moment the
> > server heap is overwhelmed.
> >
> > Is this your case?
> >
> > If you need help diagnosing, let us help. When
> hbase OOME's, it dumps
> > the heap. Put it somewhere we can pull it.
> >
> > The server keeps account of heap used except here at
> the edge where
> > RPC is taking in requests.
> >
> > The fix is a little awkward but we'll get to it.
> Meantime,
> > workarounds are up server heap or cut the number of
> handlers or use
> > smaller client write buffer or don't try loading cells
> > 10MB or so --
> > use HDFS direct and keep location in hbase (hbase not
> suited to
> > carrying large stuff in cells).
> >
> > Who are the 'several people' reporting OOMEs? I
> see this week Ted Yu
> > talking of an OOME. It looks like evidence for
> large cells in his
> > case so hypothesis outlined above would seem to
> hold for his case.
> >
> >
> > >>
> > >> Are you doing large multiputs? Do you
> have lots of handlers running?
> > >> If the multiputs are held up because things
> are running slow, memory
> > >> used out on the handlers could throw you over
> especially if your heap
> > >> is small.
> > >>
> > >> What size heap are you running with?
> > >>
> > >
> >
> > You don't answer my questions above.
> >
> I ran 6GB now, but I really wonder why I should not run
> 12GB. (People said
> if I have too much heapsize, GC will kill me)
>
>
> >
> >
> > > By the way, can someone talk about the optimal
> heap size? Say, I have
> > 16GB
> > > in my box, and I use 2GB for my
> DataNode/TaskTracker etc. Presumably, I'd
> > > like to set up my RS heapsize >=12GB to cache
> as much data in memory as
> > > possible. But I heard people saying that too much
> heap size will cause GC
> > > pause issue.
> > >
> >
> > 4-8G is what fellas normally run with.
> >
> > > Can someone give a detailed analysis for what I
> should do?
> > >
> >
> > What you need beyond the above?
> >
> It would be really helpful if we can get a detail
> suggestion on the heapsize
> configuration. Theoretically, a) the more heap size I
> configure, the more
> data I can hold in memory; b) but does GC pause insanely
> long on a large
> heap size JVM?
> If the answer to b) is yes, then I see two factors that
> drags HBase users to
> opposite directions. And we'd better to know the tradeoff,
> a balance point
> -- if we know the underhood mechanisms, we will be able to
> configure our
> system judiciously, which is better than use other's
> already successfully
> used setup number.
>
> Thanks,
> Sean
>
>
> > St.Ack
> >
>
Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Sean Bigdatafun <se...@gmail.com>.
On Sat, Aug 14, 2010 at 11:36 AM, Stack <st...@duboce.net> wrote:
> On Sat, Aug 14, 2010 at 1:26 AM, Sean Bigdatafun
> <se...@gmail.com> wrote:
> > On Tue, Aug 10, 2010 at 3:40 PM, Stack <st...@duboce.net> wrote:
> >
> >> OOME may manifest in one place but be caused by some other behavior
> >> altogether. Its an Error. You can't tell for sure what damage its
> >> done to the running process (Though, in your stack trace, an OOME
> >> during the array copy could likely be because of very large cells).
> >> Rather than let the damaged server continue, HBase is conservative and
> >> shuts itself down to minimize possible dataloss whenever it gets an
> >> OOME (It has kept aside an emergency memory supply that it releases on
> >> OOME so the shutdown can 'complete' successfully).
> >>
> > I understand that your above saying meant that HBase shut down the
> service
> > for data protection. But can't HBase avoid OOME at the first place? Or
> the
> > OOME situation is a pending bug in HBase?
> >
> > It sounds that HBase can give OOME whenever it under heavy loads -- I
> recall
> > that several people reporting OOME for unknown reasons.
> >
>
> There is always a reason for an OOME.
>
> In our experience, the only remaining cause of OOME in hbase is
> because clients are trying to load up many megabyte cells concurrently
> or they are using large client write buffers so big payloads are being
> passed to the server in each RPC request. Our RPC is not streaming.
> It passes byte arrays. If lots of handlers in the server and all are
> being passed big payloads, then its possible that at that moment the
> server heap is overwhelmed.
>
> Is this your case?
>
> If you need help diagnosing, let us help. When hbase OOME's, it dumps
> the heap. Put it somewhere we can pull it.
>
> The server keeps account of heap used except here at the edge where
> RPC is taking in requests.
>
> The fix is a little awkward but we'll get to it. Meantime,
> workarounds are up server heap or cut the number of handlers or use
> smaller client write buffer or don't try loading cells > 10MB or so --
> use HDFS direct and keep location in hbase (hbase not suited to
> carrying large stuff in cells).
>
> Who are the 'several people' reporting OOMEs? I see this week Ted Yu
> talking of an OOME. It looks like evidence for large cells in his
> case so hypothesis outlined above would seem to hold for his case.
>
>
> >>
> >> Are you doing large multiputs? Do you have lots of handlers running?
> >> If the multiputs are held up because things are running slow, memory
> >> used out on the handlers could throw you over especially if your heap
> >> is small.
> >>
> >> What size heap are you running with?
> >>
> >
>
> You don't answer my questions above.
>
I ran 6GB now, but I really wonder why I should not run 12GB. (People said
if I have too much heapsize, GC will kill me)
>
>
> > By the way, can someone talk about the optimal heap size? Say, I have
> 16GB
> > in my box, and I use 2GB for my DataNode/TaskTracker etc. Presumably, I'd
> > like to set up my RS heapsize >=12GB to cache as much data in memory as
> > possible. But I heard people saying that too much heap size will cause GC
> > pause issue.
> >
>
> 4-8G is what fellas normally run with.
>
> > Can someone give a detailed analysis for what I should do?
> >
>
> What you need beyond the above?
>
It would be really helpful if we can get a detail suggestion on the heapsize
configuration. Theoretically, a) the more heap size I configure, the more
data I can hold in memory; b) but does GC pause insanely long on a large
heap size JVM?
If the answer to b) is yes, then I see two factors that drags HBase users to
opposite directions. And we'd better to know the tradeoff, a balance point
-- if we know the underhood mechanisms, we will be able to configure our
system judiciously, which is better than use other's already successfully
used setup number.
Thanks,
Sean
> St.Ack
>
Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Stack <st...@duboce.net>.
On Sat, Aug 14, 2010 at 1:26 AM, Sean Bigdatafun
<se...@gmail.com> wrote:
> On Tue, Aug 10, 2010 at 3:40 PM, Stack <st...@duboce.net> wrote:
>
>> OOME may manifest in one place but be caused by some other behavior
>> altogether. Its an Error. You can't tell for sure what damage its
>> done to the running process (Though, in your stack trace, an OOME
>> during the array copy could likely be because of very large cells).
>> Rather than let the damaged server continue, HBase is conservative and
>> shuts itself down to minimize possible dataloss whenever it gets an
>> OOME (It has kept aside an emergency memory supply that it releases on
>> OOME so the shutdown can 'complete' successfully).
>>
> I understand that your above saying meant that HBase shut down the service
> for data protection. But can't HBase avoid OOME at the first place? Or the
> OOME situation is a pending bug in HBase?
>
> It sounds that HBase can give OOME whenever it under heavy loads -- I recall
> that several people reporting OOME for unknown reasons.
>
There is always a reason for an OOME.
In our experience, the only remaining cause of OOME in hbase is
because clients are trying to load up many megabyte cells concurrently
or they are using large client write buffers so big payloads are being
passed to the server in each RPC request. Our RPC is not streaming.
It passes byte arrays. If lots of handlers in the server and all are
being passed big payloads, then its possible that at that moment the
server heap is overwhelmed.
Is this your case?
If you need help diagnosing, let us help. When hbase OOME's, it dumps
the heap. Put it somewhere we can pull it.
The server keeps account of heap used except here at the edge where
RPC is taking in requests.
The fix is a little awkward but we'll get to it. Meantime,
workarounds are up server heap or cut the number of handlers or use
smaller client write buffer or don't try loading cells > 10MB or so --
use HDFS direct and keep location in hbase (hbase not suited to
carrying large stuff in cells).
Who are the 'several people' reporting OOMEs? I see this week Ted Yu
talking of an OOME. It looks like evidence for large cells in his
case so hypothesis outlined above would seem to hold for his case.
>>
>> Are you doing large multiputs? Do you have lots of handlers running?
>> If the multiputs are held up because things are running slow, memory
>> used out on the handlers could throw you over especially if your heap
>> is small.
>>
>> What size heap are you running with?
>>
>
You don't answer my questions above.
> By the way, can someone talk about the optimal heap size? Say, I have 16GB
> in my box, and I use 2GB for my DataNode/TaskTracker etc. Presumably, I'd
> like to set up my RS heapsize >=12GB to cache as much data in memory as
> possible. But I heard people saying that too much heap size will cause GC
> pause issue.
>
4-8G is what fellas normally run with.
> Can someone give a detailed analysis for what I should do?
>
What you need beyond the above?
St.Ack
Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Sean Bigdatafun <se...@gmail.com>.
On Tue, Aug 10, 2010 at 3:40 PM, Stack <st...@duboce.net> wrote:
> OOME may manifest in one place but be caused by some other behavior
> altogether. Its an Error. You can't tell for sure what damage its
> done to the running process (Though, in your stack trace, an OOME
> during the array copy could likely be because of very large cells).
> Rather than let the damaged server continue, HBase is conservative and
> shuts itself down to minimize possible dataloss whenever it gets an
> OOME (It has kept aside an emergency memory supply that it releases on
> OOME so the shutdown can 'complete' successfully).
>
I understand that your above saying meant that HBase shut down the service
for data protection. But can't HBase avoid OOME at the first place? Or the
OOME situation is a pending bug in HBase?
It sounds that HBase can give OOME whenever it under heavy loads -- I recall
that several people reporting OOME for unknown reasons.
>
> Are you doing large multiputs? Do you have lots of handlers running?
> If the multiputs are held up because things are running slow, memory
> used out on the handlers could throw you over especially if your heap
> is small.
>
> What size heap are you running with?
>
By the way, can someone talk about the optimal heap size? Say, I have 16GB
in my box, and I use 2GB for my DataNode/TaskTracker etc. Presumably, I'd
like to set up my RS heapsize >=12GB to cache as much data in memory as
possible. But I heard people saying that too much heap size will cause GC
pause issue.
Can someone give a detailed analysis for what I should do?
Thanks,
Sean
>
> St.Ack
>
>
>
> On Tue, Aug 10, 2010 at 3:26 PM, Stuart Smith <st...@yahoo.com> wrote:
> > Hello,
> >
> > I'm seeing errors like so:
> >
> > 010-08-10 12:58:38,938 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher: Got
> ZooKeeper event, state: Disconnected, type: None, path: null
> > 2010-08-10 12:58:38,939 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> >
> > 2010-08-10 12:58:38,941 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError,
> aborting.
> > java.lang.OutOfMemoryError: Java heap space
> > at java.util.Arrays.copyOf(Arrays.java:2786)
> > at
> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:942)
> >
> > Then I see:
> >
> > 2010-08-10 12:58:39,408 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 79 on 60020, call close(-2793534857581898004) from
> 192.168.195.88:41233: error: java.io.IOException: Server not running,
> aborting
> > java.io.IOException: Server not running, aborting
> >
> > And finally:
> >
> > 2010-08-10 12:58:39,514 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stop requested, clearing
> toDo despite exception
> > 2010-08-10 12:58:39,515 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> > 2010-08-10 12:58:39,515 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 1 on 60020: exiting
> >
> > And the server begins to shut down.
> >
> > Now, it's very likely these are due to retrieving unusually large cells -
> in fact, that's my current assumption.. I'm seeing M/R tasks fail with
> intermittently with the same issue on the read of cell data.
> >
> > My question is why does this bring the whole regionserver down? I would
> think the regionserver would just fail the Get(), and move on...
> >
> > Am I misdiagnosing the error? Or is it the case that if I want different
> behavior, I should pony up with some code? :)
> >
> > Take care,
> > -stu
> >
> >
> >
> >
> >
>
Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Stack <st...@duboce.net>.
On Thu, Aug 12, 2010 at 11:56 AM, Stuart Smith <st...@yahoo.com> wrote:
> What I actually ended up doing was catching the OOME's in my M/R tasks, and looking at the cell size. One of the cells was 500 MB :|. So that was bad. I've taken to avoiding large cells in the M/R task, and things have smoothed out.
>
> It looks like I should just be a little more circumspect with how much data I cram in a cell. Mostly I limit them to 64 MB, but for one particular tasks I limited to 512 MB.. and I'm getting a decent amount of data now, so inevitably I hit the limit...
>
Good one.
If your cells are that size, you might be better off using hdfs direct
perhaps keeping index to data up in hbase? Or, can you break up the
cell content?
St.Ack
Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Stuart Smith <st...@yahoo.com>.
Hello Stack,
> Rather than let the damaged server continue, HBase is
> conservative
Ah. I see. That does make sense.
> Are you doing large multiputs? Do you have lots of
> handlers running?
Actually this was a M/R task doing lots of reads. But I do have automation (standalone java hbase client, not M/R) that runs every hour doing lots of puts. I think the two could have overlapped and caused issues.
> What size heap are you running with?
Hbase has 4GB, Hadoop had 2GB (on regionserver/datanode/tasktracker computers).
What I actually ended up doing was catching the OOME's in my M/R tasks, and looking at the cell size. One of the cells was 500 MB :|. So that was bad. I've taken to avoiding large cells in the M/R task, and things have smoothed out.
It looks like I should just be a little more circumspect with how much data I cram in a cell. Mostly I limit them to 64 MB, but for one particular tasks I limited to 512 MB.. and I'm getting a decent amount of data now, so inevitably I hit the limit...
Thanks!
Take care,
-stu
--- On Tue, 8/10/10, Stack <st...@duboce.net> wrote:
> From: Stack <st...@duboce.net>
> Subject: Re: Avoiding OutOfMemory Java heap space in region servers
> To: user@hbase.apache.org
> Date: Tuesday, August 10, 2010, 6:40 PM
> OOME may manifest in one place but be
> caused by some other behavior
> altogether. Its an Error. You can't tell for
> sure what damage its
> done to the running process (Though, in your stack trace,
> an OOME
> during the array copy could likely be because of very large
> cells).
> Rather than let the damaged server continue, HBase is
> conservative and
> shuts itself down to minimize possible dataloss whenever it
> gets an
> OOME (It has kept aside an emergency memory supply that it
> releases on
> OOME so the shutdown can 'complete' successfully).
>
> Are you doing large multiputs? Do you have lots of
> handlers running?
> If the multiputs are held up because things are running
> slow, memory
> used out on the handlers could throw you over especially if
> your heap
> is small.
>
> What size heap are you running with?
>
> St.Ack
>
>
>
> On Tue, Aug 10, 2010 at 3:26 PM, Stuart Smith <st...@yahoo.com>
> wrote:
> > Hello,
> >
> > I'm seeing errors like so:
> >
> > 010-08-10 12:58:38,938 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher:
> Got ZooKeeper event, state: Disconnected, type: None, path:
> null
> > 2010-08-10 12:58:38,939 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got
> ZooKeeper event, state: Disconnected, type: None, path:
> null
> >
> > 2010-08-10 12:58:38,941 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> OutOfMemoryError, aborting.
> > java.lang.OutOfMemoryError: Java heap space
> > at
> java.util.Arrays.copyOf(Arrays.java:2786)
> > at
> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:942)
> >
> > Then I see:
> >
> > 2010-08-10 12:58:39,408 INFO
> org.apache.hadoop.ipc.HBaseServer: IPC Server handler 79 on
> 60020, call close(-2793534857581898004) from
> 192.168.195.88:41233: error: java.io.IOException: Server not
> running, aborting
> > java.io.IOException: Server not running, aborting
> >
> > And finally:
> >
> > 2010-08-10 12:58:39,514 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stop
> requested, clearing toDo despite exception
> > 2010-08-10 12:58:39,515 INFO
> org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> > 2010-08-10 12:58:39,515 INFO
> org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on
> 60020: exiting
> >
> > And the server begins to shut down.
> >
> > Now, it's very likely these are due to retrieving
> unusually large cells - in fact, that's my current
> assumption.. I'm seeing M/R tasks fail with intermittently
> with the same issue on the read of cell data.
> >
> > My question is why does this bring the whole
> regionserver down? I would think the regionserver would just
> fail the Get(), and move on...
> >
> > Am I misdiagnosing the error? Or is it the case that
> if I want different behavior, I should pony up with some
> code? :)
> >
> > Take care,
> > -stu
> >
> >
> >
> >
> >
>
Re: Avoiding OutOfMemory Java heap space in region servers
Posted by Stack <st...@duboce.net>.
OOME may manifest in one place but be caused by some other behavior
altogether. Its an Error. You can't tell for sure what damage its
done to the running process (Though, in your stack trace, an OOME
during the array copy could likely be because of very large cells).
Rather than let the damaged server continue, HBase is conservative and
shuts itself down to minimize possible dataloss whenever it gets an
OOME (It has kept aside an emergency memory supply that it releases on
OOME so the shutdown can 'complete' successfully).
Are you doing large multiputs? Do you have lots of handlers running?
If the multiputs are held up because things are running slow, memory
used out on the handlers could throw you over especially if your heap
is small.
What size heap are you running with?
St.Ack
On Tue, Aug 10, 2010 at 3:26 PM, Stuart Smith <st...@yahoo.com> wrote:
> Hello,
>
> I'm seeing errors like so:
>
> 010-08-10 12:58:38,938 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher: Got ZooKeeper event, state: Disconnected, type: None, path: null
> 2010-08-10 12:58:38,939 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
>
> 2010-08-10 12:58:38,941 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError, aborting.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:942)
>
> Then I see:
>
> 2010-08-10 12:58:39,408 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 79 on 60020, call close(-2793534857581898004) from 192.168.195.88:41233: error: java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
>
> And finally:
>
> 2010-08-10 12:58:39,514 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stop requested, clearing toDo despite exception
> 2010-08-10 12:58:39,515 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> 2010-08-10 12:58:39,515 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting
>
> And the server begins to shut down.
>
> Now, it's very likely these are due to retrieving unusually large cells - in fact, that's my current assumption.. I'm seeing M/R tasks fail with intermittently with the same issue on the read of cell data.
>
> My question is why does this bring the whole regionserver down? I would think the regionserver would just fail the Get(), and move on...
>
> Am I misdiagnosing the error? Or is it the case that if I want different behavior, I should pony up with some code? :)
>
> Take care,
> -stu
>
>
>
>
>