You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/11/18 22:25:44 UTC
[jira] Created: (HBASE-1006) Resevoir of memory not freed on OOME
Resevoir of memory not freed on OOME
------------------------------------
Key: HBASE-1006
URL: https://issues.apache.org/jira/browse/HBASE-1006
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Here is the OOME:
{code}
#
2008-11-17 17:04:18,812 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
#
org.apache.hadoop.hbase.DroppedSnapshotException: region: streamitems,^@^@^@^@^A�?�,1226968617756
#
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:865)
#
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:761)
#
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:179)
#
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:232)
#
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
#
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1312)
#
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
#
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
#
at java.lang.reflect.Method.invoke(Method.java:597)
#
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
#
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
#
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
#
at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:62)
#
at org.apache.hadoop.hbase.HStoreKey.readFields(HStoreKey.java:589)
#
at org.apache.hadoop.io.MapFile$Writer.checkKey(MapFile.java:213)
#
at org.apache.hadoop.io.MapFile$Writer.append(MapFile.java:192)
#
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Writer.append(BloomFilterMapFile.java:201)
#
at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:674)
#
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:627)
#
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:852)
#
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:852)
{code}
Looking in heap, the resevoir had not been released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1006) Resevoir of memory not freed on OOME
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1006.
--------------------------
Resolution: Won't Fix
Resevoir is cleared in call to abort. It happens silently. With above exception we do not abort because in BatchUpdate, we see the DSE as an IOE as it goes by, not as an OOME. OOMEs can happen anywhere. Thats a bit of a problem given as there is no single entry point into HRS.
Resolving as fixed as part of HBASE-1020. Over there, I'll add a check for OOME everywhere in HRS. If an OOME, will abort. Abort will then do the clear of the resevoir so the abort can run smoothly. Will also dump metrics.
As to whether abort and shutdown on OOME is the right thing to do, thats we should discuss.
> Resevoir of memory not freed on OOME
> ------------------------------------
>
> Key: HBASE-1006
> URL: https://issues.apache.org/jira/browse/HBASE-1006
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Here is the OOME:
> {code}
> #
> 2008-11-17 17:04:18,812 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> #
> org.apache.hadoop.hbase.DroppedSnapshotException: region: streamitems,^@^@^@^@^A�?�,1226968617756
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:865)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:761)
> #
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:179)
> #
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:232)
> #
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> #
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1312)
> #
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> #
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> #
> at java.lang.reflect.Method.invoke(Method.java:597)
> #
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> #
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> #
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> #
> at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:62)
> #
> at org.apache.hadoop.hbase.HStoreKey.readFields(HStoreKey.java:589)
> #
> at org.apache.hadoop.io.MapFile$Writer.checkKey(MapFile.java:213)
> #
> at org.apache.hadoop.io.MapFile$Writer.append(MapFile.java:192)
> #
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Writer.append(BloomFilterMapFile.java:201)
> #
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:674)
> #
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:627)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:852)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:852)
> {code}
> Looking in heap, the resevoir had not been released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.