You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Rong-En Fan (JIRA)" <ji...@apache.org> on 2009/05/30 17:16:17 UTC
[jira] Created: (HBASE-1459) CME during memcache flush forces HRS
shutdown
CME during memcache flush forces HRS shutdown
---------------------------------------------
Key: HBASE-1459
URL: https://issues.apache.org/jira/browse/HBASE-1459
Project: Hadoop HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.19.1
Reporter: Rong-En Fan
Fix For: 0.20.0, 0.19.4
I just saw this on my 0.19.1 cluster:
2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
Caused by: java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717549#action_12717549 ]
Rong-En Fan commented on HBASE-1459:
------------------------------------
just saw it on latest 0.19 branch on a single-box
{code}
2009-06-08 13:31:30,832 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Flush requested on table,,1244366951415
2009-06-08 13:31:30,937 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region table,,1244366951415. Current region memcache size 64.0m
[... Memcache: blahblahb: expired or in deletes, skipped ...]
2009-06-08 13:31:35,261 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: table,,1244366951415
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:903)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:796)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:265)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:148)
Caused by: java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:888)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:796)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:265)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:148)
{code}
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.19.5
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715335#action_12715335 ]
Rong-En Fan commented on HBASE-1459:
------------------------------------
stack, here you go, and I'm ok to move out this from 0.19.4.
2009-05-30 03:47:52,079 INFO org.apache.hadoop.hbase.regionserver.HLog: Closed hdfs://nn:8020/hbase/log_x.x.x.x_1243602404362_60020/hlog.dat.1243625003085, entries=100001. New log writer: /hbase/log_x.x.x.x_1243602404362_60020/hlog.dat.1243626472077
2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: table,row,1243547152622
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.j ava:897)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)Caused
by: java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
... 3 more
2009-05-30 03:57:19,572 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 60020, call batchUpdates([B@4051c386, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@37fbb367) from x.x.x.y:37001: error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2109)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1618)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.20.0, 0.19.4
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rong-En Fan updated HBASE-1459:
-------------------------------
Fix Version/s: (was: 0.19.4)
(was: 0.20.0)
0.19.5
move to 0.19.5 and remove 0.20.0 as it's not an issue there.
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.19.5
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715362#action_12715362 ]
stack commented on HBASE-1459:
------------------------------
I was hoping there'd be a clue in the log but doesn't seem to be. All seems normal. Oh well. I'll move it to 0.19.5 (smile). I'll roll a 0.19.4 with the meta/root fixes. Thats important.
Thanks Rong-en.
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.20.0, 0.19.4
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717658#action_12717658 ]
Rong-En Fan commented on HBASE-1459:
------------------------------------
I keep seeing this on a 1 master + 3 HRS setup w/ latest rev from branch-0.19. Good news is w/ 0.19.4 my cluster recovered after 15~20 mins. Let me dig into Memcache implementation to see if I can smell something...
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.19.5
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717702#action_12717702 ]
Rong-En Fan commented on HBASE-1459:
------------------------------------
Now I recall after a change in my client on May 27, I started to see this. I was seeing more today because I was pushing harder from client.
The client did something like this:
HTable.deleteAll(rowKey)
HTable.commit(BatchUpdate(rowKey, /data/))
in a very frequent way (many times per seconds). I was trying to clear out previous data because not every col may be overwritten when commit() is called.
Hope this helps.
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.19.5
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714744#action_12714744 ]
stack commented on HBASE-1459:
------------------------------
This should never happen (smile). We're iterating a memcache but it should be offlined when we are here; other threads should not be inserting. This issue we won't see in TRUNK because memcache is different type now (skiplistset).
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.20.0, 0.19.4
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714899#action_12714899 ]
stack commented on HBASE-1459:
------------------------------
Does this happen often Rong-en?
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.20.0, 0.19.4
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714903#action_12714903 ]
Rong-En Fan commented on HBASE-1459:
------------------------------------
Nope, it's the first time I saw on this 0.19.1 instance (for two months)
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.20.0, 0.19.4
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1459) CME during memcache flush forces HRS
shutdown
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715285#action_12715285 ]
stack commented on HBASE-1459:
------------------------------
Rong-en: Looking at this one, its an odd one. Its saying that the snapshot is being modified while we're flushing. The memcache makes a snapshot when it moves current memcache aside replacing it with a new one. Until the snapshot is successfully flushed, its read-only. Somehow a write is going into the snapshot. Should never happen. Can I have logs from around the above exception if you still have them.
Mind if I move this out of 0.19.4? It looks rare and its not going to happen in 0.20.0 because the memcache datastructure now can be iterated without throwing CME. Thanks Rong-en.
> CME during memcache flush forces HRS shutdown
> ---------------------------------------------
>
> Key: HBASE-1459
> URL: https://issues.apache.org/jira/browse/HBASE-1459
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.19.1
> Reporter: Rong-En Fan
> Fix For: 0.20.0, 0.19.4
>
>
> I just saw this on my 0.19.1 cluster:
> 2009-05-30 03:57:19,559 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: table,rowkey,1243547152622
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
> Caused by: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1136)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1131)
> at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:678)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:636)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:882)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.