You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/01/04 10:23:45 UTC
[jira] Created: (HBASE-3409) Failed server shutdown processing when
retrying hlog split
Failed server shutdown processing when retrying hlog split
----------------------------------------------------------
Key: HBASE-3409
URL: https://issues.apache.org/jira/browse/HBASE-3409
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.0
Reporter: Todd Lipcon
Priority: Blocker
Fix For: 0.90.0
2011-01-04 01:14:17,353 WARN org.apache.hadoop.hbase.master.MasterFileSystem: Retrying splitting because of:
org.apache.hadoop.hbase.regionserver.wal.OrphanHLogAfterSplitException: Discovered orphan hlog after split. Maybe the HRegionServer was not dead when we started
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:286)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:187)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:196)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2011-01-04 01:14:17,353 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_SHUTDOWN
java.lang.IllegalStateException: An HLogSplitter instance may only be used once
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:170)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:199)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3409) Failed server shutdown processing when
retrying hlog split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3409:
-------------------------
Attachment: 3409.txt
Small fix. Review please.
> Failed server shutdown processing when retrying hlog split
> ----------------------------------------------------------
>
> Key: HBASE-3409
> URL: https://issues.apache.org/jira/browse/HBASE-3409
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3409.txt
>
>
> 2011-01-04 01:14:17,353 WARN org.apache.hadoop.hbase.master.MasterFileSystem: Retrying splitting because of:
> org.apache.hadoop.hbase.regionserver.wal.OrphanHLogAfterSplitException: Discovered orphan hlog after split. Maybe the HRegionServer was not dead when we started
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:286)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:187)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:196)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-01-04 01:14:17,353 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_SHUTDOWN
> java.lang.IllegalStateException: An HLogSplitter instance may only be used once
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:170)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:199)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3409) Failed server shutdown processing
when retrying hlog split
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977396#action_12977396 ]
Jean-Daniel Cryans commented on HBASE-3409:
-------------------------------------------
It seems it was introduced by HBASE-3323, at least the ISE was.
> Failed server shutdown processing when retrying hlog split
> ----------------------------------------------------------
>
> Key: HBASE-3409
> URL: https://issues.apache.org/jira/browse/HBASE-3409
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
>
> 2011-01-04 01:14:17,353 WARN org.apache.hadoop.hbase.master.MasterFileSystem: Retrying splitting because of:
> org.apache.hadoop.hbase.regionserver.wal.OrphanHLogAfterSplitException: Discovered orphan hlog after split. Maybe the HRegionServer was not dead when we started
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:286)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:187)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:196)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-01-04 01:14:17,353 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_SHUTDOWN
> java.lang.IllegalStateException: An HLogSplitter instance may only be used once
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:170)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:199)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3409) Failed server shutdown processing
when retrying hlog split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3409.
--------------------------
Resolution: Fixed
Assignee: stack
Hadoop Flags: [Reviewed]
Committed to branch and trunk. Thanks for review J-D.
> Failed server shutdown processing when retrying hlog split
> ----------------------------------------------------------
>
> Key: HBASE-3409
> URL: https://issues.apache.org/jira/browse/HBASE-3409
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3409.txt
>
>
> 2011-01-04 01:14:17,353 WARN org.apache.hadoop.hbase.master.MasterFileSystem: Retrying splitting because of:
> org.apache.hadoop.hbase.regionserver.wal.OrphanHLogAfterSplitException: Discovered orphan hlog after split. Maybe the HRegionServer was not dead when we started
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:286)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:187)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:196)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-01-04 01:14:17,353 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_SHUTDOWN
> java.lang.IllegalStateException: An HLogSplitter instance may only be used once
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:170)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:199)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3409) Failed server shutdown processing
when retrying hlog split
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977981#action_12977981 ]
Hudson commented on HBASE-3409:
-------------------------------
Integrated in HBase-TRUNK #1703 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1703/])
> Failed server shutdown processing when retrying hlog split
> ----------------------------------------------------------
>
> Key: HBASE-3409
> URL: https://issues.apache.org/jira/browse/HBASE-3409
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3409.txt
>
>
> 2011-01-04 01:14:17,353 WARN org.apache.hadoop.hbase.master.MasterFileSystem: Retrying splitting because of:
> org.apache.hadoop.hbase.regionserver.wal.OrphanHLogAfterSplitException: Discovered orphan hlog after split. Maybe the HRegionServer was not dead when we started
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:286)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:187)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:196)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-01-04 01:14:17,353 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_SHUTDOWN
> java.lang.IllegalStateException: An HLogSplitter instance may only be used once
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:170)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:199)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3409) Failed server shutdown processing
when retrying hlog split
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977551#action_12977551 ]
Jean-Daniel Cryans commented on HBASE-3409:
-------------------------------------------
A test would be nice eventually... +1 on the fix in the mean time.
> Failed server shutdown processing when retrying hlog split
> ----------------------------------------------------------
>
> Key: HBASE-3409
> URL: https://issues.apache.org/jira/browse/HBASE-3409
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: Todd Lipcon
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: 3409.txt
>
>
> 2011-01-04 01:14:17,353 WARN org.apache.hadoop.hbase.master.MasterFileSystem: Retrying splitting because of:
> org.apache.hadoop.hbase.regionserver.wal.OrphanHLogAfterSplitException: Discovered orphan hlog after split. Maybe the HRegionServer was not dead when we started
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:286)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:187)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:196)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-01-04 01:14:17,353 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_SHUTDOWN
> java.lang.IllegalStateException: An HLogSplitter instance may only be used once
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:170)
> at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:199)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:96)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.