You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Clint Morgan (JIRA)" <ji...@apache.org> on 2008/06/19 02:37:44 UTC
[jira] Created: (HBASE-698) HLog recovery is not performed after
master failure
HLog recovery is not performed after master failure
---------------------------------------------------
Key: HBASE-698
URL: https://issues.apache.org/jira/browse/HBASE-698
Project: Hadoop HBase
Issue Type: Bug
Components: master, regionserver
Affects Versions: 0.1.2
Reporter: Clint Morgan
I have a local cluster running, and its logging to
<hbase>/log_X.X.X.X_1213228101021_60020/
Then I kill both master and regionserver, and restart. Looking through
the logs I don't see anything about trying to recover from this hlog,
it just creates a new hlog alongside the existing one (with a new
startcode). The older hlog seems to be ignored, and the tables
created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729691#action_12729691 ]
stack commented on HBASE-698:
-----------------------------
I took another look at the patch. A backport don't look like it'd be that hard and it'd be a nice fix to have. Good on you J-D
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728371#action_12728371 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
If you scan the file system for region server logs, you can ignore the ones for region servers that ZK says
are alive.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728439#action_12728439 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
Jim, since you worked on this issue, would you like to try it before I commit? (thanks stack)
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730934#action_12730934 ]
Dave Latham commented on HBASE-698:
-----------------------------------
@JD
Thanks for the helpful info. It might be worth putting out a warning then with a 0.19.4 release to do that - possibly for migration to 0.20.0 also?
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728383#action_12728383 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
bq. If you scan the file system for region server logs, you can ignore the ones for region servers that ZK says are alive.
That's what my patch does.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728263#action_12728263 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
Ok, then I'll simply do a check of the .logs folder. It's going to be part of the verifyClusterState() method so that I have knowledge of existing RSs.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619082#action_12619082 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
Currently the master will only recover logs during startup if it finds information in in the ROOT or META that indicate that a region was being served. Without appends in hadoop, the edits to the ROOT or META could be lost and so the master would not look for the log of the server that was killed.
This is not a good way to do this. Once appends make log files persistent, the master should start up in safe mode, and first recover all the log files it finds. Region servers will not create their logs until the master has finished recovering all the existing logs and tells the region servers it has finished.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Fix For: 0.3.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Issue Type: Bug (was: Sub-task)
Parent: (was: HBASE-678)
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Fix For: 0.19.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728309#action_12728309 ]
stack commented on HBASE-698:
-----------------------------
I wonder why the code in BaseScanner was insufficient?
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Fix Version/s: (was: 0.19.0)
0.20.0
Moving this issue to 0.20.0 as it requires Zookeeper to close all the holes.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reassigned HBASE-698:
----------------------------------------
Assignee: Jean-Daniel Cryans
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729697#action_12729697 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
+1 on backport if it is not too complicated.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730930#action_12730930 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
@Dave
I have the same situation here and I'm doing some tests. I saw that edits going to since deleted regions will still create a directory and put the oldlogfile.log there. This is not so bad but not so good either, it eats up disk space for no reason... Edits going to META are more problematic, you could see old regions coming back and completely mess up your data. I'm currently trying to recreate such a setup to confirm.
So, if you want to apply this patch, I currently suggest that you delete those folders.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728331#action_12728331 ]
stack commented on HBASE-698:
-----------------------------
Ok.
So, j-d, why not mod BaseScanner to check zk rather than do the filesystem check? Filesystem check is more comprehensive?
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-698:
-------------------------------------
Attachment: hbase-698.patch
This patch fixes the original issue (or as much we can do without appends). This happens on a master failover:
{code}
2009-07-07 14:45:28,097 INFO org.apache.hadoop.hbase.master.HMaster: Found log folder : jdcryans.local,60020,1246991630883
2009-07-07 14:45:28,097 INFO org.apache.hadoop.hbase.master.HMaster: Log folder belongs to an existing region server
{code}
Whn know about the alive RS since we just scanned the rs folder in ZK. Then if I kill -9 both Master and RS then I restart the cluster I see:
{code}
2009-07-07 14:47:10,034 DEBUG org.apache.hadoop.hbase.master.HMaster: This is a fresh start, proceeding with normal startup
2009-07-07 14:47:10,038 INFO org.apache.hadoop.hbase.master.HMaster: Found log folder : jdcryans.local,60020,1246991630883
2009-07-07 14:47:10,038 INFO org.apache.hadoop.hbase.master.HMaster: Log folder doesn't belong to a known region server, splitting
2009-07-07 14:47:10,043 INFO org.apache.hadoop.hbase.regionserver.HLog: Splitting 1 hlog(s) in hdfs://localhost:9000/hbase/.logs/jdcryans.local,60020,1246991630883
...
{code}
I also removed the log splitting stuff in BaseScanner.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704821#action_12704821 ]
Andrew Purtell commented on HBASE-698:
--------------------------------------
The scope of this problem is narrowed by new developments. With multiple masters on standby and fail over via ZK, the possibility of encountering this situation is reduced.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731199#action_12731199 ]
stack commented on HBASE-698:
-----------------------------
Regards 0.20.x, we don't need to flag folks. Logs are now in a subdirectory named .logs. Old log files in old location will be ignored.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728112#action_12728112 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
In HBASE-1143 we set a very low flush size on .META. to make it flush after every ~10 updates which lessens the impact of a lost hlog. So the original situation can still happen in the way it is described or if a table split was done and that the region server holding META died pretty much right after that (because either a flush will happen or after 1 hour we will roll the commit log ). The hole is there but it's very small. The only thing missing is appends now. We could punt this to 0.21.0, unless someone votes for a hlog ZK orchestration for 0.20.0.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728167#action_12728167 ]
stack commented on HBASE-698:
-----------------------------
Can we not do something basic here? Even if its master looking into all regions on filesystem for logs to recover on startup.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HBASE-698:
-----------------------------------
Assignee: Jim Kellerman
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Fix For: 0.3.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728322#action_12728322 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
I don't think it had any idea of region servers that were running, so it would recover *all* logs, including
those from running region servers, which really messed things up.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Issue Type: Bug (was: Sub-task)
Parent: (was: HBASE-678)
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Fix Version/s: (was: 0.18.0)
0.19.0
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Fix For: 0.19.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711718#action_12711718 ]
stack commented on HBASE-698:
-----------------------------
HBASE-1439 is talking about ownership of commit logs being orchestrated by zk; perhaps on startup (or on assumption of master role) master can check zk if commit logs to replay.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-698:
-------------------------------------
Fix Version/s: 0.20.0
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HBASE-678
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Fix For: 0.19.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728416#action_12728416 ]
stack commented on HBASE-698:
-----------------------------
Ok. Thanks lads.
+1 on patch.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730935#action_12730935 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
Ah well I tested it and it's all OK because of the sequence ID. I used current 0.19, created a table, inserted 500 rows, then killed -9. I redid the same thing in the second run but with a different column family name and shut down clean. I then applied the patch, restarted and I saw:
{code}
2009-07-14 12:06:05,783 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region .META.,,1/1028785192
2009-07-14 12:06:05,789 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /tmp/hbase-jdcryans/hbase/.META./1028785192/info/info/7874001023615296020, isReference=false, sequence id=512, length=559, majorCompaction=false
2009-07-14 12:06:05,789 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 1 file(s) in hstore 1028785192/info, max sequence id 512
2009-07-14 12:06:05,791 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Applied 0, skipped 7 because sequence id <= 512
{code}
So basically it's ok.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730943#action_12730943 ]
stack commented on HBASE-698:
-----------------------------
Yes. Sequenceid should prevent our taking on old edits. I like the idea of release noting cleanup of dirs going 0.19.3 to 0.19.4 and I'll add cleanup to migration to 0.20.0. +1 on patch.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-698:
-----------------------------------
Assignee: (was: Jim Kellerman)
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728262#action_12728262 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
Yes, we need to do something. In particular, the master needs to know what region servers are running and what their start code is, so it does not try to recover a log out from under a running region server.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729661#action_12729661 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
I committed this to trunk. Should I make a patch for 0.19?
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-698:
-------------------------------------
Attachment: hbase-698-0.19.patch
Patch against 0.19 branch. It will recover any log directory found at the root HBase folder. I tried it on my machine by creating a table, inserting 500 rows into it and then killing --9 both Master and RS. After restart everything was there. This patch doesn't cover the situation where the logs weren't flush to disk (inserting 10 rows didn't trigger it) and it doesn't take into account existing RS like the patch for trunk does.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717848#action_12717848 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
When checking the cluster start in a master failover situation, we should check for HLogs to process.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-698:
------------------------
Fix Version/s: 0.2.0
Moving into 0.2 for now. Lets take a look at least at whats going on.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Fix For: 0.2.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-698:
-------------------------------------
Affects Version/s: 0.2.1
0.19.3
Fix Version/s: (was: 0.20.0)
0.19.4
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705671#action_12705671 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
> Andrew Purtell added a comment - 30/Apr/09 03:28 PM
> The scope of this problem is narrowed by new developments. With multiple masters on
> standby and fail over via ZK, the possibility of encountering this situation is reduced.
Agreed that the window for missing a lease loss narrows the window greatly.
Do the standby masters watch the region server leases?
The gist of the patch I was working on was to move the code from BaseScanner.checkAssigned
(lines 367 - 392) to a new method in HMaster that is called from HMaster.run() *before*
HMaster.startServiceThreads is called.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-698.
--------------------------------------
Resolution: Fixed
Release Note: Old setups running 0.19 should first delete everything that's log_* in HBase root HDFS folder before updating. Make sure HBase is down before doing that. Also this will be done by migration in 0.20.0
Hadoop Flags: [Reviewed]
Committed to branch 0.19.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646376#action_12646376 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
Removed Zookeeper integration as a blocker.
While Zookeeper will make this much easier, we need an interim solution.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728341#action_12728341 ]
Jean-Daniel Cryans commented on HBASE-698:
------------------------------------------
bq. I wonder why the code in BaseScanner was insufficient?
It was reacting only when seeing stale data in META or ROOT so if you lost an edit in META, you won't see it back in BaseScanner. I think BaseScanner never recovered anything ever because we don't have appends. Now we will have the issue of recovering regions without a META entry tho...
bq. So, j-d, why not mod BaseScanner to check zk rather than do the filesystem check? Filesystem check is more comprehensive?
ZK won't tell you about dead region servers, only the live ones.
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646344#action_12646344 ]
Jim Kellerman commented on HBASE-698:
-------------------------------------
There is a very simple fix if the master comes back up and knows a region server is dead.
However, if the master dies, region servers hang around until the master comes back up. Thus the master cannot know which HLogs to recover and which belong to running region servers. ("recovering" a HLog from a running region server would produce unpredictable results, most likely leading to data corruption).
Relying on hdfs lease timeouts on the log files is also not an option as the lease timeout interval is too long for this purpose.
The master can therefore not recover any region server logs unless it knows that region server is dead. This cannot be accomplished without Zookeeper integration, which will monitor the region servers (and the regions they serve) using ephemeral files. At that point, if the master dies and is restarted, it will know which region servers are alive, which ones have died and all the regions that are currently being served. Then it will know which region server logs to recover and which ones can be ignored (because the region server writing it is still alive).
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Priority: Blocker (was: Major)
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Fix Version/s: (was: 0.2.0)
0.3.0
Moving to 0.3.0 because it really depends on hbase "safe-mode" and hadoop appends which aren't supported until hadoop 0.18
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Fix For: 0.3.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730921#action_12730921 ]
Dave Latham commented on HBASE-698:
-----------------------------------
I have a cluster where I can see several old log directories in the root hbase folder. What would be the affect of deploying this patch? Would it be best to manually delete those directories first?
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.1.2, 0.2.1, 0.19.3
> Reporter: Clint Morgan
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.20.0, 0.19.4
>
> Attachments: hbase-698-0.19.patch, hbase-698.patch
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-698) HLog recovery is not performed after
master failure
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-698:
--------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HBASE-678
> HLog recovery is not performed after master failure
> ---------------------------------------------------
>
> Key: HBASE-698
> URL: https://issues.apache.org/jira/browse/HBASE-698
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Affects Versions: 0.1.2
> Reporter: Clint Morgan
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> I have a local cluster running, and its logging to
> <hbase>/log_X.X.X.X_1213228101021_60020/
> Then I kill both master and regionserver, and restart. Looking through
> the logs I don't see anything about trying to recover from this hlog,
> it just creates a new hlog alongside the existing one (with a new
> startcode). The older hlog seems to be ignored, and the tables
> created in the inital session are all gone.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.