You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Sebastien Rainville (JIRA)" <ji...@apache.org> on 2008/07/11 20:06:31 UTC
[jira] Created: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
bin/hbase migrate upgrade fails when redo logs exists
-----------------------------------------------------
Key: HBASE-743
URL: https://issues.apache.org/jira/browse/HBASE-743
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0
Reporter: Sebastien Rainville
Priority: Minor
I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "Izaak Rubin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614911#action_12614911 ]
Izaak Rubin commented on HBASE-743:
-----------------------------------
I've updated the HBase Wiki page on migrations, mentioning the issue of unrecovered redo logs:
http://wiki.apache.org/hadoop/Hbase/HowToMigrate
I'm not sure if there's a way for HBase to go through and merge-in these redo logs - if so, I'll post it to the migration page as well. If something like this doesn't exist, it might be a good idea to make it.
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "Sebastien Rainville (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615437#action_12615437 ]
Sebastien Rainville commented on HBASE-743:
-------------------------------------------
It's good enough.
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "Izaak Rubin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613484#action_12613484 ]
Izaak Rubin commented on HBASE-743:
-----------------------------------
Hi Sebastien,
I'm having difficulty replicating the problem on my machine. I don't totally understand what you mean by the hbase-0.1.3 cleanup, but I'll explain what I did and you can tell me if I did something different. I started a version of hbase-0.1.3 running on top of hadoop-0.16.4, created a table with a bunch of data, shut down hbase, and then upgraded my hadoop from 0.16.4 to 0.17.1. I followed the hadoop upgrading instructions at http://wiki.apache.org/hadoop/Hadoop%20Upgrade. Then I went to my hbase-trunk and ran:
{code}
./bin/hbase migrate update
{code}
Which ran successfully. I then started up hbase-trunk and the table i had created looked unchanged.
It's true that the migrate script will fail if it sees these stray log files in the dfs. Shutting down hbase normally should clean up these log files... This is where I'm a bit confused about what you mean by "hbase-0.1.3 cleanup," and that even if you do stop it it'll leave redo-logs? Wouldn't it be that if you didn't stop the cleanup it leaves redo-logs? (Since the cleanup should get rid of the redo logs?)
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-743) bin/hbase migrate upgrade
fails when redo logs exists
Posted by "Izaak Rubin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614779#action_12614779 ]
irubin edited comment on HBASE-743 at 7/18/08 3:08 PM:
------------------------------------------------------------
I see, that makes sense. I'm not sure that this is necessarily a problem with the migrate command, although I suppose it could be a bit more tolerant. I'll ask Jim or Stack about this.
Still, I think the right thing to do would be to just warn people about this before they migrate, and show them how to remove any stray redo logs. I'll make a note of this on the wiki page.
was (Author: irubin):
I see, that makes sense. I'm not sure that this is necessarily a problem with the migrate command, although I suppose it could be a bit more tolerant. I'll ask Jim or Stack about this.
Still, I think the right thing to do would be to just warn people about this before they migrate, and point them to the directory where any redo logs might be hanging around. I'll make a note of this on the wiki page.
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615383#action_12615383 ]
stack commented on HBASE-743:
-----------------------------
Sebastien, what with Izaak's documentation above and my change to the message that comes out of the migration when logs are present, is that sufficient to close this issue?
Here is the message that comes out of the migration when redo logs are present:
{code}
if (unrecoveredLogs.size() != 0) {
throw new IOException("There are " + unrecoveredLogs.size() +
" unrecovered region server logs. Please uninstall this version of " +
"HBase, re-install the previous version, start your cluster and " +
"shut it down cleanly, so that all region server logs are recovered" +
" and deleted. Or, if you are sure logs are vestige of old " +
"failures in hbase, remove them and then rerun the migration. " +
"See 'Redo Logs' in http://wiki.apache.org/hadoop/Hbase/HowToMigrate. " +
"Here are the problem log files: " + unrecoveredLogs);
}
{code}
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "Izaak Rubin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Izaak Rubin reassigned HBASE-743:
---------------------------------
Assignee: Izaak Rubin
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-743.
-------------------------
Resolution: Fixed
Fix Version/s: 0.2.0
Resolving (with Sebastien's OK)
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
> Fix For: 0.2.0
>
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "Izaak Rubin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614779#action_12614779 ]
Izaak Rubin commented on HBASE-743:
-----------------------------------
I see, that makes sense. I'm not sure that this is necessarily a problem with the migrate command, although I suppose it could be a bit more tolerant. I'll ask Jim or Stack about this.
Still, I think the right thing to do would be to just warn people about this before they migrate, and point them to the directory where any redo logs might be hanging around. I'll make a note of this on the wiki page.
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-743) bin/hbase migrate upgrade fails when
redo logs exists
Posted by "Sebastien Rainville (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614690#action_12614690 ]
Sebastien Rainville commented on HBASE-743:
-------------------------------------------
Here's how I reproduce the problem:
(in hbase-0.1.3)
rm -Rf $HBASE_DATA_DIR
bin/start-hbase.sh
ls $HBASE_DATA_DIR => a redo log exists
bin/stop-hbase.sh
ls $HBASE_DATA_DIR => the redo log is not there anymore: so far so good
bin/start-hbase.sh
ls $HBASE_DATA_DIR => a redo log exists
kill -9 $HBASE_MASTER_PID
ls $HBASE_DATA_DIR => a redo log still exists
bin/start-hbase.sh
ls $HBASE_DATA_DIR => now there are 2 redo logs
bin/stop-hbase.sh
ls $HBASE_DATA_DIR => now there is only 1 redo log: the bad one
(in hbase-trunk)
bin/hbase migrate upgrade => FAIL because of an unrecovered redo log
Now, I don't know how that's supposed to be handled but the thing is that HBase wasn't always very *stable* so remaining redo logs like that are likely to exist for other people so it's going to come up again.
> bin/hbase migrate upgrade fails when redo logs exists
> -----------------------------------------------------
>
> Key: HBASE-743
> URL: https://issues.apache.org/jira/browse/HBASE-743
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Sebastien Rainville
> Assignee: Izaak Rubin
> Priority: Minor
>
> I migrated several hbase-0.1.3 instances to hbase trunk and even if I stop hbase-0.1.3 cleanup it leaves redo logs on hdfs. The problems is that when migrating the data with hbase-trunk it fails because it finds these redo-logs and quit with a error message saying that we should reinstall the old hbase and shut it down cleanly and that in theory it erases the redo logs. The work around has been to delete the redo logs manually... which is bad.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.