You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/09/13 21:57:10 UTC
[jira] [Created] (HBASE-4389) Address lots of issues with migration
from 90 to trunk
Address lots of issues with migration from 90 to trunk
------------------------------------------------------
Key: HBASE-4389
URL: https://issues.apache.org/jira/browse/HBASE-4389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Priority: Critical
Fix For: 0.92.0
Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4389) Address lots of issues with migration
from 90 to trunk
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4389:
---------------------------------
Fix Version/s: (was: 0.94.1)
(was: 0.92.2)
Release Note: Unscheduling. It doesn't look like anything is working on this.
> Address lots of issues with migration from 90 to trunk
> ------------------------------------------------------
>
> Key: HBASE-4389
> URL: https://issues.apache.org/jira/browse/HBASE-4389
> Project: HBase
> Issue Type: Bug
> Components: master, migration
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
>
> Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4389) Address lots of issues with migration
from 90 to trunk
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-4389:
-------------------------------
Component/s: migration
> Address lots of issues with migration from 90 to trunk
> ------------------------------------------------------
>
> Key: HBASE-4389
> URL: https://issues.apache.org/jira/browse/HBASE-4389
> Project: HBase
> Issue Type: Bug
> Components: master, migration
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4389) Address lots of issues with
migration from 90 to trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133306#comment-13133306 ]
stack commented on HBASE-4389:
------------------------------
Patch in hbase-4388 addresses some of above.
+ It increments HRI VERSION (and makes HRI able to deserialize the old format that has HTD)
+ We no longer have a migrated flag but a version in root for meta table (a short)
+ Migration should be idempotent now; we look at each row and check its HRI VERSION and if not current, do update
+ Duplicated code has been removed
+ We no longer do createTableDescriptor on each HRI scanned; just on the first.
Outstanding:
+ Migration code has been coalesced some but still code spread about
+ There is no sanity checking that all HTDs for a table are equal. Not sure what we'd do if they were not equal other than warn. We wouldn't know which of them all to take. Patch in hbase-4388 takes the first regions HTD. This is usually the first updated; later updates could fail so its the first is likely the 'latest' iteration on HTD I'd guess.
> Address lots of issues with migration from 90 to trunk
> ------------------------------------------------------
>
> Key: HBASE-4389
> URL: https://issues.apache.org/jira/browse/HBASE-4389
> Project: HBase
> Issue Type: Bug
> Components: master, migration
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4389) Address lots of issues with
migration from 90 to trunk
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103938#comment-13103938 ]
Todd Lipcon commented on HBASE-4389:
------------------------------------
After a quick pass through the migration code, here are the various issues I see:
- HRegionInfo didn't have its VERSION incremented. Hence exception catching is used to try to determine which version is being read.
- A single "migrated" boolean flag is used in ROOT to indicate that META has been updated to the new format. This leaves us no room for future migrations. "migrated" is not a boolean. It should instead be "migratedToVersion" or something
- Migration should be idempotent - ie even if the "migratedToVersion" flag didn't get updated, migration should be able to re-run without crashing
- Duplicated code between updateRootWithNewRegionInfo and updateMetaWithNewRegionInfo
- Each region that is processed results in a call to createTableDescriptor, which results in calls to the NN - this will take a long time on a big cluster, and is unnecessary
- No sanity checking that all of the HTDs for a table are equal
- Migration code should ideally be moved to a separate class, instead of mixed with the non-migration code paths
> Address lots of issues with migration from 90 to trunk
> ------------------------------------------------------
>
> Key: HBASE-4389
> URL: https://issues.apache.org/jira/browse/HBASE-4389
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4389) Address lots of issues with migration
from 90 to trunk
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-4389:
---------------------------------
Fix Version/s: (was: 0.94.0)
0.94.1
Are we trying to fix these? Moving to 0.94.1 for now.
> Address lots of issues with migration from 90 to trunk
> ------------------------------------------------------
>
> Key: HBASE-4389
> URL: https://issues.apache.org/jira/browse/HBASE-4389
> Project: HBase
> Issue Type: Bug
> Components: master, migration
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Fix For: 0.92.2, 0.94.1
>
>
> Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4389) Address lots of issues with migration
from 90 to trunk
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4389:
-------------------------
Priority: Major (was: Critical)
Making this major rather than critical since a good few of the listed items have been addressed by hbase-4388.
> Address lots of issues with migration from 90 to trunk
> ------------------------------------------------------
>
> Key: HBASE-4389
> URL: https://issues.apache.org/jira/browse/HBASE-4389
> Project: HBase
> Issue Type: Bug
> Components: master, migration
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Fix For: 0.92.0
>
>
> Looking over the migration code that removes HTD from HRI, there are lots of issues. This JIRA is to redo this code in a way that will be less bug prone, and also future proof.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira