You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Clint Morgan (JIRA)" <ji...@apache.org> on 2009/07/30 23:29:14 UTC
[jira] Created: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Migration to 0.20 looses index information from HTableDescriptor
----------------------------------------------------------------
Key: HBASE-1726
URL: https://issues.apache.org/jira/browse/HBASE-1726
Project: Hadoop HBase
Issue Type: Bug
Components: contrib
Reporter: Clint Morgan
Fix For: 0.20.0
In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737321#action_12737321 ]
Andrew Purtell commented on HBASE-1726:
---------------------------------------
Maybe we need some kind of hook out to contribs for migration?
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737744#action_12737744 ]
Clint Morgan commented on HBASE-1726:
-------------------------------------
Edited the migration wiki:
Removed the line about unknown mirgation for trx/indexing.
Added a paragraph about running the tool above.
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
> Attachments: 1726.patch
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737727#action_12737727 ]
stack commented on HBASE-1726:
------------------------------
Yeah, usually we'd add the old HTD class under migration.nineteen and then use it in deserializing the bytes we got scanning.
We're not doing that in this migration because since HTD and HCD were moved to be backed my a Map, then rare is need to read the old data with the old class (Does that make sense? Most Key/Values are in the map now). Unfortunately not all are in the Map. That seems to be the case here?
Its a pain having to put in place the old classes. Can you not do something simple like set a flag on migration that says "I'm migrating a ITHBase". Then down in the scan of catalog regions, add in the missing 'indexed' flag.
Or, sounds like you figured a trick already to move forward ITHBase. Mind adding documentation to the wiki on how you did it? Then we can close this issue?
THanks Clint.
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1726.
--------------------------
Resolution: Fixed
Committed to 0.19 branch. Reviewed wiki update. All looks good. Thanks for the patch Clint.
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
> Attachments: 1726.patch
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737353#action_12737353 ]
Andrew Purtell commented on HBASE-1726:
---------------------------------------
Could do it with annotations and classpath scanning. Then add to wiki and javadoc instructions to the effect that any user with some contrib should make sure to put the new contrib version on the classpath along with the new core jar version before running migration. Migration utility can scan the classpath for annotated classes which implement a migration interface with methods preMigration(), migrate(), and postMigration() or similar.
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737721#action_12737721 ]
Clint Morgan commented on HBASE-1726:
-------------------------------------
To get this migration to work. I need to get a reference to the old HTD which has the indexes.
Digging around a bit, I don't see where the old HTD's are migrated to the new HTD format (which droppes the indexes stuff). Where does this happen? Wouldn't we need a copy of the old HTD to read in the old HTD's bytes.
For my migration, I wrote/ran it against 0.19. Go through all tables, if they have indexes, move that info into the HTD.values, then HBaseAdmin.modifyTable to save the HTD. After that, do normal 0.20 migration. Then the migrated indexes worked. Nothing to migrate to get THBase to work.
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clint Morgan updated HBASE-1726:
--------------------------------
Attachment: 1726.patch
This is what I did for my migration. Patch is for 0.19 branch
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
> Attachments: 1726.patch
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737351#action_12737351 ]
stack commented on HBASE-1726:
------------------------------
Leaving in 0.20.0. I don't think this enough to sink the 0.20.0 RC. Can apply to head of the 0.20.0 branch and those who need it can update to catch this fixup.
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1726) Migration to 0.20 looses index
information from HTableDescriptor
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737350#action_12737350 ]
stack commented on HBASE-1726:
------------------------------
Good idea Andrew. Would call each contrib in turn at various stages: before, during, after. Lets make an issue but not bother with dev. till needed (smile).
In this case, change should be part of the core migration since transaction was core in 0.19.
So, Clint, with this fix, you were able to migrate a 0.19 THBase to 0.20? If so, update the wiki where it says we don't migrate THBase nor ITHBase.
You'd patch http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/util/Migrate.java?revision=797273. See migrate6to7. You'd probably want to patch the rewriteHRegionInfo method.
> Migration to 0.20 looses index information from HTableDescriptor
> ----------------------------------------------------------------
>
> Key: HBASE-1726
> URL: https://issues.apache.org/jira/browse/HBASE-1726
> Project: Hadoop HBase
> Issue Type: Bug
> Components: contrib
> Reporter: Clint Morgan
> Fix For: 0.20.0
>
>
> In 0.19 index info was stored as a private field, and written specially in the Writable impl of HTD. In 0.20 it is stored in the general values map under the key "INDEXES". So migration needs to read in the old HTD format, and store the indexes in the values map. I wrote a quick standalone thing to do this. Can someone point me where to plug this into general migration process?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.