You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Doug Meil (Created) (JIRA)" <ji...@apache.org> on 2012/01/17 20:29:44 UTC
[jira] [Created] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
[book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
---------------------------------------------------------------------------
Key: HBASE-5218
URL: https://issues.apache.org/jira/browse/HBASE-5218
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Trivial
Attachments: book_hbase_5218.xml.patch
Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187958#comment-13187958 ]
stack commented on HBASE-5218:
------------------------------
+1
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189012#comment-13189012 ]
Hudson commented on HBASE-5218:
-------------------------------
Integrated in HBase-TRUNK #2639 (See [https://builds.apache.org/job/HBase-TRUNK/2639/])
hbase-5218 book.xml - Arch/HFile, added link to HFile v2 info in appendix from this section.
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "Doug Meil (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187947#comment-13187947 ]
Doug Meil commented on HBASE-5218:
----------------------------------
All these 'hbck' comments are probably for HBASE-5128, not this issue.
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "Doug Meil (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-5218:
-----------------------------
Attachment: book_hbase_5218.xml.patch
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188387#comment-13188387 ]
Hudson commented on HBASE-5218:
-------------------------------
Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/])
hbase-5218 book.xml - Arch/HFile, added link to HFile v2 info in appendix from this section.
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187923#comment-13187923 ]
jiraposter@reviews.apache.org commented on HBASE-5218:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3435/#review4263
-----------------------------------------------------------
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
<https://reviews.apache.org/r/3435/#comment9627>
This causes unit test failure.
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
<https://reviews.apache.org/r/3435/#comment9628>
This causes unit test failure.
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
<https://reviews.apache.org/r/3435/#comment9629>
This causes unit test failure.
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
<https://reviews.apache.org/r/3435/#comment9630>
This causes unit test failure.
- jmhsieh
On 2012-01-09 08:58:17, jmhsieh wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3435/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-01-09 08:58:17)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, and Michael Stack.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. I'm posting a preliminary version that I'm currently testing on real clusters. The tests are flakey on the 0.90 branch (so there is something async that I didn't synchronize properly), and there are a few more TODO's I want to knock out before this is ready for full review to be considered for committing. It's got some problems I need some advice figuring out.
bq.
bq. Problem 1:
bq.
bq. In the unit tests, I have a few cases where I fabricate new regions and try to force the overlapping regions to be closed. For some of these, I cannot delete a table after it is repaired without causing subsequent tests to fail. I think this is due to a few things:
bq.
bq. 1) The disable table handler uses in-memory assignment manager state while delete uses in META assignment information.
bq. 2) Currently I'm using the sneaky closeRegion that purposely doesn't go through the master and in turn doesn't modify in-memory state – disable uses out of date in-memory region assignments. If I use the unassign method sends RIT transitions to the master, but which ends up attempting to assign it again, causing timing/transient states.
bq.
bq. What is a good way to clear the HMaster's assignment manager's assignment data for particular regions or to force it to re-read from META? (without modifying the 0.90 HBase's it is meant to repair).
bq.
bq. Problem 2:
bq.
bq. Sometimes test fail reporting HOLE_IN_REGION_CHAIN and SERVER_DOES_NOT_MATCH_META. This means the old and new regions are confiused with each other and basically something is still happening asynchronously. I think this is the new region is being assigned and is still transitioning. Sound about right? To make the unit test deterministic, should hbck wait for these to settle or should just the unit test wait?
bq.
bq.
bq. This addresses bug HBASE-5218.
bq. https://issues.apache.org/jira/browse/HBASE-5218
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 6d3401d
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java a3d8b8b
bq. src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 29e8bb2
bq. src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a640d57
bq. src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java dbb97f8
bq. src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java 3e8729d
bq. src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java 11a1151
bq. src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java 4a09ce2
bq.
bq. Diff: https://reviews.apache.org/r/3435/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. All unit tests pass sometimes. Some fail sometimes (generally the cases that fabricate new regions).
bq.
bq. Not ready for commit.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "Jonathan Hsieh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187970#comment-13187970 ]
Jonathan Hsieh commented on HBASE-5218:
---------------------------------------
@Doug
I updated review board -- sorry for my minor dyslexia.
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "Doug Meil (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-5218:
-----------------------------
Status: Patch Available (was: Open)
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "Doug Meil (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-5218:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile,
added link to HFile v2 info in appendix
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187922#comment-13187922 ]
jiraposter@reviews.apache.org commented on HBASE-5218:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3435/
-----------------------------------------------------------
Review request for hbase, Todd Lipcon, Ted Yu, and Michael Stack.
Summary
-------
I'm posting a preliminary version that I'm currently testing on real clusters. The tests are flakey on the 0.90 branch (so there is something async that I didn't synchronize properly), and there are a few more TODO's I want to knock out before this is ready for full review to be considered for committing. It's got some problems I need some advice figuring out.
Problem 1:
In the unit tests, I have a few cases where I fabricate new regions and try to force the overlapping regions to be closed. For some of these, I cannot delete a table after it is repaired without causing subsequent tests to fail. I think this is due to a few things:
1) The disable table handler uses in-memory assignment manager state while delete uses in META assignment information.
2) Currently I'm using the sneaky closeRegion that purposely doesn't go through the master and in turn doesn't modify in-memory state – disable uses out of date in-memory region assignments. If I use the unassign method sends RIT transitions to the master, but which ends up attempting to assign it again, causing timing/transient states.
What is a good way to clear the HMaster's assignment manager's assignment data for particular regions or to force it to re-read from META? (without modifying the 0.90 HBase's it is meant to repair).
Problem 2:
Sometimes test fail reporting HOLE_IN_REGION_CHAIN and SERVER_DOES_NOT_MATCH_META. This means the old and new regions are confiused with each other and basically something is still happening asynchronously. I think this is the new region is being assigned and is still transitioning. Sound about right? To make the unit test deterministic, should hbck wait for these to settle or should just the unit test wait?
This addresses bug HBASE-5218.
https://issues.apache.org/jira/browse/HBASE-5218
Diffs
-----
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 6d3401d
src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java a3d8b8b
src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 29e8bb2
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a640d57
src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java dbb97f8
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java 3e8729d
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java 11a1151
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java 4a09ce2
Diff: https://reviews.apache.org/r/3435/diff
Testing
-------
All unit tests pass sometimes. Some fail sometimes (generally the cases that fabricate new regions).
Not ready for commit.
Thanks,
jmhsieh
> [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
> ---------------------------------------------------------------------------
>
> Key: HBASE-5218
> URL: https://issues.apache.org/jira/browse/HBASE-5218
> Project: HBase
> Issue Type: Improvement
> Reporter: Doug Meil
> Assignee: Doug Meil
> Priority: Trivial
> Attachments: book_hbase_5218.xml.patch
>
>
> Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira