You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jonathan Hsieh (Created) (JIRA)" <ji...@apache.org> on 2012/02/09 01:58:00 UTC
[jira] [Created] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
[uberhbck] Add options for how to handle offline split parents.
----------------------------------------------------------------
Key: HBASE-5360
URL: https://issues.apache.org/jira/browse/HBASE-5360
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.0, 0.90.7, 0.92.1
Reporter: Jonathan Hsieh
In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
More safe guards to prevent "mega-merges" are being added on HBASE-5128.
This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400010#comment-13400010 ]
Hudson commented on HBASE-5360:
-------------------------------
Integrated in HBase-0.94-security #37 (See [https://builds.apache.org/job/HBase-0.94-security/37/])
HBASE-5360 [uberhbck] Add options for how to handle offline split parents. (Revision 1351182)
Result = SUCCESS
jxiang :
Files :
* /hbase/branches/0.94/CHANGES.txt
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485299#comment-13485299 ]
Hudson commented on HBASE-5360:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #240 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/240/])
HBASE-6223 Document hbck improvements: HBASE-6173, HBASE-5360 (Revision 1402650)
Result = FAILURE
jxiang :
Files :
* /hbase/trunk/src/docbkx/book.xml
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.94.1, 0.96.0
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang updated HBASE-5360:
-------------------------------
Status: Open (was: Patch Available)
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.94.0, 0.92.1, 0.90.7
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-5360) [uberhbck] Add options for how
to handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400592#comment-13400592 ]
Jimmy Xiang edited comment on HBASE-5360 at 6/25/12 5:12 PM:
-------------------------------------------------------------
Good catch. Fixed as an addendum. Thanks.
was (Author: jxiang):
Good catch. I will fix it in a separate jira. Thanks.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang reassigned HBASE-5360:
----------------------------------
Assignee: Jimmy Xiang
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485225#comment-13485225 ]
Hudson commented on HBASE-5360:
-------------------------------
Integrated in HBase-TRUNK #3489 (See [https://builds.apache.org/job/HBase-TRUNK/3489/])
HBASE-6223 Document hbck improvements: HBASE-6173, HBASE-5360 (Revision 1402650)
Result = FAILURE
jxiang :
Files :
* /hbase/trunk/src/docbkx/book.xml
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.94.1, 0.96.0
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Anoop Sam John (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253154#comment-13253154 ]
Anoop Sam John commented on HBASE-5360:
---------------------------------------
@Jon
HBASE-5719 Handling this issue?
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403274#comment-13403274 ]
Hudson commented on HBASE-5360:
-------------------------------
Integrated in HBase-0.94-security #38 (See [https://builds.apache.org/job/HBase-0.94-security/38/])
HBASE-5360 [uberhbck] Add options for how to handle offline split parents, addendum (Revision 1353659)
Result = FAILURE
jxiang :
Files :
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang updated HBASE-5360:
-------------------------------
Attachment: 5360_hbase_v4.patch
5360-0.94-hbase.patch
5360-0.92-hbase.patch
5360-0.90-hbase.patch
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl closed HBASE-5360.
--------------------------------
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.94.1, 0.96.0
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405115#comment-13405115 ]
Jimmy Xiang commented on HBASE-5360:
------------------------------------
@Ram, good question. For lingering parent, the idea is to clear the split flag and let the rest to be taken care by hbck, which will handle assignment too. I considered to fix it in one shot. But it is some duplicate effort already in hbck. It is much clean and safe to use hbck to handle the assignment,
region overlaps, and so on.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405074#comment-13405074 ]
ramkrishna.s.vasudevan commented on HBASE-5360:
-----------------------------------------------
@Jimmy
I have a question here. Currently the LINGERING_PARENT is added back to META but still the region is not deployed. In the testcase added we try to do doFsck(conf, true) which does the assignment also.
What happens is when we have a lingering parent and we say fixSplitParent we fix the META but do not assign it. So when we run hbck it says Region Not Deployed. Do we need to do assignment also here? Correct me if am wrong .
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290523#comment-13290523 ]
Zhihong Ted Yu commented on HBASE-5360:
---------------------------------------
Using other options to fix is fine.
The explanation should be put in documentation for hbck.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290448#comment-13290448 ]
Jimmy Xiang commented on HBASE-5360:
------------------------------------
For the children, they will be handled by other options like fixHdfsHoles and fixHdfsOverlaps. Not sure if we need flags for the two scenarios now. We can add if there many such requests.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290422#comment-13290422 ]
Jimmy Xiang commented on HBASE-5360:
------------------------------------
Those helper functions are in two places: StoreFile and LoadIncrementalHFiles. I need to add some more utils. I think it is better to put them in one class.
You are right, a child could be added multiple times. I will move it outside. I will address the test cases too. Thanks.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jonathan Hsieh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253187#comment-13253187 ]
Jonathan Hsieh commented on HBASE-5360:
---------------------------------------
@Anoop. Partially. HBASE-5719 sidelines potential overlapping regions but doesn't take .META.'s info about if the region is offline into account. It could sideline a live region while it could have been more efficient to sideline an offline region. However, HBASE-5719 was sufficient for the case we were dealing with.
Let's make this a placeholder for ways to improve it. Some ideas include:
* Taking a region's offline/splitparent state into account if .META. entry is present
* Making decisions about sidelining vs merging based on region size and max region size (instead of range) xml properties.
* Improving the heuristic used to decide which region are sidelined.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405180#comment-13405180 ]
ramkrishna.s.vasudevan commented on HBASE-5360:
-----------------------------------------------
@Jimmy
Thanks for your reply. :)
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401047#comment-13401047 ]
Hudson commented on HBASE-5360:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #68 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/68/])
HBASE-5360 [uberhbck] Add options for how to handle offline split parents, addendum (Revision 1353658)
Result = FAILURE
jxiang :
Files :
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400381#comment-13400381 ]
ramkrishna.s.vasudevan commented on HBASE-5360:
-----------------------------------------------
Small nit
{code}
p.add(HConstants.CATALOG_FAMILY, HConstants.SPLITA_QUALIFIER,
Writables.getBytes(a));
p.add(HConstants.CATALOG_FAMILY, HConstants.SPLITA_QUALIFIER,
Writables.getBytes(b));
{code}
In the testcase, the second put should be for SPLITB?
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang updated HBASE-5360:
-------------------------------
Status: Patch Available (was: Open)
This patch is for trunk. I will put it on review board once it is online again.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.94.0, 0.92.1, 0.90.7
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang updated HBASE-5360:
-------------------------------
Fix Version/s: 0.94.1
0.96.0
0.92.2
0.90.7
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.94.0, 0.92.1, 0.90.7
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400592#comment-13400592 ]
Jimmy Xiang commented on HBASE-5360:
------------------------------------
Good catch. I will fix it in a separate jira. Thanks.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291186#comment-13291186 ]
Jimmy Xiang commented on HBASE-5360:
------------------------------------
Sure. I posted the code review here:
https://review.cloudera.org/r/2122/
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290300#comment-13290300 ]
Zhihong Ted Yu commented on HBASE-5360:
---------------------------------------
{code}
- Path referencePath = getReferredToFile(p);
+ Path referencePath = StoreFileUtil.getReferredToFile(p);
{code}
Can you outline the rationale behind moving methods to StoreFileUtil ?
For sidelineSplitParent():
{code}
+ if (needReassign) {
+ toBeReassigned.add(child);
+ }
{code}
The above code is inside inner loop. Would a child be added to toBeReassigned multiple times ?
{code}
+ // scenario (2)
+ errors.reportError(ERROR_CODE.FAILED_SPLIT_PARENT, "Region "
+ + descriptiveName + ", key=" + key + ", on HDFS, failed split parent");
+ if (shouldFixSplitParents()) {
+ resetSplitParent(hbi);
{code}
Do we need to do something about the children in above scenario ?
I think we should provide two flags to user corresponding to the two scenarios so that they can decide which scenario(s) to fix.
{code}
+ * Daughters do refer to parent.
+ */
+ @Test
+ public void testLingeringSplitParent2() throws Exception {
{code}
Please give the two test cases meaningful names that are consistent with javadoc.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang updated HBASE-5360:
-------------------------------
Attachment: hbase-5360.path
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393473#comment-13393473 ]
Jonathan Hsieh commented on HBASE-5360:
---------------------------------------
I've +1'd on reviewboard with a few minor nits. Feel free to update and commit if you see fit.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393640#comment-13393640 ]
Hadoop QA commented on HBASE-5360:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12532353/5360_hbase_v4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2178//console
This message is automatically generated.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289848#comment-13289848 ]
Hadoop QA commented on HBASE-5360:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531032/hbase-5360.path
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestSplitLogManager
org.apache.hadoop.hbase.client.TestFromClientSide
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2112//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2112//console
This message is automatically generated.
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Attachments: hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5360) [uberhbck] Add options for how to
handle offline split parents.
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jimmy Xiang updated HBASE-5360:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> [uberhbck] Add options for how to handle offline split parents.
> ----------------------------------------------------------------
>
> Key: HBASE-5360
> URL: https://issues.apache.org/jira/browse/HBASE-5360
> Project: HBase
> Issue Type: Improvement
> Components: hbck
> Affects Versions: 0.90.7, 0.92.1, 0.94.0
> Reporter: Jonathan Hsieh
> Assignee: Jimmy Xiang
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 5360-0.90-hbase.patch, 5360-0.92-hbase.patch, 5360-0.94-hbase.patch, 5360_hbase_v4.patch, hbase-5360.path
>
>
> In a recent case, we attempted to repair a cluster that suffered from HBASE-4238 that had about 6-7 generations of "leftover" split data. The hbck repair options in an development version of HBASE-5128 treat HDFS as ground truth but didn't check SPLIT and OFFLINE flags only found in meta. The net effect was that it essentially attempted to merge many regions back into its eldest geneneration's parent's range.
> More safe guards to prevent "mega-merges" are being added on HBASE-5128.
> This issue would automate the handling of the "mega-merge" avoiding cases such as "lingering grandparents". The strategy here would be to add more checks against .META., and perform part of the catalog janitor's responsibilities for lingering grandparents. This would potentially include options to sideline regions, deleting grandparent regions, min size for sidelining, and mechanisms for cleaning .META..
> Note: There already exists an mechanism to reload these regions -- the bulk loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents (automatically splitting them if necessary) to HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira