You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2010/09/22 08:42:32 UTC
[jira] Created: (HBASE-3028) No basescanner means no GC'ing of
split, offlined parent regions
No basescanner means no GC'ing of split, offlined parent regions
----------------------------------------------------------------
Key: HBASE-3028
URL: https://issues.apache.org/jira/browse/HBASE-3028
Project: HBase
Issue Type: Bug
Components: master
Reporter: stack
Fix For: 0.90.0
I need to add back cleanup of split parents probably as a Chore in master. Can use new MetaReader code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3028) No basescanner means no GC'ing of
split, offlined parent regions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3028:
-------------------------
Attachment: 3028-v2.txt
Adding unit tests. Need to add a few more then cluster test.
> No basescanner means no GC'ing of split, offlined parent regions
> ----------------------------------------------------------------
>
> Key: HBASE-3028
> URL: https://issues.apache.org/jira/browse/HBASE-3028
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: stack
> Fix For: 0.90.0
>
> Attachments: 3028-v2.txt, 3028.txt
>
>
> I need to add back cleanup of split parents probably as a Chore in master. Can use new MetaReader code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3028) No basescanner means no GC'ing of
split, offlined parent regions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914336#action_12914336 ]
stack commented on HBASE-3028:
------------------------------
Did some cluster testing. When it first ran, it did a bunch of this:
{code}
2010-09-24 05:29:33,011 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughter usertable,user992373275,1285090830082.d821675081aebe369b1fb9c11d48b012. reference splitA from usertable,user992373275,1285019557053.4ae8a9836d6ba0e97e6017547bac9e6c. .META.
2010-09-24 05:29:33,014 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughter usertable,user996181626,1285090830082.77acffdf0fbb0128ef6a4afd800906ab. reference splitB from usertable,user992373275,1285019557053.4ae8a9836d6ba0e97e6017547bac9e6c. .META.
2010-09-24 05:29:33,014 INFO org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region usertable,user992373275,1285019557053.4ae8a9836d6ba0e97e6017547bac9e6c. because daughter splits no longer hold references
2010-09-24 05:29:33,014 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://sv2borg180:20000/hbase/usertable/4ae8a9836d6ba0e97e6017547bac9e6c
2010-09-24 05:29:33,033 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region usertable,user992373275,1285019557053.4ae8a9836d6ba0e97e6017547bac9e6c. from META
2010-09-24 05:29:33,033 INFO org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1089 catalog row(s) and gc'd 375 unreferenced parent region(s)
{code}
I let it run a while. Checked table is wholesome still and it passes.
Going to commit.
> No basescanner means no GC'ing of split, offlined parent regions
> ----------------------------------------------------------------
>
> Key: HBASE-3028
> URL: https://issues.apache.org/jira/browse/HBASE-3028
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: stack
> Fix For: 0.90.0
>
> Attachments: 3028-v2.txt, 3028-v4.txt, 3028.txt
>
>
> I need to add back cleanup of split parents probably as a Chore in master. Can use new MetaReader code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3028) No basescanner means no GC'ing of
split, offlined parent regions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913423#action_12913423 ]
stack commented on HBASE-3028:
------------------------------
Or, look into having the regionserver do it after its completed a compaction (sloppy scanner process might be better way to go)
> No basescanner means no GC'ing of split, offlined parent regions
> ----------------------------------------------------------------
>
> Key: HBASE-3028
> URL: https://issues.apache.org/jira/browse/HBASE-3028
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: stack
> Fix For: 0.90.0
>
>
> I need to add back cleanup of split parents probably as a Chore in master. Can use new MetaReader code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3028) No basescanner means no GC'ing of
split, offlined parent regions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3028:
-------------------------
Attachment: 3028.txt
Adding back a basescanner that looks for parents to clean. This attached patch is not yet complete. Its mostly rejiggering the 0.20 scanners to work in new context.
> No basescanner means no GC'ing of split, offlined parent regions
> ----------------------------------------------------------------
>
> Key: HBASE-3028
> URL: https://issues.apache.org/jira/browse/HBASE-3028
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: stack
> Fix For: 0.90.0
>
> Attachments: 3028.txt
>
>
> I need to add back cleanup of split parents probably as a Chore in master. Can use new MetaReader code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3028) No basescanner means no GC'ing of
split, offlined parent regions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3028:
-------------------------
Attachment: 3028-v4.txt
Here is a complete patch:
{code}
M src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
Test of new basescanner (now called catalogjanitor). Mocks up
a 'Server' and a 'MasterServies'. Tests the check for references
and gc'ing of let-go of offlined parent regions.
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(removeRegionFromMeta): Removed. Users should be going via
MetaEditor instead.
M src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
(isSplitParent): Added.
M src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
Minor formatting
M src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
New chore that runs every 5 minutes. It scans .META. and if
it finds a parent row that is offlined and marked as product of
a split, it will check daughters to see if they have references.
If none, the row is removed and its content in FS is cleaned up.
This is the old 0.20 BaseScanner brought forward into TRUNK
only now it does this one check only (The fixup of dropped
splits is done as part of server shutdown handling in TRUNK).
M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Added in start and stop of new CatalogJanitor Chore. Also
made a stopServiceThreads to match startServiceThreads. I
moved into this new method from 'run' the stopping of all threads
started in startServiceThreads.
M src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java
Refactored fullScan so it now works by taking a Visitor. Default
fullScan does as it used to only via a Visitor now. Adding in
this facility means I can use the fullScan code when doing the
catalog janitorial chore.
M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
(deleteDaughterReferenceInparent): Added.
M src/main/java/org/apache/hadoop/hbase/util/Threads.java
Added a setDaemonThreadRunning override that gets the name to use
from the passed thread.
{code}
> No basescanner means no GC'ing of split, offlined parent regions
> ----------------------------------------------------------------
>
> Key: HBASE-3028
> URL: https://issues.apache.org/jira/browse/HBASE-3028
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: stack
> Fix For: 0.90.0
>
> Attachments: 3028-v2.txt, 3028-v4.txt, 3028.txt
>
>
> I need to add back cleanup of split parents probably as a Chore in master. Can use new MetaReader code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3028) No basescanner means no GC'ing of
split, offlined parent regions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3028.
--------------------------
Assignee: stack
Resolution: Fixed
Committed on TRUNK
> No basescanner means no GC'ing of split, offlined parent regions
> ----------------------------------------------------------------
>
> Key: HBASE-3028
> URL: https://issues.apache.org/jira/browse/HBASE-3028
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: stack
> Assignee: stack
> Fix For: 0.90.0
>
> Attachments: 3028-v2.txt, 3028-v4.txt, 3028.txt
>
>
> I need to add back cleanup of split parents probably as a Chore in master. Can use new MetaReader code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.