[jira] Commented: (HBASE-722) Shutdown and Compactions

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645107#action_12645107 ] 

stack commented on HBASE-722:
-----------------------------

Billy, if you are around, see what you think of this patch.  I'll test it more but would appreciate an opinion.

> Shutdown and Compactions
> ------------------------
>
>                 Key: HBASE-722
>                 URL: https://issues.apache.org/jira/browse/HBASE-722
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>            Reporter: Billy Pearson
>            Assignee: stack
>             Fix For: 0.19.0
>
>         Attachments: 722.patch
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to  shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the  MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact 
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.