You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Billy Pearson (JIRA)" <ji...@apache.org> on 2008/07/04 07:31:33 UTC
[jira] Created: (HBASE-722) Shutdown and Compactions
Shutdown and Compactions
------------------------
Key: HBASE-722
URL: https://issues.apache.org/jira/browse/HBASE-722
Project: Hadoop HBase
Issue Type: Improvement
Components: regionserver
Reporter: Billy Pearson
Priority: Trivial
Fix For: 0.3.0
Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
This causes the shutdown time to take a vary long time depending on what region the server is compacting.
I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
So it closes region 1 and waits for region 2 to finish compaction before closing it.
The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
With a lot of regions this could hang some region server a long time to Shutdown the cluster.
Solutions 1 or 2 below will work I thank
1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-722) Shutdown and Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611691#action_12611691 ]
stack commented on HBASE-722:
-----------------------------
I think the compactions are inlined with region open at the moment but I do not think it would be much work to queue a compaction and proceed with the region deploy.
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Priority: Trivial
> Fix For: 0.3.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-722) Shutdown and Compactions
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-722:
--------------------------------
Priority: Major (was: Trivial)
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Jim Kellerman
> Fix For: 0.19.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-722) Shutdown and Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645107#action_12645107 ]
stack commented on HBASE-722:
-----------------------------
Billy, if you are around, see what you think of this patch. I'll test it more but would appreciate an opinion.
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 722.patch
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-722) Shutdown and Compactions
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-722:
--------------------------------
Issue Type: Bug (was: Improvement)
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Jim Kellerman
> Priority: Trivial
> Fix For: 0.19.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-722) Shutdown and Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611448#action_12611448 ]
stack commented on HBASE-722:
-----------------------------
If a shutdown has been asked for, yeah, I think we should close rather than compact and close.
We'll compact on open too if we find region needs it so open will be slow if we don't do it on close.
We need to do it in one of the places else system can be overrun.
What do you think more important; fast shutdown or deploy?
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Priority: Trivial
> Fix For: 0.3.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-722) Shutdown and Compactions
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611459#action_12611459 ]
Billy Pearson commented on HBASE-722:
-------------------------------------
Sense we compact in the background if needed after loading the region on startup and the compaction thread does not effect the loading of newly assigned region opens from the master. I thank shutdown should close all regions unless compacting and close the last one after the compaction is done. That will not always be a fast shutdown but it will be faster then if there is compaction on more then one region while shutting down.
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Priority: Trivial
> Fix For: 0.3.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-722) Shutdown and Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-722.
-------------------------
Resolution: Fixed
Committed (Thanks for trying it Billy).
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 722.patch
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-722) Shutdown and Compactions
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646139#action_12646139 ]
Billy Pearson commented on HBASE-722:
-------------------------------------
+1 looks good I only tested with 2 regions but looks like the regions are both getting closed in parallel.
that will solve the problem of multi regions compactions happening on the same server at shutdown
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 722.patch
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-722) Shutdown and Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-722:
---------------------------
Assignee: stack (was: Jim Kellerman)
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: stack
> Fix For: 0.19.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-722) Shutdown and Compactions
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HBASE-722:
-----------------------------------
Assignee: Jim Kellerman
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: Jim Kellerman
> Priority: Trivial
> Fix For: 0.19.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-722) Shutdown and Compactions
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611828#action_12611828 ]
Billy Pearson commented on HBASE-722:
-------------------------------------
I thank we have always queue up the compaction's on start up. might check but I have always seen this when start with a lot of regions needing compaction. so I thank that part is fine.
The part on shutdown the simplest solution above is the second one
Empty the compaction queue so it has no other regions to compact when done with the current one on shutdown message received.
I thank that would be a simple fix and should workout fine for stopping the compaction of more then the current compaction on shutdown.
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Priority: Trivial
> Fix For: 0.3.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-722) Shutdown and Compactions
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646039#action_12646039 ]
Billy Pearson commented on HBASE-722:
-------------------------------------
If it still apply to trunk I will test tonight.
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 722.patch
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-722) Shutdown and Compactions
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-722:
--------------------------------
Fix Version/s: (was: 0.18.0)
0.19.0
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Billy Pearson
> Priority: Trivial
> Fix For: 0.19.0
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-722) Shutdown and Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-722:
------------------------
Attachment: 722.patch
Add more checks for server stop to compactsplit thread. Don't start split or even compaction if we've been asked stop. Don't queue requests for compaction if we've been asked stop. Threaded close of regions so all run in parallel. Move static methods from head of HRegion to tail. People expect constructors at top of classes, not utility methods.
> Shutdown and Compactions
> ------------------------
>
> Key: HBASE-722
> URL: https://issues.apache.org/jira/browse/HBASE-722
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Billy Pearson
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 722.patch
>
>
> Currently I thank the region server closes the region spits in a set order this same order is used in the compaction/spit thread after a startup and all regions need compaction.
> This causes the shutdown time to take a vary long time depending on what region the server is compacting.
> I have seen my cluster take more then 60 mins to shutdown waiting for several regions to finish compaction.
> The problem I am seeing is say the compaction thread is working on a on region 2 of 4 in the list.
> When the region server get the MSG_REGIONSERVER_QUIESCE from the master it closes in order from 1-4
> So it closes region 1 and waits for region 2 to finish compaction before closing it.
> The problem is it never closes region split 3 or 4 while waiting for region 2 to complete the compaction.
> So example say 3 and 4 regions are waiting to be compacted also. When region 2 is done region 3 start compaction within milliseconds of 2's finished compaction.
> This happens faster then the region server can close region 3 so the region server hangs around compacting regions until it run out of open regions needing compact.
> With a lot of regions this could hang some region server a long time to Shutdown the cluster.
> Solutions 1 or 2 below will work I thank
> 1. Close all regions not currently getting compaction so when the compaction thread complete it has no other regions open to compact
> 2. Empty the compaction que so it has no other regions to compact when done with the current one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.