You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/06/25 03:07:07 UTC
[jira] Created: (HBASE-1583) Start/Stop of large cluster untenable
Start/Stop of large cluster untenable
-------------------------------------
Key: HBASE-1583
URL: https://issues.apache.org/jira/browse/HBASE-1583
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.20.0
Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
At pset with our > 100 nodes carrying 6k regions:
+ shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
+ startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724350#action_12724350 ]
Andrew Purtell commented on HBASE-1583:
---------------------------------------
My understanding for 0.21 is the region assignment process is going to be largely unmediated by the master, except for the case where the master finds an unassigned region in META and puts up a node into the "to be assigned" queue out in ZK. My opinion this is the way to go, but how a regionserver is to judge its load relative to others, or even learn about the load of others, is an unanswered question. Furthermore, there must be a mechanism in place such that some regionserver will take on a new region if all others have passed on it. Is there an issue up for this type of stuff yet?
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723845#action_12723845 ]
ryan rawson commented on HBASE-1583:
------------------------------------
some thoughts:
- get rid of compactions during startup/shutdown. There is no reason to run compactions at those times.
- dont worry about compacting reference regions as quickly, we can wait a few minutes if necessary.
- make shut down flush memstore as fast as possible, then get the heck outta there.
we need to do more work on bringing up regions as fast as possible in 0.21. The master needs to blast region assignments out there in a threaded/performant manner, rather than waiting for 3-second checkins to do their magic.
Finally I have decided to up the size of a region on my own set-up. 256MB seems a little too small, and with even twice as large regions, i end up with less than half the region count!
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727145#action_12727145 ]
stack commented on HBASE-1583:
------------------------------
Small jgray suggested optimization; don't compact on open unless has References
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724588#action_12724588 ]
Jean-Daniel Cryans commented on HBASE-1583:
-------------------------------------------
+1 on this issue
Also btw the Master redesign is HBASE-1110.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732156#action_12732156 ]
stack commented on HBASE-1583:
------------------------------
With this patch applied and with HBASE-1058 reverted, up and down as well as enable/disable is much smoother.
Let me now take a look at 'safe mode'. Talking with JK, looks like its not working properly. All regions were supposed to be assigned while in safe mode but not working.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch, 1583-v2-nocompactonopenclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724373#action_12724373 ]
stack commented on HBASE-1583:
------------------------------
We should open an issue Andrew. What'll we call it? Master redesign?
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732170#action_12732170 ]
Andrew Purtell commented on HBASE-1583:
---------------------------------------
Ack on safe mode having oddities. I brought down a cluster with 133 regions cleanly and restarted it just now. Right away ~128 regions were assigned out. The rest were assigned out a few minutes later. Invoking 'enable' on a incompletely assigned table prodded the master into some action but did not bring things up all the way as this kludge has done in the past.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch, 1583-v2-nocompactonopenclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1583) Start/Stop of large cluster untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1583.
--------------------------
Resolution: Fixed
Release Note: No compaction on disable or shutdown of cluster. No disable on open or enable unless the region has references.
Hadoop Flags: [Reviewed]
Committed to branch and trunk. Resolving this issue. There is more we can do but this I think is enough for 0.20.0.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch, 1583-v2-nocompactonopenclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724327#action_12724327 ]
Jonathan Gray commented on HBASE-1583:
--------------------------------------
I like your idea, ryan. Next generation of balancing needs to take into account actual usage that comprises a complex notion of "load" including memcache, er memstore, usage, block cache usage, and as you say, an additional factor could be how many other regions from the same table are already assigned to that regionserver.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732122#action_12732122 ]
Jonathan Gray commented on HBASE-1583:
--------------------------------------
Looks good to me.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch, 1583-v2-nocompactonopenclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1583) Start/Stop of large cluster untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1583:
-------------------------
Attachment: 1583-v2-nocompactonopenclose.patch
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch, 1583-v2-nocompactonopenclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723883#action_12723883 ]
stack commented on HBASE-1583:
------------------------------
Safe mode is still there. Thats just period during which all machines report in and during which we hand out catalog regions. After safe mode elapses, then the mayhem breaks out as master tries to hand out 6k regions ten or so at a time balancing at same time.
Region assignment needs to part of larger scale rewrite of master function. Hows does a master figure a region unassigned? It reads .META. table to figure current state. We need to be careful how we bridge scan of .META. and read of zk.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725220#action_12725220 ]
Billy Pearson commented on HBASE-1583:
--------------------------------------
+1 on the kill compaction on shutdown why not just kill the thread and not let it finish it will get done on restart so we will not lose anything.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723885#action_12723885 ]
ryan rawson commented on HBASE-1583:
------------------------------------
It would also be neato to have more optimal global assignment of regions. Any given table should be spread out as much as possible.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732100#action_12732100 ]
Jonathan Gray commented on HBASE-1583:
--------------------------------------
+1 on 1583-nocompactonclose.patch
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723861#action_12723861 ]
Jim Kellerman commented on HBASE-1583:
--------------------------------------
@stack:
> startup is a mess with our assigning out regions an rebalancing at same time. By time that the
> compactions on open run, it can be near an hour before whole thing settles down and becomes
> useable
safe mode was to prevent rebalancing during startup. Are we not using safe mode anymore?
@ryan
> The master needs to blast region assignments out there in a threaded/performant manner
What I was planning to do for 0.21 when I put region assignments in ZK was to make a znode
whose children are unassigned regions. A region server can then decide if it is too busy or
not, and if not, remove the unassigned region from the list and add it to its list of regions being
served once the region is available.
This removes the master from region assignment. It would then only need to detect unassigned
regions.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732243#action_12732243 ]
stack commented on HBASE-1583:
------------------------------
I took a look at this safe mode stuff. Its broken. Will open an issue. Whats happening is that we exit safe mode near immediately after startup because initial MetaScanner scan does nothing except set that initial scan has completed (though it did nothing -- original idea was that initialScan would do first scan of the newly deployed .META.). So, we exit safe mode near immediately after startup.
Fixing metascanner so initial scan doesn't happen till we've scanned actual deploy so safe mode stays in place while deploy is going on kills our assignment rate. It crawls. I gave up trying to debug more since these above patches undoing compactions on close and open seem to be enough to close this issue at least for 0.20.0 release.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch, 1583-v2-nocompactonopenclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1583) Start/Stop of large cluster untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1583:
-------------------------
Attachment: 1583-nocompactonclose.patch
Here is a patch to disable compaction on close of a region (helps with disable of table too).
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725218#action_12725218 ]
Billy Pearson commented on HBASE-1583:
--------------------------------------
I suggested that we do not do come out of safe mode until all regions have been assigned when we added safe mode and make the regions not run compactions while in safe mode I thank that would be an easy fix for this problem
I have seen the same thing when you have region that are behind on compactions after a shutdown on start up compaction tie up reassignments.
Billy
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1583) Start/Stop of large cluster untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-1583:
----------------------------
Assignee: stack
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1583) Start/Stop of large cluster
untenable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732103#action_12732103 ]
stack commented on HBASE-1583:
------------------------------
This patch adds to the previous. It adds no compaction on open unless region has references.
> Start/Stop of large cluster untenable
> -------------------------------------
>
> Key: HBASE-1583
> URL: https://issues.apache.org/jira/browse/HBASE-1583
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1583-nocompactonclose.patch, 1583-v2-nocompactonopenclose.patch
>
>
> Starting and stopping a loaded large cluster is way too flakey and takes too long. This is 0.19.x but same issues apply to TRUNK I'd say.
> At pset with our > 100 nodes carrying 6k regions:
> + shutdown takes way too long.... maybe ten minutes or so. We compact regions inline with shutdown. We should just go down. It doesn't seem like all regionservers go down everytime either.
> + startup is a mess with our assigning out regions an rebalancing at same time. By time that the compactions on open run, it can be near an hour before whole thing settles down and becomes useable
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.