You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2010/08/03 04:16:16 UTC
[jira] Created: (HBASE-2896) Retain assignment information between
cluster shutdown/startup
Retain assignment information between cluster shutdown/startup
--------------------------------------------------------------
Key: HBASE-2896
URL: https://issues.apache.org/jira/browse/HBASE-2896
Project: HBase
Issue Type: New Feature
Components: master, regionserver
Reporter: Jonathan Gray
Fix For: 0.90.0
Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment normally.
This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2896) Retain assignment information
between cluster shutdown/startup
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929220#action_12929220 ]
HBase Review Board commented on HBASE-2896:
-------------------------------------------
Message from: "Jonathan Gray" <jg...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1186/
-----------------------------------------------------------
Review request for hbase, stack and khemani.
Summary
-------
Adds a configuration option (default true in this patch) for retaining assignment on startup. If true, initial assignment plan attempts to use existing assignment info from META (we don't delete assignment info in META when RS closes region).
Also changes initial assignment to skip regions from disabled tables rather than looking at offline flag.
This addresses bug HBASE-2896.
http://issues.apache.org/jira/browse/HBASE-2896
Diffs
-----
trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 1032133
trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1032133
trunk/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java 1032133
trunk/src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java 1032133
Diff: http://review.cloudera.org/r/1186/diff
Testing
-------
I will write a new unit test of the load balancing method to make sure it does what is expected.
I could write a cluster-level TestRetainAssignment test which actually tests it works end to end but I'm not sure it's necessary. Could also add some modification to TestClusterRestart or whatever that one is.
Thanks,
Jonathan
> Retain assignment information between cluster shutdown/startup
> --------------------------------------------------------------
>
> Key: HBASE-2896
> URL: https://issues.apache.org/jira/browse/HBASE-2896
> Project: HBase
> Issue Type: New Feature
> Components: master, regionserver
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.92.0
>
>
> Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment randomly.
> This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2896) Retain assignment information between
cluster shutdown/startup
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-2896:
---------------------------------
Fix Version/s: (was: 0.90.0)
0.92.0
Assignee: Jonathan Gray
Punting to 0.92 and assigning to myself. I'm going to start work on this in 1-2 weeks.
> Retain assignment information between cluster shutdown/startup
> --------------------------------------------------------------
>
> Key: HBASE-2896
> URL: https://issues.apache.org/jira/browse/HBASE-2896
> Project: HBase
> Issue Type: New Feature
> Components: master, regionserver
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.92.0
>
>
> Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment randomly.
> This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2896) Retain assignment information between
cluster shutdown/startup
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-2896:
---------------------------------
Description:
Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment randomly.
This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
was:
Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment normally.
This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
(changed normally to randomly)
> Retain assignment information between cluster shutdown/startup
> --------------------------------------------------------------
>
> Key: HBASE-2896
> URL: https://issues.apache.org/jira/browse/HBASE-2896
> Project: HBase
> Issue Type: New Feature
> Components: master, regionserver
> Reporter: Jonathan Gray
> Fix For: 0.90.0
>
>
> Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment randomly.
> This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2896) Retain assignment information between
cluster shutdown/startup
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-2896:
---------------------------------
Attachment: HBASE-2896-v2.patch
Patch committed. Same as patch reviewed by stack plus new unit test.
> Retain assignment information between cluster shutdown/startup
> --------------------------------------------------------------
>
> Key: HBASE-2896
> URL: https://issues.apache.org/jira/browse/HBASE-2896
> Project: HBase
> Issue Type: New Feature
> Components: master, regionserver
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.92.0
>
> Attachments: HBASE-2896-v2.patch
>
>
> Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment randomly.
> This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2896) Retain assignment information between
cluster shutdown/startup
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray resolved HBASE-2896.
----------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.92.0)
0.90.0
Hadoop Flags: [Reviewed]
Committed to trunk.
> Retain assignment information between cluster shutdown/startup
> --------------------------------------------------------------
>
> Key: HBASE-2896
> URL: https://issues.apache.org/jira/browse/HBASE-2896
> Project: HBase
> Issue Type: New Feature
> Components: master, regionserver
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.90.0
>
> Attachments: HBASE-2896-v2.patch
>
>
> Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment randomly.
> This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2896) Retain assignment information
between cluster shutdown/startup
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929940#action_12929940 ]
stack commented on HBASE-2896:
------------------------------
I committed this to see if it fixes build failure up on hudson:
{code}
pynchon-379:trunk stack$ svn diff src
Index: src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
===================================================================
--- src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java (revision 1032837)
+++ src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java (working copy)
@@ -404,7 +404,8 @@
assignments.put(server, new ArrayList<HRegionInfo>());
}
for (Map.Entry<HRegionInfo, HServerAddress> region : regions.entrySet()) {
- HServerInfo server = serverMap.get(region.getValue());
+ HServerAddress hsa = region.getValue();
+ HServerInfo server = hsa == null? null: serverMap.get(hsa);
if (server != null) {
assignments.get(server).add(region.getKey());
} else {
{code}
See https://hudson.apache.org/hudson/job/HBase-TRUNK/1629/ failures.
> Retain assignment information between cluster shutdown/startup
> --------------------------------------------------------------
>
> Key: HBASE-2896
> URL: https://issues.apache.org/jira/browse/HBASE-2896
> Project: HBase
> Issue Type: New Feature
> Components: master, regionserver
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.90.0
>
> Attachments: HBASE-2896-v2.patch
>
>
> Over in HBASE-57 we want to consider block locations for region assignment. This is most important during cluster startup where you currently lose all locality because regions are assignment randomly.
> This jira is about a shot-term solution to the cluster startup problem by retaining assignment information after a cluster shutdown and using it on the next cluster startup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.