You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Gopinathan A (JIRA)" <ji...@apache.org> on 2012/05/18 15:20:06 UTC
[jira] [Created] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Gopinathan A created HBASE-6046:
-----------------------------------
Summary: Master retry on ZK session expiry causes inconsistent region assignments.
Key: HBASE-6046
URL: https://issues.apache.org/jira/browse/HBASE-6046
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.92.1
Reporter: Gopinathan A
Fix For: 0.92.2, 0.94.1
1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl closed HBASE-6046.
--------------------------------
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1, 0.96.0
>
> Attachments: HBASE_6046_0.92.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch, HBASE_6046_0.94.patch, HBASE_6046-trunk.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289842#comment-13289842 ]
Hudson commented on HBASE-6046:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #42 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/42/])
HBASE-6046 Master retry on ZK session expiry causes inconsistent region assignments. (Asutosh) (Revision 1346460)
Result = FAILURE
ramkrishna :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.92.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288835#comment-13288835 ]
stack commented on HBASE-6046:
------------------------------
+1 on patch. I'm surprised it works for all startup paths but if there is a code path that has a master coming up only part-initialized, I'll blame bad code coverage rather than this patch so go commit Ram.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Status: Open (was: Patch Available)
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0, 0.92.1
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288492#comment-13288492 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
Some minor changes may be needed in this patch. Will upload once done.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288756#comment-13288756 ]
Zhihong Ted Yu commented on HBASE-6046:
---------------------------------------
bq. New ServerManager is not created
I think there is a typo above: 'not' -> 'now'
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284008#comment-13284008 ]
Zhihong Yu commented on HBASE-6046:
-----------------------------------
If master doesn't know a region server has come down, where would it call finishInitialization() again ?
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287628#comment-13287628 ]
Hadoop QA commented on HBASE-6046:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530583/HBASE_6046_0.94_2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2086//console
This message is automatically generated.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278839#comment-13278839 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
The problem here is when the master retries to come out of zk expiry exception and if he succeeds the entire master is almost recreated in the sense
{code}
try {
if (!becomeActiveMaster(status)) {
return Boolean.FALSE;
}
initializeZKBasedSystemTrackers();
// Update in-memory structures to reflect our earlier Root/Meta assignment.
assignRootAndMeta(status);
// process RIT if any
// TODO: Why does this not call AssignmentManager.joinCluster? Otherwise
// we are not processing dead servers if any.
assignmentManager.processDeadServersAndRegionsInTransition();
{code}
Here the initializeZKBasedSystemTrackers() will even create new AssignmentManager. So what ever he does in processDeadServersAndRegionsInTransition() is like a fresh start.
So in processDeadServersAndRegionsInTransition()
{code}
for (Map.Entry<HRegionInfo, ServerName> e: this.regions.entrySet()) {
if (!e.getKey().isMetaTable()
&& e.getValue() != null) {
LOG.debug("Found " + e + " out on cluster");
this.failover = true;
break;
}
{code}
Though all the RS is online we will have the 'this.regions' empty and hence we go with completely new assignment.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Jindal updated HBASE-6046:
-----------------------------------
Hadoop Flags: (was: Reviewed)
Status: Patch Available (was: Open)
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0, 0.92.1
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287467#comment-13287467 ]
Ashutosh Jindal commented on HBASE-6046:
----------------------------------------
Please check the second testcase added testLogSplittingAfterMasterRecoveryDueToZKExpiry() .If the testcase is run without the patch , stackOverFlow exception is thrown.
{code}
java.lang.StackOverflowError
at java.lang.System.getProperty(System.java:647)
at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:67)
at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.PrintWriter.<init>(PrintWriter.java:78)
at java.io.PrintWriter.<init>(PrintWriter.java:62)
at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:58)
at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:485)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:623)
at org.apache.zookeeper.ClientCnxn$EventThread.queuePacket(ClientCnxn.java:477)
at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:640)
at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:658)
at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1274)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:975)
at org.apache.hadoop.hbase.master.SplitLogManager.deleteNode(SplitLogManager.java:626)
at org.apache.hadoop.hbase.master.SplitLogManager.access$17(SplitLogManager.java:620)
at org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback.processResult(SplitLogManager.java:1104)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:619)
at org.apache.zookeeper.ClientCnxn$EventThread.queuePacket(ClientCnxn.java:477)
at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:640)
at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:658)
at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1274)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:975)
at org.apache.hadoop.hbase.master.SplitLogManager.deleteNode(SplitLogManager.java:626)
at org.apache.hadoop.hbase.master.SplitLogManager.access$17(SplitLogManager.java:620)
at org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback.processResult(SplitLogManager.java:1104)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:619)
at org.apache.zookeeper.ClientCnxn$EventThread.queuePacket(ClientCnxn.java:477)
{code}
This is coming because the listener for splitLogManager is not registered after the master recovers from expired zk session.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289695#comment-13289695 ]
Hudson commented on HBASE-6046:
-------------------------------
Integrated in HBase-0.92 #450 (See [https://builds.apache.org/job/HBase-0.92/450/])
HBASE-6046 Master retry on ZK session expiry causes inconsistent region assignments. (Ashutosh) (Revision 1346487)
Result = FAILURE
ramkrishna :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.92.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287451#comment-13287451 ]
Ashutosh Jindal commented on HBASE-6046:
----------------------------------------
Attached patch for 0.94 version. Please review and provide your suggestion/comments.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284026#comment-13284026 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
When the master is retrying to come up in
{code}
private boolean tryRecoveringExpiredZKSession() throws InterruptedException,
IOException, KeeperException, ExecutionException {
{code}
We tend to just initialize the zk trackers, assign root and meta and finally assign any regions in transition.
But in this time if an RS has gone down we totally miss those callback and the logs are never splitted. Also as the AM is reinitialized we always treat as a clean cluster start.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289527#comment-13289527 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
Committed to 0.94 and trunk. Working on a patch for 0.92 as it is not uploaded here.
Thanks Ashutosh for the patch.
Thanks to Ted and Stack for the review.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287453#comment-13287453 ]
Hadoop QA commented on HBASE-6046:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530554/HBASE_6046_0.94.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2083//console
This message is automatically generated.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288844#comment-13288844 ]
Hadoop QA commented on HBASE-6046:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530823/HBASE_6046-trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.replication.TestReplication
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2099//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2099//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2099//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2099//console
This message is automatically generated.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283976#comment-13283976 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
I think its better to do finishInitialization step fully. Ashutosh and myself saw problems in the current way of handlng
-> MAster not knowing that a RS has gone down.
-> So no split happens
-> So we need to do all the steps in finish initialization.
-> We tried to do all the steps from waitForRegionServers till the end, but here again the masterfilesystem is initializing the splitlogmanager.
So can we call finish initialization itself once again?
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289536#comment-13289536 ]
Hudson commented on HBASE-6046:
-------------------------------
Integrated in HBase-TRUNK #2989 (See [https://builds.apache.org/job/HBase-TRUNK/2989/])
HBASE-6046 Master retry on ZK session expiry causes inconsistent region assignments. (Asutosh) (Revision 1346460)
Result = FAILURE
ramkrishna :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6046) Master retry on ZK session
expiry causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283976#comment-13283976 ]
ramkrishna.s.vasudevan edited comment on HBASE-6046 at 5/26/12 1:08 PM:
------------------------------------------------------------------------
I think its better to do finishInitialization step fully. Ashutosh and myself saw problems in the current way of handlng
-> MAster not knowing that a RS has gone down.
-> So no split happens
-> We tried to do all the steps from waitForRegionServers till the end, but here again the masterfilesystem is initializing the splitlogmanager.
So can we call finish initialization itself once again?
was (Author: ram_krish):
I think its better to do finishInitialization step fully. Ashutosh and myself saw problems in the current way of handlng
-> MAster not knowing that a RS has gone down.
-> So no split happens
-> So we need to do all the steps in finish initialization.
-> We tried to do all the steps from waitForRegionServers till the end, but here again the masterfilesystem is initializing the splitlogmanager.
So can we call finish initialization itself once again?
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Attachment: HBASE_6046-trunk.patch
Updated patch for 0.94 and trunk.
New ServerManager is not created. This is needed because the Timeout monitor in AM and splitlogmanager uses the server manager.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Attachment: HBASE_6046_0.94_2.patch
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Jindal updated HBASE-6046:
-----------------------------------
Status: Open (was: Patch Available)
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0, 0.92.1
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287632#comment-13287632 ]
Zhihong Yu commented on HBASE-6046:
-----------------------------------
Patch v2 looks good.
Minor comment:
{code}
- ".splitLogManagerTimeoutMonitor");
+ public void finishInitialization(boolean masterRecovery) {
{code}
Add javadoc for the method and masterRecovery parameter.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289580#comment-13289580 ]
Hudson commented on HBASE-6046:
-------------------------------
Integrated in HBase-0.94 #248 (See [https://builds.apache.org/job/HBase-0.94/248/])
HBASE-6046 Master retry on ZK session expiry causes inconsistent region assignments. (Ashutosh) (Revision 1346461)
Result = FAILURE
ramkrishna :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.92.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287476#comment-13287476 ]
Hadoop QA commented on HBASE-6046:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530559/HBASE_6046_0.94_1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2084//console
This message is automatically generated.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287526#comment-13287526 ]
Zhihong Yu commented on HBASE-6046:
-----------------------------------
I ran the new test and it passed.
{code}
}
+ public void finishInitialization() {
+ finishInitialization(false);
{code}
Please add javadoc for the above method. Leave one empty line between the previous method and finishInitialization().
In test code:
{code}
+ public static class MockLoadBalancer extends DefaultLoadBalancer {
{code}
The above class can be private.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283973#comment-13283973 ]
Ashutosh Jindal commented on HBASE-6046:
----------------------------------------
In the above scenario, if an RS goes down after abtaining a new zookeeper and before initializing ZKBasedSystemTrackers, then the master will not know about it and no log will be split.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Jindal updated HBASE-6046:
-----------------------------------
Fix Version/s: (was: 0.94.1)
(was: 0.92.2)
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0, 0.92.1
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Status: Patch Available (was: Open)
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.0, 0.92.1
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287475#comment-13287475 ]
Ashutosh Jindal commented on HBASE-6046:
----------------------------------------
@Anoop
Thanks for the review. Uplaoded patch addressing the comments.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan reassigned HBASE-6046:
---------------------------------------------
Assignee: ramkrishna.s.vasudevan
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287932#comment-13287932 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
If the patch is ok can i prepare for trunk also. I can fix the comment on commit.
@Stack
Pls reivew and provide your comments on this.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Jindal updated HBASE-6046:
-----------------------------------
Attachment: HBASE_6046_0.94.patch
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289516#comment-13289516 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
@Ted
Forgot to reply to your comment.
{code}
+ if (!masterRecovery) {
+ this.executorService = new ExecutorService(getServerName().toString());
+ this.serverManager = createServerManager(this, this);
+ }
{code}
In master recovery case server manager is 'not' created again.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Anoop Sam John (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287460#comment-13287460 ]
Anoop Sam John commented on HBASE-6046:
---------------------------------------
One immediate comment after seeing the patch
{code}
+ this.fileSystemManager = new MasterFileSystem(this, this, metrics, masterRecovery ? true
+ : false);
{code}
Can pass the boolean variable masterRecovery directly.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286305#comment-13286305 ]
ramkrishna.s.vasudevan commented on HBASE-6046:
-----------------------------------------------
We are working on a fix. It is almost HMaster.finishInitialization() but not fully. I think it is necessary to do that so that the split log of the RS that has gone down during this time can be done. Otherwise that goes unnoticed.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286169#comment-13286169 ]
stack commented on HBASE-6046:
------------------------------
What you think the fix is Ram?
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289800#comment-13289800 ]
Hudson commented on HBASE-6046:
-------------------------------
Integrated in HBase-0.94-security #35 (See [https://builds.apache.org/job/HBase-0.94-security/35/])
HBASE-6046 Master retry on ZK session expiry causes inconsistent region assignments. (Ashutosh) (Revision 1346461)
Result = SUCCESS
ramkrishna :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.92.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279051#comment-13279051 ]
stack commented on HBASE-6046:
------------------------------
Nice work Ram.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.94.1
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Attachment: HBASE_6046_0.92.patch
0.92 patch.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.92.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "Ashutosh Jindal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Jindal updated HBASE-6046:
-----------------------------------
Attachment: HBASE_6046_0.94_1.patch
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Attachment: HBASE_6046_0.94_3.patch
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.92 also.
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.92.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6046) Master retry on ZK session expiry
causes inconsistent region assignments.
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6046:
------------------------------------------
Fix Version/s: 0.94.1
0.96.0
0.92.2
> Master retry on ZK session expiry causes inconsistent region assignments.
> -------------------------------------------------------------------------
>
> Key: HBASE-6046
> URL: https://issues.apache.org/jira/browse/HBASE-6046
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Gopinathan A
> Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE_6046-trunk.patch, HBASE_6046_0.92.patch, HBASE_6046_0.94.patch, HBASE_6046_0.94_1.patch, HBASE_6046_0.94_2.patch, HBASE_6046_0.94_3.patch
>
>
> 1> ZK Session timeout in the hmaster leads to bulk assignment though all the RSs are online.
> 2> While doing bulk assignment, if the master again goes down & restart(or backup comes up) all the node created in the ZK will now be tried to reassign to the new RSs. This is leading to double assignment.
> we had 2800 regions, among this 1900 region got double assignment, taking the region count to 4700.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira