You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Stephen Chu (Created) (JIRA)" <ji...@apache.org> on 2012/03/06 09:50:58 UTC
[jira] [Created] (HDFS-3052) HA: hdfs balancer throws
StandbyException
HA: hdfs balancer throws StandbyException
-----------------------------------------
Key: HDFS-3052
URL: https://issues.apache.org/jira/browse/HDFS-3052
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Affects Versions: 0.24.0
Reporter: Stephen Chu
Attachments: balancer_styx01, balancer_styx02
The hdfs balancer tool throws a StandbyException.
Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
{noformat}
12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
. Exiting ...
Balancing took 650.0 milliseconds
{noformat}
After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
Failing over again results in the same StandbyException.
Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293531#comment-13293531 ]
Hudson commented on HDFS-3052:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1074 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/])
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package private. (Revision 1348998)
Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348998
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293604#comment-13293604 ]
Hudson commented on HDFS-3052:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1107 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/])
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package private. (Revision 1348998)
Result = FAILURE
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348998
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Stephen Chu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephen Chu updated HDFS-3052:
------------------------------
Attachment: balancer_styx02
balancer_styx01
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Stephen Chu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224009#comment-13224009 ]
Stephen Chu commented on HDFS-3052:
-----------------------------------
Yep, my snapshot was before HDFS-2979 was committed. Should I retest on the most current snapshot, or should we just close this out?
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293272#comment-13293272 ]
Hudson commented on HDFS-3052:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2342 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2342/])
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package private. (Revision 1348998)
Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348998
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224994#comment-13224994 ]
Uma Maheswara Rao G commented on HDFS-3052:
-------------------------------------------
Thanks Stephen for confirming this issue fixed in latest.
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223960#comment-13223960 ]
Aaron T. Myers commented on HDFS-3052:
--------------------------------------
Hey Stephen, I strongly suspect this issue has already been fixed by HDFS-2979. Is it possible that you encountered this error with a snapshot of HDFS from before that was committed?
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293113#comment-13293113 ]
Hudson commented on HDFS-3052:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2362 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2362/])
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package private. (Revision 1348998)
Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348998
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224029#comment-13224029 ]
Uma Maheswara Rao G commented on HDFS-3052:
-------------------------------------------
Hi Stephen, I agree with Aaron, this should have already fixed in latest.
If you have time, could you please rerun the test once with latest and close it out? If you still facing the issue, then w I will take a look at this.
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293277#comment-13293277 ]
Hudson commented on HDFS-3052:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2415 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2415/])
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package private. (Revision 1348998)
Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348998
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Stephen Chu (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephen Chu resolved HDFS-3052.
-------------------------------
Resolution: Duplicate
Just tested on a version with HDFS-2979 fix, and the problem has already been resolved. Resolving as duplicate.
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Stephen Chu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224034#comment-13224034 ]
Stephen Chu commented on HDFS-3052:
-----------------------------------
Sounds good. I will rerun the test on the latest snapshot and update afterwards.
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3052) HA: hdfs balancer throws
StandbyException
Posted by "Stephen Chu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224010#comment-13224010 ]
Stephen Chu commented on HDFS-3052:
-----------------------------------
Yep, my snapshot was before HDFS-2979 was committed. Should I retest on the most current snapshot, or should we just close this out?
[schu@styx01 ~]$ hadoop version
Hadoop 0.24.0-SNAPSHOT
Subversion git://styx01.sf.cloudera.com/home/schu/ha/hadoop-common/hadoop-common-project/hadoop-common -r 31e7d5866ef745a9a17bafe86c0b72b6044fca68
Compiled by schu on Tue Feb 28 14:22:42 PST 2012
>From source with checksum e38be048799c08983cf8e9a80afd8b41
> HA: hdfs balancer throws StandbyException
> -----------------------------------------
>
> Key: HDFS-3052
> URL: https://issues.apache.org/jira/browse/HDFS-3052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Stephen Chu
> Attachments: balancer_styx01, balancer_styx02
>
>
> The hdfs balancer tool throws a StandbyException.
> Originally, styx01 hosts the active NN and styx02 hosts the standby NN.
> After failing over from styx01 NN to styx02 NN, the _hdfs balancer_ command thows a StandbyException:
> {noformat}
> 12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/172.29.5.192:12020, nn2=styx02.sf.cloudera.com/172.29.5.193:12020}}
> 12/03/06 00:34:01 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
> . Exiting ...
> Balancing took 650.0 milliseconds
> {noformat}
> After failing back so that active is on styx01 and standby is on styx02, the _hdfs balancer_ command runs without exception.
> Failing over again results in the same StandbyException.
> Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira