You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by orahad bigdata <or...@gmail.com> on 2013/08/29 22:09:08 UTC
Hadoop HA error "JOURNAL is not supported in state standby"
Hi,
I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
the NameNode started in standby mode and not changing the state.
When I tried to do health check through "hdfs haadmin -checkhealth
<service id> " it's giving me below error.
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is:
"clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
I checked the logs at NN side.
2013-08-30 00:49:16,074 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.ipc.StandbyException: Operation category
JOURNAL is not supported in state standby
2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 8020, call
org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
from 192.168.126.31:48266: error:
org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
not supported in state standby
2013-08-30 00:49:32,391 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
log roll on remote NameNode clone2:8020
2013-08-30 00:49:32,403 WARN
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
trigger a roll of the active NN
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category JOURNAL is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
at org.apache.hadoop.ipc.Client.call(Client.java:1235)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy11.rollEditLog(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
Did I missed something?
Thanks
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Totally agreed with Jing's reply,
I faced the same issue previously, At that time I was doing cluster upgrade.
However I upgraded all the nodes but in one of my node hdfs bin pointing
to previous version, So I changed the PATH and it works fine for me.
Thanks
On Fri, Aug 30, 2013 at 2:10 AM, Jing Zhao <ji...@hortonworks.com> wrote:
> Looks like you have some incompatibility between your client side and
> the server side? Are you also running 2.0.5 in your client side?
>
> As Harsh mentioned, the NN side warning msg is not related to your
> InvalidProtocolBufferException. The warning msg indicates that both of
> your NN are in the Standby state.
>
> Thanks,
> -Jing
>
> On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com>
> wrote:
> > Thanks Harsh,
> >
> > I don't have auto failover configuration, but also I have tried to do
> > this manually but didn't get success.
> >
> > hdfs haadmin -transitionToActive node1
> >
> > Failed on local exception:
> > com.google.protobuf.InvalidProtocolBufferException: Message missing
> > required
> > fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX";
> > destination host is: "clone1":8020;
> >
> > So is there any alternative to resolve this issue?.
> >
> > Thanks
> >
> > On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> >> On the actual issue though: Do you also have auto-failover configured?
> >>
> >> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> >> wrote:
> >>> Hi,
> >>>
> >>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> >>> the NameNode started in standby mode and not changing the state.
> >>>
> >>> When I tried to do health check through "hdfs haadmin -checkhealth
> >>> <service id> " it's giving me below error.
> >>>
> >>> Failed on local exception:
> >>> com.google.protobuf.InvalidProtocolBufferException: Message missing
> >>> required fields: callId, status; Host Details : local host is:
> >>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
> >>>
> >>> I checked the logs at NN side.
> >>>
> >>> 2013-08-30 00:49:16,074 ERROR
> >>> org.apache.hadoop.security.UserGroupInformation:
> >>> PriviledgedActionException as:hadoop (auth:SIMPLE)
> >>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> >>> JOURNAL is not supported in state standby
> >>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> >>> handler 1 on 8020, call
> >>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> >>> from 192.168.126.31:48266: error:
> >>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> >>> not supported in state standby
> >>> 2013-08-30 00:49:32,391 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> >>> log roll on remote NameNode clone2:8020
> >>> 2013-08-30 00:49:32,403 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> >>> trigger a roll of the active NN
> >>>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> >>> Operation category JOURNAL is not supported in state standby
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:396)
> >>> at
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >>>
> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> >>> at $Proxy11.rollEditLog(Unknown Source)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> >>> at
> >>>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
> >>>
> >>> Did I missed something?
> >>>
> >>> Thanks
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Totally agreed with Jing's reply,
I faced the same issue previously, At that time I was doing cluster upgrade.
However I upgraded all the nodes but in one of my node hdfs bin pointing
to previous version, So I changed the PATH and it works fine for me.
Thanks
On Fri, Aug 30, 2013 at 2:10 AM, Jing Zhao <ji...@hortonworks.com> wrote:
> Looks like you have some incompatibility between your client side and
> the server side? Are you also running 2.0.5 in your client side?
>
> As Harsh mentioned, the NN side warning msg is not related to your
> InvalidProtocolBufferException. The warning msg indicates that both of
> your NN are in the Standby state.
>
> Thanks,
> -Jing
>
> On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com>
> wrote:
> > Thanks Harsh,
> >
> > I don't have auto failover configuration, but also I have tried to do
> > this manually but didn't get success.
> >
> > hdfs haadmin -transitionToActive node1
> >
> > Failed on local exception:
> > com.google.protobuf.InvalidProtocolBufferException: Message missing
> > required
> > fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX";
> > destination host is: "clone1":8020;
> >
> > So is there any alternative to resolve this issue?.
> >
> > Thanks
> >
> > On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> >> On the actual issue though: Do you also have auto-failover configured?
> >>
> >> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> >> wrote:
> >>> Hi,
> >>>
> >>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> >>> the NameNode started in standby mode and not changing the state.
> >>>
> >>> When I tried to do health check through "hdfs haadmin -checkhealth
> >>> <service id> " it's giving me below error.
> >>>
> >>> Failed on local exception:
> >>> com.google.protobuf.InvalidProtocolBufferException: Message missing
> >>> required fields: callId, status; Host Details : local host is:
> >>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
> >>>
> >>> I checked the logs at NN side.
> >>>
> >>> 2013-08-30 00:49:16,074 ERROR
> >>> org.apache.hadoop.security.UserGroupInformation:
> >>> PriviledgedActionException as:hadoop (auth:SIMPLE)
> >>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> >>> JOURNAL is not supported in state standby
> >>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> >>> handler 1 on 8020, call
> >>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> >>> from 192.168.126.31:48266: error:
> >>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> >>> not supported in state standby
> >>> 2013-08-30 00:49:32,391 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> >>> log roll on remote NameNode clone2:8020
> >>> 2013-08-30 00:49:32,403 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> >>> trigger a roll of the active NN
> >>>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> >>> Operation category JOURNAL is not supported in state standby
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:396)
> >>> at
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >>>
> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> >>> at $Proxy11.rollEditLog(Unknown Source)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> >>> at
> >>>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
> >>>
> >>> Did I missed something?
> >>>
> >>> Thanks
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Totally agreed with Jing's reply,
I faced the same issue previously, At that time I was doing cluster upgrade.
However I upgraded all the nodes but in one of my node hdfs bin pointing
to previous version, So I changed the PATH and it works fine for me.
Thanks
On Fri, Aug 30, 2013 at 2:10 AM, Jing Zhao <ji...@hortonworks.com> wrote:
> Looks like you have some incompatibility between your client side and
> the server side? Are you also running 2.0.5 in your client side?
>
> As Harsh mentioned, the NN side warning msg is not related to your
> InvalidProtocolBufferException. The warning msg indicates that both of
> your NN are in the Standby state.
>
> Thanks,
> -Jing
>
> On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com>
> wrote:
> > Thanks Harsh,
> >
> > I don't have auto failover configuration, but also I have tried to do
> > this manually but didn't get success.
> >
> > hdfs haadmin -transitionToActive node1
> >
> > Failed on local exception:
> > com.google.protobuf.InvalidProtocolBufferException: Message missing
> > required
> > fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX";
> > destination host is: "clone1":8020;
> >
> > So is there any alternative to resolve this issue?.
> >
> > Thanks
> >
> > On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> >> On the actual issue though: Do you also have auto-failover configured?
> >>
> >> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> >> wrote:
> >>> Hi,
> >>>
> >>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> >>> the NameNode started in standby mode and not changing the state.
> >>>
> >>> When I tried to do health check through "hdfs haadmin -checkhealth
> >>> <service id> " it's giving me below error.
> >>>
> >>> Failed on local exception:
> >>> com.google.protobuf.InvalidProtocolBufferException: Message missing
> >>> required fields: callId, status; Host Details : local host is:
> >>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
> >>>
> >>> I checked the logs at NN side.
> >>>
> >>> 2013-08-30 00:49:16,074 ERROR
> >>> org.apache.hadoop.security.UserGroupInformation:
> >>> PriviledgedActionException as:hadoop (auth:SIMPLE)
> >>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> >>> JOURNAL is not supported in state standby
> >>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> >>> handler 1 on 8020, call
> >>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> >>> from 192.168.126.31:48266: error:
> >>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> >>> not supported in state standby
> >>> 2013-08-30 00:49:32,391 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> >>> log roll on remote NameNode clone2:8020
> >>> 2013-08-30 00:49:32,403 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> >>> trigger a roll of the active NN
> >>>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> >>> Operation category JOURNAL is not supported in state standby
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:396)
> >>> at
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >>>
> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> >>> at $Proxy11.rollEditLog(Unknown Source)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> >>> at
> >>>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
> >>>
> >>> Did I missed something?
> >>>
> >>> Thanks
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Totally agreed with Jing's reply,
I faced the same issue previously, At that time I was doing cluster upgrade.
However I upgraded all the nodes but in one of my node hdfs bin pointing
to previous version, So I changed the PATH and it works fine for me.
Thanks
On Fri, Aug 30, 2013 at 2:10 AM, Jing Zhao <ji...@hortonworks.com> wrote:
> Looks like you have some incompatibility between your client side and
> the server side? Are you also running 2.0.5 in your client side?
>
> As Harsh mentioned, the NN side warning msg is not related to your
> InvalidProtocolBufferException. The warning msg indicates that both of
> your NN are in the Standby state.
>
> Thanks,
> -Jing
>
> On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com>
> wrote:
> > Thanks Harsh,
> >
> > I don't have auto failover configuration, but also I have tried to do
> > this manually but didn't get success.
> >
> > hdfs haadmin -transitionToActive node1
> >
> > Failed on local exception:
> > com.google.protobuf.InvalidProtocolBufferException: Message missing
> > required
> > fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX";
> > destination host is: "clone1":8020;
> >
> > So is there any alternative to resolve this issue?.
> >
> > Thanks
> >
> > On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> >> On the actual issue though: Do you also have auto-failover configured?
> >>
> >> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> >> wrote:
> >>> Hi,
> >>>
> >>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> >>> the NameNode started in standby mode and not changing the state.
> >>>
> >>> When I tried to do health check through "hdfs haadmin -checkhealth
> >>> <service id> " it's giving me below error.
> >>>
> >>> Failed on local exception:
> >>> com.google.protobuf.InvalidProtocolBufferException: Message missing
> >>> required fields: callId, status; Host Details : local host is:
> >>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
> >>>
> >>> I checked the logs at NN side.
> >>>
> >>> 2013-08-30 00:49:16,074 ERROR
> >>> org.apache.hadoop.security.UserGroupInformation:
> >>> PriviledgedActionException as:hadoop (auth:SIMPLE)
> >>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> >>> JOURNAL is not supported in state standby
> >>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> >>> handler 1 on 8020, call
> >>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> >>> from 192.168.126.31:48266: error:
> >>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> >>> not supported in state standby
> >>> 2013-08-30 00:49:32,391 INFO
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> >>> log roll on remote NameNode clone2:8020
> >>> 2013-08-30 00:49:32,403 WARN
> >>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> >>> trigger a roll of the active NN
> >>>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> >>> Operation category JOURNAL is not supported in state standby
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:396)
> >>> at
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >>>
> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> >>> at
> >>>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> >>> at $Proxy11.rollEditLog(Unknown Source)
> >>> at
> >>>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> >>> at
> >>>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> >>> at
> >>>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
> >>>
> >>> Did I missed something?
> >>>
> >>> Thanks
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jing Zhao <ji...@hortonworks.com>.
Looks like you have some incompatibility between your client side and
the server side? Are you also running 2.0.5 in your client side?
As Harsh mentioned, the NN side warning msg is not related to your
InvalidProtocolBufferException. The warning msg indicates that both of
your NN are in the Standby state.
Thanks,
-Jing
On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com> wrote:
> Thanks Harsh,
>
> I don't have auto failover configuration, but also I have tried to do
> this manually but didn't get success.
>
> hdfs haadmin -transitionToActive node1
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required
> fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
> destination host is: "clone1":8020;
>
> So is there any alternative to resolve this issue?.
>
> Thanks
>
> On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
>> On the actual issue though: Do you also have auto-failover configured?
>>
>> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
>> wrote:
>>> Hi,
>>>
>>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>>> the NameNode started in standby mode and not changing the state.
>>>
>>> When I tried to do health check through "hdfs haadmin -checkhealth
>>> <service id> " it's giving me below error.
>>>
>>> Failed on local exception:
>>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>>> required fields: callId, status; Host Details : local host is:
>>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>>
>>> I checked the logs at NN side.
>>>
>>> 2013-08-30 00:49:16,074 ERROR
>>> org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>>> JOURNAL is not supported in state standby
>>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 8020, call
>>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>>> from 192.168.126.31:48266: error:
>>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>>> not supported in state standby
>>> 2013-08-30 00:49:32,391 INFO
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>>> log roll on remote NameNode clone2:8020
>>> 2013-08-30 00:49:32,403 WARN
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>>> trigger a roll of the active NN
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category JOURNAL is not supported in state standby
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>>> at $Proxy11.rollEditLog(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>>> at
>>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>>
>>> Did I missed something?
>>>
>>> Thanks
>>
>>
>>
>> --
>> Harsh J
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jing Zhao <ji...@hortonworks.com>.
Looks like you have some incompatibility between your client side and
the server side? Are you also running 2.0.5 in your client side?
As Harsh mentioned, the NN side warning msg is not related to your
InvalidProtocolBufferException. The warning msg indicates that both of
your NN are in the Standby state.
Thanks,
-Jing
On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com> wrote:
> Thanks Harsh,
>
> I don't have auto failover configuration, but also I have tried to do
> this manually but didn't get success.
>
> hdfs haadmin -transitionToActive node1
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required
> fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
> destination host is: "clone1":8020;
>
> So is there any alternative to resolve this issue?.
>
> Thanks
>
> On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
>> On the actual issue though: Do you also have auto-failover configured?
>>
>> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
>> wrote:
>>> Hi,
>>>
>>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>>> the NameNode started in standby mode and not changing the state.
>>>
>>> When I tried to do health check through "hdfs haadmin -checkhealth
>>> <service id> " it's giving me below error.
>>>
>>> Failed on local exception:
>>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>>> required fields: callId, status; Host Details : local host is:
>>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>>
>>> I checked the logs at NN side.
>>>
>>> 2013-08-30 00:49:16,074 ERROR
>>> org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>>> JOURNAL is not supported in state standby
>>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 8020, call
>>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>>> from 192.168.126.31:48266: error:
>>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>>> not supported in state standby
>>> 2013-08-30 00:49:32,391 INFO
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>>> log roll on remote NameNode clone2:8020
>>> 2013-08-30 00:49:32,403 WARN
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>>> trigger a roll of the active NN
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category JOURNAL is not supported in state standby
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>>> at $Proxy11.rollEditLog(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>>> at
>>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>>
>>> Did I missed something?
>>>
>>> Thanks
>>
>>
>>
>> --
>> Harsh J
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jing Zhao <ji...@hortonworks.com>.
Looks like you have some incompatibility between your client side and
the server side? Are you also running 2.0.5 in your client side?
As Harsh mentioned, the NN side warning msg is not related to your
InvalidProtocolBufferException. The warning msg indicates that both of
your NN are in the Standby state.
Thanks,
-Jing
On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com> wrote:
> Thanks Harsh,
>
> I don't have auto failover configuration, but also I have tried to do
> this manually but didn't get success.
>
> hdfs haadmin -transitionToActive node1
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required
> fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
> destination host is: "clone1":8020;
>
> So is there any alternative to resolve this issue?.
>
> Thanks
>
> On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
>> On the actual issue though: Do you also have auto-failover configured?
>>
>> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
>> wrote:
>>> Hi,
>>>
>>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>>> the NameNode started in standby mode and not changing the state.
>>>
>>> When I tried to do health check through "hdfs haadmin -checkhealth
>>> <service id> " it's giving me below error.
>>>
>>> Failed on local exception:
>>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>>> required fields: callId, status; Host Details : local host is:
>>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>>
>>> I checked the logs at NN side.
>>>
>>> 2013-08-30 00:49:16,074 ERROR
>>> org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>>> JOURNAL is not supported in state standby
>>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 8020, call
>>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>>> from 192.168.126.31:48266: error:
>>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>>> not supported in state standby
>>> 2013-08-30 00:49:32,391 INFO
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>>> log roll on remote NameNode clone2:8020
>>> 2013-08-30 00:49:32,403 WARN
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>>> trigger a roll of the active NN
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category JOURNAL is not supported in state standby
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>>> at $Proxy11.rollEditLog(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>>> at
>>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>>
>>> Did I missed something?
>>>
>>> Thanks
>>
>>
>>
>> --
>> Harsh J
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Jing Zhao <ji...@hortonworks.com>.
Looks like you have some incompatibility between your client side and
the server side? Are you also running 2.0.5 in your client side?
As Harsh mentioned, the NN side warning msg is not related to your
InvalidProtocolBufferException. The warning msg indicates that both of
your NN are in the Standby state.
Thanks,
-Jing
On Thu, Aug 29, 2013 at 1:36 PM, orahad bigdata <or...@gmail.com> wrote:
> Thanks Harsh,
>
> I don't have auto failover configuration, but also I have tried to do
> this manually but didn't get success.
>
> hdfs haadmin -transitionToActive node1
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required
> fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
> destination host is: "clone1":8020;
>
> So is there any alternative to resolve this issue?.
>
> Thanks
>
> On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
>> On the actual issue though: Do you also have auto-failover configured?
>>
>> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
>> wrote:
>>> Hi,
>>>
>>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>>> the NameNode started in standby mode and not changing the state.
>>>
>>> When I tried to do health check through "hdfs haadmin -checkhealth
>>> <service id> " it's giving me below error.
>>>
>>> Failed on local exception:
>>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>>> required fields: callId, status; Host Details : local host is:
>>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>>
>>> I checked the logs at NN side.
>>>
>>> 2013-08-30 00:49:16,074 ERROR
>>> org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>>> JOURNAL is not supported in state standby
>>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 8020, call
>>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>>> from 192.168.126.31:48266: error:
>>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>>> not supported in state standby
>>> 2013-08-30 00:49:32,391 INFO
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>>> log roll on remote NameNode clone2:8020
>>> 2013-08-30 00:49:32,403 WARN
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>>> trigger a roll of the active NN
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>>> Operation category JOURNAL is not supported in state standby
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>>> at $Proxy11.rollEditLog(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>>> at
>>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>>
>>> Did I missed something?
>>>
>>> Thanks
>>
>>
>>
>> --
>> Harsh J
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Thanks Harsh,
I don't have auto failover configuration, but also I have tried to do
this manually but didn't get success.
hdfs haadmin -transitionToActive node1
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required
fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
destination host is: "clone1":8020;
So is there any alternative to resolve this issue?.
Thanks
On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> On the actual issue though: Do you also have auto-failover configured?
>
> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> wrote:
>> Hi,
>>
>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>> the NameNode started in standby mode and not changing the state.
>>
>> When I tried to do health check through "hdfs haadmin -checkhealth
>> <service id> " it's giving me below error.
>>
>> Failed on local exception:
>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>> required fields: callId, status; Host Details : local host is:
>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>
>> I checked the logs at NN side.
>>
>> 2013-08-30 00:49:16,074 ERROR
>> org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>> JOURNAL is not supported in state standby
>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 8020, call
>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>> from 192.168.126.31:48266: error:
>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>> not supported in state standby
>> 2013-08-30 00:49:32,391 INFO
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>> log roll on remote NameNode clone2:8020
>> 2013-08-30 00:49:32,403 WARN
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>> trigger a roll of the active NN
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>> Operation category JOURNAL is not supported in state standby
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>> at $Proxy11.rollEditLog(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>> at
>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>
>> Did I missed something?
>>
>> Thanks
>
>
>
> --
> Harsh J
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Thanks Harsh,
I don't have auto failover configuration, but also I have tried to do
this manually but didn't get success.
hdfs haadmin -transitionToActive node1
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required
fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
destination host is: "clone1":8020;
So is there any alternative to resolve this issue?.
Thanks
On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> On the actual issue though: Do you also have auto-failover configured?
>
> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> wrote:
>> Hi,
>>
>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>> the NameNode started in standby mode and not changing the state.
>>
>> When I tried to do health check through "hdfs haadmin -checkhealth
>> <service id> " it's giving me below error.
>>
>> Failed on local exception:
>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>> required fields: callId, status; Host Details : local host is:
>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>
>> I checked the logs at NN side.
>>
>> 2013-08-30 00:49:16,074 ERROR
>> org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>> JOURNAL is not supported in state standby
>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 8020, call
>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>> from 192.168.126.31:48266: error:
>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>> not supported in state standby
>> 2013-08-30 00:49:32,391 INFO
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>> log roll on remote NameNode clone2:8020
>> 2013-08-30 00:49:32,403 WARN
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>> trigger a roll of the active NN
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>> Operation category JOURNAL is not supported in state standby
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>> at $Proxy11.rollEditLog(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>> at
>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>
>> Did I missed something?
>>
>> Thanks
>
>
>
> --
> Harsh J
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Thanks Harsh,
I don't have auto failover configuration, but also I have tried to do
this manually but didn't get success.
hdfs haadmin -transitionToActive node1
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required
fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
destination host is: "clone1":8020;
So is there any alternative to resolve this issue?.
Thanks
On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> On the actual issue though: Do you also have auto-failover configured?
>
> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> wrote:
>> Hi,
>>
>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>> the NameNode started in standby mode and not changing the state.
>>
>> When I tried to do health check through "hdfs haadmin -checkhealth
>> <service id> " it's giving me below error.
>>
>> Failed on local exception:
>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>> required fields: callId, status; Host Details : local host is:
>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>
>> I checked the logs at NN side.
>>
>> 2013-08-30 00:49:16,074 ERROR
>> org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>> JOURNAL is not supported in state standby
>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 8020, call
>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>> from 192.168.126.31:48266: error:
>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>> not supported in state standby
>> 2013-08-30 00:49:32,391 INFO
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>> log roll on remote NameNode clone2:8020
>> 2013-08-30 00:49:32,403 WARN
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>> trigger a roll of the active NN
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>> Operation category JOURNAL is not supported in state standby
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>> at $Proxy11.rollEditLog(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>> at
>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>
>> Did I missed something?
>>
>> Thanks
>
>
>
> --
> Harsh J
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Thanks Harsh,
I don't have auto failover configuration, but also I have tried to do
this manually but didn't get success.
hdfs haadmin -transitionToActive node1
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required
fields: callId, status; Host Details : local host is: "clone2/XX.XX.XX.XX";
destination host is: "clone1":8020;
So is there any alternative to resolve this issue?.
Thanks
On 8/30/13, Harsh J <ha...@cloudera.com> wrote:
> On the actual issue though: Do you also have auto-failover configured?
>
> On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com>
> wrote:
>> Hi,
>>
>> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
>> the NameNode started in standby mode and not changing the state.
>>
>> When I tried to do health check through "hdfs haadmin -checkhealth
>> <service id> " it's giving me below error.
>>
>> Failed on local exception:
>> com.google.protobuf.InvalidProtocolBufferException: Message missing
>> required fields: callId, status; Host Details : local host is:
>> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>>
>> I checked the logs at NN side.
>>
>> 2013-08-30 00:49:16,074 ERROR
>> org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:hadoop (auth:SIMPLE)
>> cause:org.apache.hadoop.ipc.StandbyException: Operation category
>> JOURNAL is not supported in state standby
>> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 8020, call
>> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
>> from 192.168.126.31:48266: error:
>> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
>> not supported in state standby
>> 2013-08-30 00:49:32,391 INFO
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
>> log roll on remote NameNode clone2:8020
>> 2013-08-30 00:49:32,403 WARN
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
>> trigger a roll of the active NN
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
>> Operation category JOURNAL is not supported in state standby
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>> at $Proxy11.rollEditLog(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>> at
>> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>> at
>> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>>
>> Did I missed something?
>>
>> Thanks
>
>
>
> --
> Harsh J
>
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
On the actual issue though: Do you also have auto-failover configured?
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
Re: 答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Hi,
Many thanks to everyone.
Now issue got resolve after changing client version.
Regards
On Fri, Aug 30, 2013 at 1:12 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Did you start up your ZKFC service on both of your name nodes ?
>
> Thanks,
> Francis.Hu
>
> -----邮件原件-----
> 发件人: orahad bigdata [mailto:oraclehad@gmail.com]
> 发送时间: Friday, August 30, 2013 4:09
> 收件人: user
> 主题: Hadoop HA error "JOURNAL is not supported in state standby"
>
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
> ):
> Operation category JOURNAL is not supported in state standby
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
> yState.java:87)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
> tion(NameNode.java:1411)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
> tem.java:859)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
> .java:4445)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
> eRpcServer.java:766)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
> lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
> lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
> bufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> va:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
> a:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
> amenodeProtocolTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
> (EditLogTailer.java:268)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
> iler.java:61)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> doWork(EditLogTailer.java:310)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> access$200(EditLogTailer.java:279)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
> 1.run(EditLogTailer.java:296)
> at
>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
> va:456)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
>
>
Re: 答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Hi,
Many thanks to everyone.
Now issue got resolve after changing client version.
Regards
On Fri, Aug 30, 2013 at 1:12 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Did you start up your ZKFC service on both of your name nodes ?
>
> Thanks,
> Francis.Hu
>
> -----邮件原件-----
> 发件人: orahad bigdata [mailto:oraclehad@gmail.com]
> 发送时间: Friday, August 30, 2013 4:09
> 收件人: user
> 主题: Hadoop HA error "JOURNAL is not supported in state standby"
>
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
> ):
> Operation category JOURNAL is not supported in state standby
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
> yState.java:87)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
> tion(NameNode.java:1411)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
> tem.java:859)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
> .java:4445)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
> eRpcServer.java:766)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
> lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
> lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
> bufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> va:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
> a:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
> amenodeProtocolTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
> (EditLogTailer.java:268)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
> iler.java:61)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> doWork(EditLogTailer.java:310)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> access$200(EditLogTailer.java:279)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
> 1.run(EditLogTailer.java:296)
> at
>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
> va:456)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
>
>
Re: 答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Hi,
Many thanks to everyone.
Now issue got resolve after changing client version.
Regards
On Fri, Aug 30, 2013 at 1:12 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Did you start up your ZKFC service on both of your name nodes ?
>
> Thanks,
> Francis.Hu
>
> -----邮件原件-----
> 发件人: orahad bigdata [mailto:oraclehad@gmail.com]
> 发送时间: Friday, August 30, 2013 4:09
> 收件人: user
> 主题: Hadoop HA error "JOURNAL is not supported in state standby"
>
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
> ):
> Operation category JOURNAL is not supported in state standby
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
> yState.java:87)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
> tion(NameNode.java:1411)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
> tem.java:859)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
> .java:4445)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
> eRpcServer.java:766)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
> lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
> lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
> bufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> va:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
> a:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
> amenodeProtocolTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
> (EditLogTailer.java:268)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
> iler.java:61)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> doWork(EditLogTailer.java:310)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> access$200(EditLogTailer.java:279)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
> 1.run(EditLogTailer.java:296)
> at
>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
> va:456)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
>
>
Re: 答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by orahad bigdata <or...@gmail.com>.
Hi,
Many thanks to everyone.
Now issue got resolve after changing client version.
Regards
On Fri, Aug 30, 2013 at 1:12 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Did you start up your ZKFC service on both of your name nodes ?
>
> Thanks,
> Francis.Hu
>
> -----邮件原件-----
> 发件人: orahad bigdata [mailto:oraclehad@gmail.com]
> 发送时间: Friday, August 30, 2013 4:09
> 收件人: user
> 主题: Hadoop HA error "JOURNAL is not supported in state standby"
>
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
> ):
> Operation category JOURNAL is not supported in state standby
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
> yState.java:87)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
> tion(NameNode.java:1411)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
> tem.java:859)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
> .java:4445)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
> eRpcServer.java:766)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
> lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
> lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
> bufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> va:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
> a:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
> amenodeProtocolTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
> (EditLogTailer.java:268)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
> iler.java:61)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> doWork(EditLogTailer.java:310)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> access$200(EditLogTailer.java:279)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
> 1.run(EditLogTailer.java:296)
> at
>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
> va:456)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
> run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
>
>
答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Did you start up your ZKFC service on both of your name nodes ?
Thanks,
Francis.Hu
-----邮件原件-----
发件人: orahad bigdata [mailto:oraclehad@gmail.com]
发送时间: Friday, August 30, 2013 4:09
收件人: user
主题: Hadoop HA error "JOURNAL is not supported in state standby"
Hi,
I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
the NameNode started in standby mode and not changing the state.
When I tried to do health check through "hdfs haadmin -checkhealth
<service id> " it's giving me below error.
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is:
"clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
I checked the logs at NN side.
2013-08-30 00:49:16,074 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.ipc.StandbyException: Operation category
JOURNAL is not supported in state standby
2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 8020, call
org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
from 192.168.126.31:48266: error:
org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
not supported in state standby
2013-08-30 00:49:32,391 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
log roll on remote NameNode clone2:8020
2013-08-30 00:49:32,403 WARN
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
trigger a roll of the active NN
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
):
Operation category JOURNAL is not supported in state standby
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
yState.java:87)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
tion(NameNode.java:1411)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
tem.java:859)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
.java:4445)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
eRpcServer.java:766)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
bufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1478)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
at org.apache.hadoop.ipc.Client.call(Client.java:1235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
a:202)
at $Proxy11.rollEditLog(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
amenodeProtocolTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
(EditLogTailer.java:268)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
iler.java:61)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
doWork(EditLogTailer.java:310)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
va:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
run(EditLogTailer.java:292)
Did I missed something?
Thanks
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
On the actual issue though: Do you also have auto-failover configured?
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Did you start up your ZKFC service on both of your name nodes ?
Thanks,
Francis.Hu
-----邮件原件-----
发件人: orahad bigdata [mailto:oraclehad@gmail.com]
发送时间: Friday, August 30, 2013 4:09
收件人: user
主题: Hadoop HA error "JOURNAL is not supported in state standby"
Hi,
I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
the NameNode started in standby mode and not changing the state.
When I tried to do health check through "hdfs haadmin -checkhealth
<service id> " it's giving me below error.
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is:
"clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
I checked the logs at NN side.
2013-08-30 00:49:16,074 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.ipc.StandbyException: Operation category
JOURNAL is not supported in state standby
2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 8020, call
org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
from 192.168.126.31:48266: error:
org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
not supported in state standby
2013-08-30 00:49:32,391 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
log roll on remote NameNode clone2:8020
2013-08-30 00:49:32,403 WARN
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
trigger a roll of the active NN
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
):
Operation category JOURNAL is not supported in state standby
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
yState.java:87)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
tion(NameNode.java:1411)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
tem.java:859)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
.java:4445)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
eRpcServer.java:766)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
bufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1478)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
at org.apache.hadoop.ipc.Client.call(Client.java:1235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
a:202)
at $Proxy11.rollEditLog(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
amenodeProtocolTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
(EditLogTailer.java:268)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
iler.java:61)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
doWork(EditLogTailer.java:310)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
va:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
run(EditLogTailer.java:292)
Did I missed something?
Thanks
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
Hello,
This WARN log is unnecessary as such a state is expected and this has
been reported at https://issues.apache.org/jira/browse/HDFS-3918.
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
On the actual issue though: Do you also have auto-failover configured?
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
Hello,
This WARN log is unnecessary as such a state is expected and this has
been reported at https://issues.apache.org/jira/browse/HDFS-3918.
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Did you start up your ZKFC service on both of your name nodes ?
Thanks,
Francis.Hu
-----邮件原件-----
发件人: orahad bigdata [mailto:oraclehad@gmail.com]
发送时间: Friday, August 30, 2013 4:09
收件人: user
主题: Hadoop HA error "JOURNAL is not supported in state standby"
Hi,
I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
the NameNode started in standby mode and not changing the state.
When I tried to do health check through "hdfs haadmin -checkhealth
<service id> " it's giving me below error.
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is:
"clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
I checked the logs at NN side.
2013-08-30 00:49:16,074 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.ipc.StandbyException: Operation category
JOURNAL is not supported in state standby
2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 8020, call
org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
from 192.168.126.31:48266: error:
org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
not supported in state standby
2013-08-30 00:49:32,391 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
log roll on remote NameNode clone2:8020
2013-08-30 00:49:32,403 WARN
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
trigger a roll of the active NN
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
):
Operation category JOURNAL is not supported in state standby
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
yState.java:87)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
tion(NameNode.java:1411)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
tem.java:859)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
.java:4445)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
eRpcServer.java:766)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
bufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1478)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
at org.apache.hadoop.ipc.Client.call(Client.java:1235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
a:202)
at $Proxy11.rollEditLog(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
amenodeProtocolTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
(EditLogTailer.java:268)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
iler.java:61)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
doWork(EditLogTailer.java:310)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
va:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
run(EditLogTailer.java:292)
Did I missed something?
Thanks
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
Hello,
This WARN log is unnecessary as such a state is expected and this has
been reported at https://issues.apache.org/jira/browse/HDFS-3918.
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
On the actual issue though: Do you also have auto-failover configured?
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
Re: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by Harsh J <ha...@cloudera.com>.
Hello,
This WARN log is unnecessary as such a state is expected and this has
been reported at https://issues.apache.org/jira/browse/HDFS-3918.
On Fri, Aug 30, 2013 at 1:39 AM, orahad bigdata <or...@gmail.com> wrote:
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category JOURNAL is not supported in state standby
> at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1411)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:859)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4445)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:766)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:268)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:310)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
--
Harsh J
答复: Hadoop HA error "JOURNAL is not supported in state standby"
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Did you start up your ZKFC service on both of your name nodes ?
Thanks,
Francis.Hu
-----邮件原件-----
发件人: orahad bigdata [mailto:oraclehad@gmail.com]
发送时间: Friday, August 30, 2013 4:09
收件人: user
主题: Hadoop HA error "JOURNAL is not supported in state standby"
Hi,
I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
the NameNode started in standby mode and not changing the state.
When I tried to do health check through "hdfs haadmin -checkhealth
<service id> " it's giving me below error.
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is:
"clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
I checked the logs at NN side.
2013-08-30 00:49:16,074 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.ipc.StandbyException: Operation category
JOURNAL is not supported in state standby
2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 8020, call
org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
from 192.168.126.31:48266: error:
org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
not supported in state standby
2013-08-30 00:49:32,391 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
log roll on remote NameNode clone2:8020
2013-08-30 00:49:32,403 WARN
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
trigger a roll of the active NN
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException
):
Operation category JOURNAL is not supported in state standby
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb
yState.java:87)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera
tion(NameNode.java:1411)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys
tem.java:859)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem
.java:4445)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod
eRpcServer.java:766)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol
lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco
lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto
bufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1478)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
at org.apache.hadoop.ipc.Client.call(Client.java:1235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav
a:202)
at $Proxy11.rollEditLog(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N
amenodeProtocolTranslatorPB.java:139)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll
(EditLogTailer.java:268)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa
iler.java:61)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
doWork(EditLogTailer.java:310)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$
1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja
va:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.
run(EditLogTailer.java:292)
Did I missed something?
Thanks