You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Manoj Samel <ma...@gmail.com> on 2014/03/26 00:55:32 UTC
Error "Failed to specify server's Kerberos principal name" with
kerberos enabled
Hi,
Hadoop version is CDH5 Beta1
Name node and Resource managers have been configured in HA mode.
After kerberos is enabled, the resource manager log shows following error
2014-03-25 22:21:06,854 WARN org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
java.lang.IllegalArgumentException: Failed to specify server's Kerberos
principal name
2014-03-25 22:21:06,855 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
The keytab files for hdfs, mapred and yarn are (seemingly) correct and show
correct list of principals using klist.
Any thoughts,
Thanks,
Re: Error "Failed to specify server's Kerberos principal name" with
kerberos enabled
Posted by Manoj Samel <ma...@gmail.com>.
Looking further @ the resource manager log,
the resource manager starts and goes in standby
Then it starts IPC Server and IPC Client ( presumably to talk to Zookeeper
to elect a active v/s standby Resource Manager).
The IPC Server seem to start and then when IPC client tries to connect to
server, it seem to detect that there is no kerberos principal defined (for
server?).
Not sure where is such kerberos principal is defined for IPC client /
server that seem to be used for ZooKeeper ?
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for user
yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL using keytab file
/etc/hadoop/conf/yarn.keytab
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioning to standby
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioned to standby
2014-03-26 01:13:31,928 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 23142
2014-03-26 01:13:31,996 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:31,997 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 23142: starting
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.0.0-beta-1--1, built on 10/28/2013
00:00 GMT
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=xxxxxx
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_51
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.7.0_51/jre
Zookeeper client start command .....
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop/lib/native
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-358.6.2.el6.x86_64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/var/lib/hadoop-yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/usr/lib/hadoop-yarn
2014-03-26 01:13:32,021 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@26bc6487
2014-03-26 01:13:32,038 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx FQDN/xxxx. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,050 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,060 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x144fbf1d0350000, negotiated timeout = 5000
2014-03-26 01:13:32,076 WARN org.apache.hadoop.ha.ActiveStandbyElector:
Ignoring stale result from old client with sessionId 0x144fbf1d0350000
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Session:
0x144fbf1d0350000 closed
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@74c35f27
2014-03-26 01:13:32,081 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2014-03-26 01:13:32,082 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx/xxxx:2181. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,083 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,088 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x344fbf1fe860001, negotiated timeout = 5000
2014-03-26 01:13:32,095 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Session connected.
2014-03-26 01:13:32,096 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8035
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8035: starting
2014-03-26 01:13:32,241 WARN org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
java.lang.IllegalArgumentException: Failed to specify server's Kerberos
principal name
2014-03-26 01:13:32,242 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:yarn/xxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
On Tue, Mar 25, 2014 at 4:55 PM, Manoj Samel <ma...@gmail.com>wrote:
> Hi,
>
> Hadoop version is CDH5 Beta1
>
> Name node and Resource managers have been configured in HA mode.
>
> After kerberos is enabled, the resource manager log shows following error
>
> 2014-03-25 22:21:06,854 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> java.lang.IllegalArgumentException: Failed to specify server's Kerberos
> principal name
> 2014-03-25 22:21:06,855 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
> cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
>
> The keytab files for hdfs, mapred and yarn are (seemingly) correct and
> show correct list of principals using klist.
>
> Any thoughts,
>
> Thanks,
>
Re: Error "Failed to specify server's Kerberos principal name" with
kerberos enabled
Posted by Manoj Samel <ma...@gmail.com>.
Looking further @ the resource manager log,
the resource manager starts and goes in standby
Then it starts IPC Server and IPC Client ( presumably to talk to Zookeeper
to elect a active v/s standby Resource Manager).
The IPC Server seem to start and then when IPC client tries to connect to
server, it seem to detect that there is no kerberos principal defined (for
server?).
Not sure where is such kerberos principal is defined for IPC client /
server that seem to be used for ZooKeeper ?
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for user
yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL using keytab file
/etc/hadoop/conf/yarn.keytab
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioning to standby
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioned to standby
2014-03-26 01:13:31,928 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 23142
2014-03-26 01:13:31,996 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:31,997 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 23142: starting
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.0.0-beta-1--1, built on 10/28/2013
00:00 GMT
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=xxxxxx
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_51
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.7.0_51/jre
Zookeeper client start command .....
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop/lib/native
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-358.6.2.el6.x86_64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/var/lib/hadoop-yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/usr/lib/hadoop-yarn
2014-03-26 01:13:32,021 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@26bc6487
2014-03-26 01:13:32,038 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx FQDN/xxxx. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,050 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,060 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x144fbf1d0350000, negotiated timeout = 5000
2014-03-26 01:13:32,076 WARN org.apache.hadoop.ha.ActiveStandbyElector:
Ignoring stale result from old client with sessionId 0x144fbf1d0350000
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Session:
0x144fbf1d0350000 closed
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@74c35f27
2014-03-26 01:13:32,081 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2014-03-26 01:13:32,082 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx/xxxx:2181. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,083 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,088 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x344fbf1fe860001, negotiated timeout = 5000
2014-03-26 01:13:32,095 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Session connected.
2014-03-26 01:13:32,096 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8035
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8035: starting
2014-03-26 01:13:32,241 WARN org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
java.lang.IllegalArgumentException: Failed to specify server's Kerberos
principal name
2014-03-26 01:13:32,242 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:yarn/xxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
On Tue, Mar 25, 2014 at 4:55 PM, Manoj Samel <ma...@gmail.com>wrote:
> Hi,
>
> Hadoop version is CDH5 Beta1
>
> Name node and Resource managers have been configured in HA mode.
>
> After kerberos is enabled, the resource manager log shows following error
>
> 2014-03-25 22:21:06,854 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> java.lang.IllegalArgumentException: Failed to specify server's Kerberos
> principal name
> 2014-03-25 22:21:06,855 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
> cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
>
> The keytab files for hdfs, mapred and yarn are (seemingly) correct and
> show correct list of principals using klist.
>
> Any thoughts,
>
> Thanks,
>
Re: Error "Failed to specify server's Kerberos principal name" with
kerberos enabled
Posted by Manoj Samel <ma...@gmail.com>.
Looking further @ the resource manager log,
the resource manager starts and goes in standby
Then it starts IPC Server and IPC Client ( presumably to talk to Zookeeper
to elect a active v/s standby Resource Manager).
The IPC Server seem to start and then when IPC client tries to connect to
server, it seem to detect that there is no kerberos principal defined (for
server?).
Not sure where is such kerberos principal is defined for IPC client /
server that seem to be used for ZooKeeper ?
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for user
yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL using keytab file
/etc/hadoop/conf/yarn.keytab
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioning to standby
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioned to standby
2014-03-26 01:13:31,928 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 23142
2014-03-26 01:13:31,996 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:31,997 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 23142: starting
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.0.0-beta-1--1, built on 10/28/2013
00:00 GMT
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=xxxxxx
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_51
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.7.0_51/jre
Zookeeper client start command .....
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop/lib/native
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-358.6.2.el6.x86_64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/var/lib/hadoop-yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/usr/lib/hadoop-yarn
2014-03-26 01:13:32,021 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@26bc6487
2014-03-26 01:13:32,038 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx FQDN/xxxx. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,050 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,060 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x144fbf1d0350000, negotiated timeout = 5000
2014-03-26 01:13:32,076 WARN org.apache.hadoop.ha.ActiveStandbyElector:
Ignoring stale result from old client with sessionId 0x144fbf1d0350000
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Session:
0x144fbf1d0350000 closed
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@74c35f27
2014-03-26 01:13:32,081 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2014-03-26 01:13:32,082 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx/xxxx:2181. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,083 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,088 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x344fbf1fe860001, negotiated timeout = 5000
2014-03-26 01:13:32,095 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Session connected.
2014-03-26 01:13:32,096 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8035
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8035: starting
2014-03-26 01:13:32,241 WARN org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
java.lang.IllegalArgumentException: Failed to specify server's Kerberos
principal name
2014-03-26 01:13:32,242 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:yarn/xxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
On Tue, Mar 25, 2014 at 4:55 PM, Manoj Samel <ma...@gmail.com>wrote:
> Hi,
>
> Hadoop version is CDH5 Beta1
>
> Name node and Resource managers have been configured in HA mode.
>
> After kerberos is enabled, the resource manager log shows following error
>
> 2014-03-25 22:21:06,854 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> java.lang.IllegalArgumentException: Failed to specify server's Kerberos
> principal name
> 2014-03-25 22:21:06,855 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
> cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
>
> The keytab files for hdfs, mapred and yarn are (seemingly) correct and
> show correct list of principals using klist.
>
> Any thoughts,
>
> Thanks,
>
Re: Error "Failed to specify server's Kerberos principal name" with
kerberos enabled
Posted by Manoj Samel <ma...@gmail.com>.
Looking further @ the resource manager log,
the resource manager starts and goes in standby
Then it starts IPC Server and IPC Client ( presumably to talk to Zookeeper
to elect a active v/s standby Resource Manager).
The IPC Server seem to start and then when IPC client tries to connect to
server, it seem to detect that there is no kerberos principal defined (for
server?).
Not sure where is such kerberos principal is defined for IPC client /
server that seem to be used for ZooKeeper ?
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for user
yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL using keytab file
/etc/hadoop/conf/yarn.keytab
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioning to standby
2014-03-26 01:13:31,851 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService:
Transitioned to standby
2014-03-26 01:13:31,928 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 23142
2014-03-26 01:13:31,996 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:31,997 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 23142: starting
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.0.0-beta-1--1, built on 10/28/2013
00:00 GMT
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=xxxxxx
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_51
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.7.0_51/jre
Zookeeper client start command .....
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop/lib/native
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-358.6.2.el6.x86_64
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/var/lib/hadoop-yarn
2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/usr/lib/hadoop-yarn
2014-03-26 01:13:32,021 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@26bc6487
2014-03-26 01:13:32,038 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx FQDN/xxxx. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,050 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,060 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x144fbf1d0350000, negotiated timeout = 5000
2014-03-26 01:13:32,076 WARN org.apache.hadoop.ha.ActiveStandbyElector:
Ignoring stale result from old client with sessionId 0x144fbf1d0350000
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Session:
0x144fbf1d0350000 closed
2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=xxxx:2181,yyyy:2181,zzzz:2181
sessionTimeout=5000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@74c35f27
2014-03-26 01:13:32,081 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2014-03-26 01:13:32,082 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server xxxx/xxxx:2181. Will not attempt to
authenticate using SASL (unknown error)
2014-03-26 01:13:32,083 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to xxxx/xxxx:2181, initiating session
2014-03-26 01:13:32,088 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server xxxx/xxxx:2181, sessionid =
0x344fbf1fe860001, negotiated timeout = 5000
2014-03-26 01:13:32,095 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Session connected.
2014-03-26 01:13:32,096 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8035
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8035: starting
2014-03-26 01:13:32,241 WARN org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
java.lang.IllegalArgumentException: Failed to specify server's Kerberos
principal name
2014-03-26 01:13:32,242 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:yarn/xxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
On Tue, Mar 25, 2014 at 4:55 PM, Manoj Samel <ma...@gmail.com>wrote:
> Hi,
>
> Hadoop version is CDH5 Beta1
>
> Name node and Resource managers have been configured in HA mode.
>
> After kerberos is enabled, the resource manager log shows following error
>
> 2014-03-25 22:21:06,854 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> java.lang.IllegalArgumentException: Failed to specify server's Kerberos
> principal name
> 2014-03-25 22:21:06,855 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:yarn/xxxxx@US-WEST-2.COMPUTE.INTERNAL (auth:KERBEROS)
> cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
> specify server's Kerberos principal name
>
> The keytab files for hdfs, mapred and yarn are (seemingly) correct and
> show correct list of principals using klist.
>
> Any thoughts,
>
> Thanks,
>