You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ian Jiang <ia...@gmail.com> on 2012/02/11 16:18:46 UTC
ERROR security.UserGroupInformation at JobTracker and TaskTracker in
setting up hadoop-1.0.0 with secutiry
I am setting up a 2 nodes hadoop cluster with security, according to
the "Hadoop Security Guide" of Cloudera
(https://ccp.cloudera.com/display/CDHDOC/CDH3+Security+Guide).
Version info.
---------------
hadoop-1.0.0
kerberos-1.8.5
JDK 1.6.0_29
Redhat rhel 4u3
Cluster info.
---------------
host "slavez.demodomain" as NameNode
host "slavey.demodomain" as DataNode
host "master.dedomain" as Kerberos KDC and NTP server
(1) Firstly, the NameNode slavez starts, the DataNode slavez can join
in, and the HDFS works well. ( I am able to put files from local fs
into the HDFS, and to modify file attributes.)
(2) Then I starts the JobTracker at slavez, ant it begins waiting for
RPC from TaskTracker.
(3) But when I starts the TaskTracker at the same node, both the
JobTracker and the TaskTracker give out a
"security.UserGroupInformation" error for user
mapred/slavez.demodomain.
The following are detail logs and any suggestion is appreciated!
----------------------------------------------------------------------------------------------
JobTracker starts
----------------------------------------------------------------------------------------------
[mapred@slavez hadoop-1.0.0]$ ./bin/hadoop jobtracker
12/02/10 18:56:53 INFO mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = slavez.demodomain/192.168.0.14
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011
************************************************************/
12/02/10 18:56:53 INFO impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
12/02/10 18:56:53 INFO impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats
registered.
12/02/10 18:56:53 INFO impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
12/02/10 18:56:53 INFO impl.MetricsSystemImpl: JobTracker metrics system started
12/02/10 18:56:53 INFO impl.MetricsSourceAdapter: MBean for source
QueueMetrics,q=default
registered.
12/02/10 18:56:54 INFO impl.MetricsSourceAdapter: MBean for source ugi
registered.
12/02/10 18:56:54 INFO security.UserGroupInformation: Login successful for user
mapred/slavez.demodomain@HADOOP.DEMO.COM using keytab file
/etc/hadoop/conf/mapred.keytab
12/02/10 18:56:54 INFO
delegation.AbstractDelegationTokenSecretManager: Updating the current
master key for generating delegation tokens
12/02/10 18:56:54 INFO mapred.JobTracker: Scheduler configured with
(memSizeForMapSlotOnJT,
memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
limitMaxMemForReduceTasks) (-1, -1, -1, -
1)
12/02/10 18:56:54 INFO util.HostsFileReader: Refreshing hosts
(include/exclude) list
12/02/10 18:56:54 INFO
delegation.AbstractDelegationTokenSecretManager: Starting expired
delegation token remover thread, tokenRemoverScanInterval=60 min(s)
12/02/10 18:56:54 INFO
delegation.AbstractDelegationTokenSecretManager: Updating the current
master key for generating delegation tokens
12/02/10 18:56:54 INFO mapred.JobTracker: Starting jobtracker with
owner as mapred
12/02/10 18:56:54 INFO impl.MetricsSourceAdapter: MBean for source
RpcDetailedActivityForPort9001 registered.
12/02/10 18:56:54 INFO impl.MetricsSourceAdapter: MBean for source
RpcActivityForPort9001
registered.
12/02/10 18:56:54 INFO ipc.Server: Starting SocketReader
12/02/10 18:56:54 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter
(org.mortbay.log) via org.mortbay.log.Slf4jLog
12/02/10 18:56:55 INFO http.HttpServer: Added global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
12/02/10 18:56:55 INFO http.HttpServer: Port returned by
webServer.getConnectors()
[0].getLocalPort() before open() is -1. Opening the listener on 50030
12/02/10 18:56:55 INFO http.HttpServer: listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
12/02/10 18:56:55 INFO http.HttpServer: Jetty bound to port 50030
12/02/10 18:56:55 INFO mortbay.log: jetty-6.1.26
12/02/10 18:56:55 WARN mortbay.log: Can't reuse
/tmp/Jetty_0_0_0_0_50030_job____yn7qmk,
using /tmp/Jetty_0_0_0_0_50030_job____yn7qmk_2577057088123560744
12/02/10 18:56:55 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
12/02/10 18:56:55 INFO impl.MetricsSourceAdapter: MBean for source jvm
registered.
12/02/10 18:56:55 INFO impl.MetricsSourceAdapter: MBean for source
JobTrackerMetrics
registered.
12/02/10 18:56:55 INFO mapred.JobTracker: JobTracker up at: 9001
12/02/10 18:56:55 INFO mapred.JobTracker: JobTracker webserver: 50030
12/02/10 18:56:55 INFO mapred.JobTracker: Cleaning up the system directory
12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a
deprecated filesystem name.
Use "hdfs://slavez.demodomain/" instead.
12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a
deprecated filesystem name.
Use "hdfs://slavez.demodomain/" instead.
12/02/10 18:56:55 INFO mapred.JobTracker: History server being
initialized in embedded mode
12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a
deprecated filesystem name.
Use "hdfs://slavez.demodomain/" instead.
12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a
deprecated filesystem name.
Use "hdfs://slavez.demodomain/" instead.
12/02/10 18:56:55 INFO mapred.JobHistoryServer: Started job history server at:
localhost:50030
12/02/10 18:56:55 INFO mapred.JobTracker: Job History Server web
address: localhost:50030
12/02/10 18:56:55 INFO mapred.CompletedJobStatusStore: Completed job
store is inactive
12/02/10 18:56:55 INFO mapred.JobTracker: Refreshing hosts information
12/02/10 18:56:55 INFO util.HostsFileReader: Setting the includes file to
12/02/10 18:56:55 INFO util.HostsFileReader: Setting the excludes file to
12/02/10 18:56:55 INFO util.HostsFileReader: Refreshing hosts
(include/exclude) list
12/02/10 18:56:55 INFO mapred.JobTracker: Decommissioning 0 nodes
12/02/10 18:56:55 INFO ipc.Server: IPC Server Responder: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server listener on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 0 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 1 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 2 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 3 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 4 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 5 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 6 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 7 on 9001: starting
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 8 on 9001: starting
12/02/10 18:56:55 INFO mapred.JobTracker: Starting RUNNING
12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 9 on 9001: starting
(begin waiting)
----------------------------------------------------------------------------------------------
TaskTracker starts
----------------------------------------------------------------------------------------------
[mapred@slavez hadoop-1.0.0]$ ./bin/hadoop tasktracker
12/02/10 18:57:09 INFO mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = slavez.demodomain/192.168.0.14
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011
************************************************************/
12/02/10 18:57:09 INFO impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
12/02/10 18:57:09 INFO impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats
registered.
12/02/10 18:57:09 INFO impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
12/02/10 18:57:09 INFO impl.MetricsSystemImpl: TaskTracker metrics
system started
12/02/10 18:57:10 INFO impl.MetricsSourceAdapter: MBean for source ugi
registered.
12/02/10 18:57:10 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter
(org.mortbay.log) via org.mortbay.log.Slf4jLog
12/02/10 18:57:10 INFO http.HttpServer: Added global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
12/02/10 18:57:11 INFO mapred.TaskLogsTruncater: Initializing logs'
truncater with
mapRetainSize=-1 and reduceRetainSize=-1
12/02/10 18:57:11 WARN security.UserGroupInformation: Terminating renewal thread
12/02/10 18:57:11 INFO security.UserGroupInformation: Asked the TGT
renewer thread to
terminate
12/02/10 18:57:11 INFO security.UserGroupInformation: Login successful for user
mapred/slavez@HADOOP.DEMO.COM using keytab file /etc/hadoop/conf/mapred.keytab
12/02/10 18:57:11 INFO mapred.TaskTracker: Starting tasktracker with
owner as mapred
12/02/10 18:57:11 INFO mapred.TaskTracker: Good mapred local directories are:
/home/hadoop0209/hadoop-cluster/mapred/local
12/02/10 18:57:11 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your
platform... using builtin-java classes where applicable
12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source jvm
registered.
12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source
TaskTrackerMetrics
registered.
12/02/10 18:57:11 INFO ipc.Server: Starting SocketReader
12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source
RpcDetailedActivityForPort32958 registered.
12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source
RpcActivityForPort32958
registered.
12/02/10 18:57:11 INFO ipc.Server: IPC Server Responder: starting
12/02/10 18:57:11 INFO ipc.Server: IPC Server listener on 32958: starting
12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 0 on 32958: starting
12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 1 on 32958: starting
12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 2 on 32958: starting
12/02/10 18:57:11 INFO mapred.TaskTracker: TaskTracker up at:
localhost.localdomain/127.0.0.1:32958
12/02/10 18:57:11 INFO mapred.TaskTracker: Starting tracker
tracker_slavez:localhost.localdomain/127.0.0.1:32958
12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 3 on 32958: starting
12/02/10 18:57:11 ERROR security.UserGroupInformation:
PriviledgedActionException
as:mapred/slavez@HADOOP.DEMO.COM
cause:org.apache.hadoop.ipc.RemoteException: Failure to
initialize security context
12/02/10 18:57:11 INFO security.UserGroupInformation: Initiating logout for
mapred/slavez@HADOOP.DEMO.COM
12/02/10 18:57:11 INFO security.UserGroupInformation: Initiating re-login for
mapred/slavez@HADOOP.DEMO.COM
12/02/10 18:57:13 INFO mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at slavez.demodomain/192.168.0.14
************************************************************/
----------------------------------------------------------------------------------------------
JobTracker response
----------------------------------------------------------------------------------------------
12/02/10 18:57:11 ERROR security.UserGroupInformation:
PriviledgedActionException
as:mapred/slavez.demodomain@HADOOP.DEMO.COM
cause:javax.security.sasl.SaslException: Failure
to initialize security context [Caused by GSSException: No valid
credentials provided
(Mechanism level: Failed to find any Kerberos Key)]
12/02/10 18:57:11 INFO ipc.Server: IPC Server listener on 9001:
readAndProcess threw
exception javax.security.sasl.SaslException: Failure to initialize
security context [Caused
by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos
Key)]. Count of bytes read: 0
javax.security.sasl.SaslException: Failure to initialize security
context [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos
Key)]
at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:95)
at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:67)
at javax.security.sasl.Sasl.createSaslServer(Sasl.java:491)
at org.apache.hadoop.ipc.Server$Connection$1.run(Server.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs
(UserGroupInformation.java:1083)
at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:986)
at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1180)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:537)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:344)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: GSSException: No valid credentials provided (Mechanism
level: Failed to find any
Kerberos Key)
at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance
(Krb5AcceptCredential.java:95)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement
(Krb5MechFactory.java:111)
at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:178)
at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:384)
at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:42)
at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:139)
at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:78)
... 13 more
12/02/10 18:57:22 INFO mapred.JobTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at slavez.demodomain/192.168.0.14
************************************************************/
--
Ian Jiang