You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Davinder Kumar <da...@ampool.io> on 2018/10/17 05:31:11 UTC
Hadoop-Token-Across-Kerberized -Cluster
Hello All,
Need one help for Kerberized cluster. Having two Ambari-clusters. Cluster A and Cluster B both are Kerberized with same KDC
Use case is : Need to access the Hive data from Cluster B to Cluster A.
Action done
- Remote Cluster B Principal and keytab are provided to Cluster A. [admadmin is the user]
- Remote Cluster Hive metastore Principal/Keytab are provided to cluster A.
- Running the spark job on cluster A to access the data from Cluster B [Spark over yarn]
- Able to connect with Hive metastore of Remote cluster B by cluster A
- Now getting the error related with Hadoop-tokens [Any help or suggestion is appreciated]. Error logs are like this
18/10/16 20:33:55 INFO RMProxy: Connecting to ResourceManager at davinderrc15.c.ampool-141120.internal/10.128.15.198:8030
18/10/16 20:33:55 INFO YarnRMClient: Registering the ApplicationMaster
18/10/16 20:33:55 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
18/10/16 20:33:55 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@10.128.15.198:38524)
18/10/16 20:33:55 INFO YarnAllocator: Submitted 2 unlocalized container requests.
18/10/16 20:33:55 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
18/10/16 20:33:56 INFO AMRMClientImpl: Received new token for : davinderrc15.c.ampool-141120.internal:45454
18/10/16 20:33:56 INFO YarnAllocator: Launching container container_e07_1539521606680_0045_02_000002 on host davinderrc15.c.ampool-141120.internal for executor with ID 1
18/10/16 20:33:56 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
18/10/16 20:33:56 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
18/10/16 20:33:56 INFO ContainerManagementProtocolProxy: Opening proxy : davinderrc15.c.ampool-141120.internal:45454
18/10/16 20:33:57 INFO YarnAllocator: Launching container container_e07_1539521606680_0045_02_000003 on host davinderrc15.c.ampool-141120.internal for executor with ID 2
18/10/16 20:33:57 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
18/10/16 20:33:57 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
18/10/16 20:33:57 INFO ContainerManagementProtocolProxy: Opening proxy : davinderrc15.c.ampool-141120.internal:45454
18/10/16 20:33:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.15.198:39686) with ID 1
18/10/16 20:33:59 INFO BlockManagerMasterEndpoint: Registering block manager davinderrc15.c.ampool-141120.internal:36291 with 366.3 MB RAM, BlockManagerId(1, davinderrc15.c.ampool-141120.internal, 36291, None)
18/10/16 20:33:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.15.198:39704) with ID 2
18/10/16 20:33:59 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
18/10/16 20:33:59 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
18/10/16 20:33:59 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse').
18/10/16 20:33:59 INFO SharedState: Warehouse path is 'file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse'.
18/10/16 20:33:59 INFO BlockManagerMasterEndpoint: Registering block manager davinderrc15.c.ampool-141120.internal:45507 with 366.3 MB RAM, BlockManagerId(2, davinderrc15.c.ampool-141120.internal, 45507, None)
18/10/16 20:34:00 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/10/16 20:34:00 INFO HiveClientImpl: Attempting to login to Kerberos using principal: admadmin/admin@AMPOOL.IO and keytab: admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:00 INFO UserGroupInformation: Login successful for user admadmin/admin@AMPOOL.IO using keytab file admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:01 INFO metastore: Trying to connect to metastore with URI thrift://10.128.0.39:9083
18/10/16 20:34:01 INFO metastore: Connected to metastore.
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/admadmin
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/90b26140-0c51-47b0-a32e-58e42fc6e764_resources
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/90b26140-0c51-47b0-a32e-58e42fc6e764
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/admadmin/90b26140-0c51-47b0-a32e-58e42fc6e764
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/90b26140-0c51-47b0-a32e-58e42fc6e764/_tmp_space.db
18/10/16 20:34:01 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse
18/10/16 20:34:01 INFO HiveClientImpl: Attempting to login to Kerberos using principal: admadmin/admin@AMPOOL.IO and keytab: admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:01 INFO UserGroupInformation: Login successful for user admadmin/admin@AMPOOL.IO using keytab file admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/3643bfd5-2400-4644-b84e-faae1b1d7cec_resources
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/3643bfd5-2400-4644-b84e-faae1b1d7cec
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/admadmin/3643bfd5-2400-4644-b84e-faae1b1d7cec
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/3643bfd5-2400-4644-b84e-faae1b1d7cec/_tmp_space.db
18/10/16 20:34:01 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse
18/10/16 20:34:01 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
18/10/16 20:34:04 INFO AmpoolDataFrame: SaveToAmpool :: table= default_orders1, parameters= Map(ampool.locator.host -> davinderrc15.c.ampool-141120.internal, path -> default_orders1, ampool.block.size -> 1000, ampool.num.splits -> 113, ampool.block.format -> AMP_BYTES, ampool.table.redundancy -> 0, ampool.locator.port -> 10334)
18/10/16 20:34:05 INFO Utils: No cache found.. trying to create using: {ampool.locator.host=davinderrc15.c.ampool-141120.internal, path=default_orders1, ampool.block.size=1000, ampool.num.splits=113, ampool.block.format=AMP_BYTES, ampool.table.redundancy=0, ampool.locator.port=10334}
18/10/16 20:34:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 380.9 KB, free 397.7 MB)
18/10/16 20:34:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 35.8 KB, free 397.7 MB)
18/10/16 20:34:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.128.15.198:36610 (size: 35.8 KB, free: 398.1 MB)
18/10/16 20:34:07 INFO SparkContext: Created broadcast 0 from
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 236 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 236 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 237 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 237 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 238 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 238 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 239 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 239 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 240 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 240 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 241 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 241 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 242 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 242 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO SparkContext: Starting job: foreachPartition at MTableWriter.scala:140
18/10/16 20:34:08 INFO DAGScheduler: Got job 0 (foreachPartition at MTableWriter.scala:140) with 14 output partitions
18/10/16 20:34:08 INFO DAGScheduler: Final stage: ResultStage 0 (foreachPartition at MTableWriter.scala:140)
18/10/16 20:34:08 INFO DAGScheduler: Parents of final stage: List()
18/10/16 20:34:08 INFO DAGScheduler: Missing parents: List()
18/10/16 20:34:08 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[24] at foreachPartition at MTableWriter.scala:140), which has no missing parents
18/10/16 20:34:08 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 22.1 KB, free 397.7 MB)
18/10/16 20:34:08 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 9.1 KB, free 397.7 MB)
18/10/16 20:34:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.128.15.198:36610 (size: 9.1 KB, free: 398.1 MB)
18/10/16 20:34:08 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
18/10/16 20:34:08 INFO DAGScheduler: Submitting 14 missing tasks from ResultStage 0 (MapPartitionsRDD[24] at foreachPartition at MTableWriter.scala:140) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13))
18/10/16 20:34:08 INFO YarnClusterScheduler: Adding task set 0.0 with 14 tasks
18/10/16 20:34:08 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, davinderrc15.c.ampool-141120.internal, executor 2, partition 0, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:08 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, davinderrc15.c.ampool-141120.internal, executor 1, partition 1, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on davinderrc15.c.ampool-141120.internal:36291 (size: 9.1 KB, free: 366.3 MB)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on davinderrc15.c.ampool-141120.internal:45507 (size: 9.1 KB, free: 366.3 MB)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on davinderrc15.c.ampool-141120.internal:36291 (size: 35.8 KB, free: 366.3 MB)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on davinderrc15.c.ampool-141120.internal:45507 (size: 35.8 KB, free: 366.3 MB)
18/10/16 20:34:10 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, davinderrc15.c.ampool-141120.internal, executor 1, partition 2, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:10 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, davinderrc15.c.ampool-141120.internal, executor 2, partition 3, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:10 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, davinderrc15.c.ampool-141120.internal, executor 2): java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "davinderrc15.c.ampool-141120.internal/10.128.15.198"; destination host is: "ar-n1-kc.c.ampool-141120.internal":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:785)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)
at com.sun.proxy.$Proxy16.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1238)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1225)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1213)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:266)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1538)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:332)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:327)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:340)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:786)
at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:109)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:251)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:250)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:720)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:683)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:770)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1620)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
... 56 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758)
... 59 more
18/10/16 20:34:10 INFO TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on davinderrc15.c.ampool-141120.internal, executor 1: java.io.IOException (Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "davinderrc15.c.ampool-141120.internal/10.128.15.198"; destination host is: "ar-n1-kc.c.ampool-141120.internal":8020; ) [duplicate 1]
18/10/16 20:34:10 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 4, davinderrc15.c.ampool-141120.internal, executor 1, partition 1, RACK_LOCAL, 5022 bytes)
-
Re: Hadoop-Token-Across-Kerberized -Cluster
Posted by Davinder Kumar <da...@ampool.io>.
Any update on this, or anybody facing/faced similar issue. Any suggestion will be appreciated.
Thanks
-Davinder
________________________________
From: Davinder Kumar <da...@ampool.io>
Sent: Wednesday, October 17, 2018 11:01 AM
To: dev
Subject: Hadoop-Token-Across-Kerberized -Cluster
Hello All,
Need one help for Kerberized cluster. Having two Ambari-clusters. Cluster A and Cluster B both are Kerberized with same KDC
Use case is : Need to access the Hive data from Cluster B to Cluster A.
Action done
- Remote Cluster B Principal and keytab are provided to Cluster A. [admadmin is the user]
- Remote Cluster Hive metastore Principal/Keytab are provided to cluster A.
- Running the spark job on cluster A to access the data from Cluster B [Spark over yarn]
- Able to connect with Hive metastore of Remote cluster B by cluster A
- Now getting the error related with Hadoop-tokens [Any help or suggestion is appreciated]. Error logs are like this
18/10/16 20:33:55 INFO RMProxy: Connecting to ResourceManager at davinderrc15.c.ampool-141120.internal/10.128.15.198:8030
18/10/16 20:33:55 INFO YarnRMClient: Registering the ApplicationMaster
18/10/16 20:33:55 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
18/10/16 20:33:55 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@10.128.15.198:38524)
18/10/16 20:33:55 INFO YarnAllocator: Submitted 2 unlocalized container requests.
18/10/16 20:33:55 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
18/10/16 20:33:56 INFO AMRMClientImpl: Received new token for : davinderrc15.c.ampool-141120.internal:45454
18/10/16 20:33:56 INFO YarnAllocator: Launching container container_e07_1539521606680_0045_02_000002 on host davinderrc15.c.ampool-141120.internal for executor with ID 1
18/10/16 20:33:56 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
18/10/16 20:33:56 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
18/10/16 20:33:56 INFO ContainerManagementProtocolProxy: Opening proxy : davinderrc15.c.ampool-141120.internal:45454
18/10/16 20:33:57 INFO YarnAllocator: Launching container container_e07_1539521606680_0045_02_000003 on host davinderrc15.c.ampool-141120.internal for executor with ID 2
18/10/16 20:33:57 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
18/10/16 20:33:57 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
18/10/16 20:33:57 INFO ContainerManagementProtocolProxy: Opening proxy : davinderrc15.c.ampool-141120.internal:45454
18/10/16 20:33:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.15.198:39686) with ID 1
18/10/16 20:33:59 INFO BlockManagerMasterEndpoint: Registering block manager davinderrc15.c.ampool-141120.internal:36291 with 366.3 MB RAM, BlockManagerId(1, davinderrc15.c.ampool-141120.internal, 36291, None)
18/10/16 20:33:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.15.198:39704) with ID 2
18/10/16 20:33:59 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
18/10/16 20:33:59 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
18/10/16 20:33:59 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse').
18/10/16 20:33:59 INFO SharedState: Warehouse path is 'file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse'.
18/10/16 20:33:59 INFO BlockManagerMasterEndpoint: Registering block manager davinderrc15.c.ampool-141120.internal:45507 with 366.3 MB RAM, BlockManagerId(2, davinderrc15.c.ampool-141120.internal, 45507, None)
18/10/16 20:34:00 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/10/16 20:34:00 INFO HiveClientImpl: Attempting to login to Kerberos using principal: admadmin/admin@AMPOOL.IO and keytab: admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:00 INFO UserGroupInformation: Login successful for user admadmin/admin@AMPOOL.IO using keytab file admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:01 INFO metastore: Trying to connect to metastore with URI thrift://10.128.0.39:9083
18/10/16 20:34:01 INFO metastore: Connected to metastore.
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/admadmin
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/90b26140-0c51-47b0-a32e-58e42fc6e764_resources
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/90b26140-0c51-47b0-a32e-58e42fc6e764
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/admadmin/90b26140-0c51-47b0-a32e-58e42fc6e764
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/90b26140-0c51-47b0-a32e-58e42fc6e764/_tmp_space.db
18/10/16 20:34:01 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse
18/10/16 20:34:01 INFO HiveClientImpl: Attempting to login to Kerberos using principal: admadmin/admin@AMPOOL.IO and keytab: admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:01 INFO UserGroupInformation: Login successful for user admadmin/admin@AMPOOL.IO using keytab file admadmin.keytab-a796621d-bacd-47e2-bd97-077090fe8aa8
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/3643bfd5-2400-4644-b84e-faae1b1d7cec_resources
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/3643bfd5-2400-4644-b84e-faae1b1d7cec
18/10/16 20:34:01 INFO SessionState: Created local directory: /hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/tmp/admadmin/3643bfd5-2400-4644-b84e-faae1b1d7cec
18/10/16 20:34:01 INFO SessionState: Created HDFS directory: /tmp/hive/admadmin/3643bfd5-2400-4644-b84e-faae1b1d7cec/_tmp_space.db
18/10/16 20:34:01 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/hadoop/yarn/local/usercache/admadmin/appcache/application_1539521606680_0045/container_e07_1539521606680_0045_02_000001/spark-warehouse
18/10/16 20:34:01 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
18/10/16 20:34:04 INFO AmpoolDataFrame: SaveToAmpool :: table= default_orders1, parameters= Map(ampool.locator.host -> davinderrc15.c.ampool-141120.internal, path -> default_orders1, ampool.block.size -> 1000, ampool.num.splits -> 113, ampool.block.format -> AMP_BYTES, ampool.table.redundancy -> 0, ampool.locator.port -> 10334)
18/10/16 20:34:05 INFO Utils: No cache found.. trying to create using: {ampool.locator.host=davinderrc15.c.ampool-141120.internal, path=default_orders1, ampool.block.size=1000, ampool.num.splits=113, ampool.block.format=AMP_BYTES, ampool.table.redundancy=0, ampool.locator.port=10334}
18/10/16 20:34:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 380.9 KB, free 397.7 MB)
18/10/16 20:34:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 35.8 KB, free 397.7 MB)
18/10/16 20:34:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.128.15.198:36610 (size: 35.8 KB, free: 398.1 MB)
18/10/16 20:34:07 INFO SparkContext: Created broadcast 0 from
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 236 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 236 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 237 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 237 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 238 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 238 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 239 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 239 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 240 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 240 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 241 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 241 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 242 for admadmin on 10.128.0.39:8020
18/10/16 20:34:08 INFO TokenCache: Got dt for hdfs://ar-n1-kc.c.ampool-141120.internal:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.128.0.39:8020, Ident: (HDFS_DELEGATION_TOKEN token 242 for admadmin)
18/10/16 20:34:08 INFO FileInputFormat: Total input paths to process : 1
18/10/16 20:34:08 INFO SparkContext: Starting job: foreachPartition at MTableWriter.scala:140
18/10/16 20:34:08 INFO DAGScheduler: Got job 0 (foreachPartition at MTableWriter.scala:140) with 14 output partitions
18/10/16 20:34:08 INFO DAGScheduler: Final stage: ResultStage 0 (foreachPartition at MTableWriter.scala:140)
18/10/16 20:34:08 INFO DAGScheduler: Parents of final stage: List()
18/10/16 20:34:08 INFO DAGScheduler: Missing parents: List()
18/10/16 20:34:08 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[24] at foreachPartition at MTableWriter.scala:140), which has no missing parents
18/10/16 20:34:08 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 22.1 KB, free 397.7 MB)
18/10/16 20:34:08 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 9.1 KB, free 397.7 MB)
18/10/16 20:34:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.128.15.198:36610 (size: 9.1 KB, free: 398.1 MB)
18/10/16 20:34:08 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
18/10/16 20:34:08 INFO DAGScheduler: Submitting 14 missing tasks from ResultStage 0 (MapPartitionsRDD[24] at foreachPartition at MTableWriter.scala:140) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13))
18/10/16 20:34:08 INFO YarnClusterScheduler: Adding task set 0.0 with 14 tasks
18/10/16 20:34:08 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, davinderrc15.c.ampool-141120.internal, executor 2, partition 0, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:08 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, davinderrc15.c.ampool-141120.internal, executor 1, partition 1, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on davinderrc15.c.ampool-141120.internal:36291 (size: 9.1 KB, free: 366.3 MB)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on davinderrc15.c.ampool-141120.internal:45507 (size: 9.1 KB, free: 366.3 MB)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on davinderrc15.c.ampool-141120.internal:36291 (size: 35.8 KB, free: 366.3 MB)
18/10/16 20:34:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on davinderrc15.c.ampool-141120.internal:45507 (size: 35.8 KB, free: 366.3 MB)
18/10/16 20:34:10 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, davinderrc15.c.ampool-141120.internal, executor 1, partition 2, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:10 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, davinderrc15.c.ampool-141120.internal, executor 2, partition 3, RACK_LOCAL, 5022 bytes)
18/10/16 20:34:10 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, davinderrc15.c.ampool-141120.internal, executor 2): java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "davinderrc15.c.ampool-141120.internal/10.128.15.198"; destination host is: "ar-n1-kc.c.ampool-141120.internal":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:785)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)
at com.sun.proxy.$Proxy16.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1238)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1225)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1213)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:266)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1538)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:332)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:327)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:340)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:786)
at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:109)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:251)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:250)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:720)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:683)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:770)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1620)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
... 56 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758)
... 59 more
18/10/16 20:34:10 INFO TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on davinderrc15.c.ampool-141120.internal, executor 1: java.io.IOException (Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "davinderrc15.c.ampool-141120.internal/10.128.15.198"; destination host is: "ar-n1-kc.c.ampool-141120.internal":8020; ) [duplicate 1]
18/10/16 20:34:10 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 4, davinderrc15.c.ampool-141120.internal, executor 1, partition 1, RACK_LOCAL, 5022 bytes)
-