You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by REYANE OUKPEDJO <r....@yahoo.com> on 2015/04/27 20:44:21 UTC

Yarn Localization failure on hadoop

Hi there,
I have  an Application that is trying to launch an AM , but the localization is failing with the below error message. The Resource visiblity is set to private which means the localizer will run through a container executor as a user that submit the job. I checked that hdfs dfs -ls / command is working fine as a user sumbitting the Application to make sure kerberos credentials for this user is not to be blamed.  a mapreduce  pi example could run successfully. Any idea what could cause this kind of issue?
Thanks
Reyane OUKPEDJO

NODE MANAGER LOGS HERE 2015-04-27 22:30:48,682 INFO  authorize.ServiceAuthorizationManager (ServiceAuthorizationManager.java:authorize(114)) - Authorization successful for testing (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB2015-04-27 22:30:49,463 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:update(932)) - DEBUG: FAILED { hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar, 1430154129414, FILE, null }, Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "datanode3.in.ibm.com/9.126.90.234"; destination host is: "datanode5.in.ibm.com":8020;2015-04-27 22:30:49,463 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(196)) - Resource hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar transitioned from DOWNLOADING to FAILED2015-04-27 22:30:49,463 INFO  container.Container (ContainerImpl.java:handle(901)) - Container container_1429925144518_0016_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED2015-04-27 22:30:49,464 INFO  localizer.LocalResourcesTrackerImpl (LocalResourcesTrackerImpl.java:handle(137)) - Container container_1429925144518_0016_01_000001 sent RELEASE event on a resource request { hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar, 1430154129414, FILE, null } not present in cache.2015-04-27 22:30:49,464 WARN  nodemanager.NMAuditLogger (NMAuditLogger.java:logFailure(150)) - USER=dsadm       OPERATION=Container Finished - Failed   TARGET=ContainerImpl    RESULT=FAILURE  DESCRIPTION=Container failed with state: LOCALIZATION_FAILED    APPID=application_1429925144518_0016    CONTAINERID=container_1429925144518_0016_01_0000012015-04-27 22:30:49,464 INFO  container.Container (ContainerImpl.java:handle(901)) - Container container_1429925144518_0016_01_000001 transitioned from LOCALIZATION_FAILED to DONE2015-04-27 22:30:49,465 INFO  application.Application (ApplicationImpl.java:transition(339)) - Removing container_1429925144518_0016_01_000001 from application application_1429925144518_00162015-04-27 22:30:49,465 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:isEnabled(169)) - Neither virutal-memory nor physical-memory monitoring is needed. Not running the monitor-thread2015-04-27 22:30:49,465 INFO  containermanager.AuxServices (AuxServices.java:handle(175)) - Got event CONTAINER_STOP for appId application_1429925144518_0016@