[jira] [Created] (YARN-7618) YARN REST APi - can't launch yarn job on Kerberised Cluster

["Alexandre Linte (JIRA)" <ji...@apache.org>]

Alexandre Linte created YARN-7618:
-------------------------------------

             Summary: YARN REST APi - can't launch yarn job on Kerberised Cluster
                 Key: YARN-7618
                 URL: https://issues.apache.org/jira/browse/YARN-7618
             Project: Hadoop YARN
          Issue Type: Bug
    Affects Versions: 2.7.4
         Environment: Hadoop 2.7.4 - Kerberized cluster
            Reporter: Alexandre Linte
            Priority: Critical


Hello,

I'm currently trying to launch a yarn job through an Hadoop kerberized cluster following documentation ( https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html )

I'm doing these operation with an active kerberos keytab.

I begin to create my new-application 
{code:title=curl new-app|borderStyle=solid}
curl --negotiate -u : -XPOST http://uabigrm01.node.com:8088/ws/v1/cluster/apps/new-application

response :
{"application-id":"application_1507815642943_271826","maximum-resource-capability":{"memory":32768,"vCores":24}}
{code}

After that I submit my application : 
{code:title=curl submit|borderStyle=solid}
curl --negotiate -u : -XPOST -H "Content-Type: application/json" --data @"submit.json" http://uabigrm01.node.com:8088/ws/v1/cluster/apps
{code}

Content of submit.json file :
{code:title=submit.json|borderStyle=solid}
{
    "application-id":   "application_1507815642943_271826",
    "application-name": "yarn-api-test-new",
    "queue": "myqueue",
    "am-container-spec": {
        "commands": {
            "command": "{{HADOOP_HOME}}/bin/yarn jar /opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar wordcount /user/mwxk0647/WORK/dataset-input /user/mwxk0647/WORK/dataset-output-test-yarn"
        },
        "environment": {
           "entry": [{
                "key": "CLASSPATH",
                "value":"{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties"
            }]
        }
    },
    "unmanaged-AM": false,
    "max-app-attempts": 2,
    "resource": {
        "memory": 1024,
        "vCores": 1
    },
    "application-type": "MAPREDUCE",
    "keep-containers-across-application-attempts": false
}
{code}

I can see job on scheduler, he is submit but He failed due to a Kerberos authentication error...
{code:title=tracelogs|borderStyle=solid}
User: 	mwxk0647
Name: 	yarn-api-test
Application Type: 	MAPREDUCE
Application Tags: 	
YarnApplicationState: 	FAILED
Queue: 	myqueue
FinalStatus Reported by AM: 	FAILED
Started: 	Wed Dec 06 14:45:56 +0100 2017
Elapsed: 	10sec
Tracking URL: 	History
Diagnostics: 	
Application application_1507815642943_424552 failed 2 times due to AM Container for appattempt_1507815642943_424552_000002 exited with exitCode: 255
For more detailed output, check application tracking page:http://uabigrm01.node.com:8188/applicationhistory/app/application_1507815642943_424552Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1507815642943_424552_02_000001
Exit code: 255
Exception message: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "uabigdata69.node.com/10.77.64.69"; destination host is: "uabigname02.node.com":8020; 
{code}

So kerberos is OK for submit app, but not for launch job.

For the moment I make application works making manually my kinit on the datanode :
{code:borderStyle=solid}
“command”: "echo <MYPASSWORD> | kinit mwxk0647 && {{HADOOP_HOME}}/bin/yarn jar... 
{code}

But It's really ugly... And on scheduler, it display a first job which fail as before, but it launch the wordcount job, which is very strange..

How can i make YARN REST API work properly on a Kerberized Environment ?

Best Regards.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org