You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Varun Vasudev (JIRA)" <ji...@apache.org> on 2015/03/27 12:55:54 UTC
[jira] [Commented] (YARN-3084) YARN REST API 2.6 - can't submit
simple job in hortonworks-allways job failes to run
[ https://issues.apache.org/jira/browse/YARN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383715#comment-14383715 ]
Varun Vasudev commented on YARN-3084:
-------------------------------------
[~Xquery] My apologies for not replying for so long. For some reason I didn't get any notification that you had replied to me. In response to your questions -
{quote{
1. First, for launching the application master, I see I need to provide it in the request. I tried to do it, but couldn’t find AppMaster.jar on the HDFS/local FS. I assume there is AppMaster per YARN-based application (MR, Pig, Hive etc.). Can you let me know how can I find/install/download such AppMaster jar?
{quote}
I would recommend using the DistributedShell jar. You can build it yourself from the Hadoop source code. The jar can be found at hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target. Copy this jar to a location on HDFS that's readable by the user running the job. I've attached a sample json file which you can use to submit to the REST API. One thing you should note - please update the values for "size" and "timestamp" under the "AppMaster.jar" key. The value for "resource" should be the full path to the jar on HDFS.
The DistributedShell app runs a command on multiple machines and exits. The commands write their output to a local file and that output is aggregated at the end of the job if log aggregation is enabled.
In the json I've uploaded, copy the script you want to run to HDFS(making sure it's readable by the user submitting the job). Set the value of the key "DISTRIBUTEDSHELLSCRIPTTIMESTAMP" to the timestamp for this script(on HDFS), the value of the key "DISTRIBUTEDSHELLSCRIPTLEN" to the size of the script and the value of "DISTRIBUTEDSHELLSCRIPTLOCATION" to the location on HDFS.
The 'num_containers' parameter(part of the "command" key) is the number of containers you wish to launch.
{quote}
2. After I launched the application master, in order to run the map reduce remotely, I need to run another rest api request (I guess), but couldn’t find any example for it. Do you have REST API example of how to run map reduce using REST? (or an explained how to/steps)
{quote}
I don't have an example for running MapReduce using REST. The MapReduce client for YARN is a thick client which does a lot of calculations such as creating the splits for the map before it submits the job. You will have to implement that logic yourself if you wish to submit MapReduce jobs. You don't need to run any other API once you submit the job. The AppMaster is responsible for scheduling your mappers and reducers.
{quote}
Also, if I want to run the application as user A password B, where I supposed to add my credentials and Identify; When I submit my map reduce job, isn’t yarn expects me to identify?
{quote}
Hadoop requires you to setup kerberos for secure mode. In secure mode, jobs are executed as the user who submitted the job. Credentials are picked up when you submit the job.
I'm going to close this issue since it doesn't seem like an issue with the REST API itself. If you have any further questions, we can discuss them offline.
> YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run
> ------------------------------------------------------------------------------------
>
> Key: YARN-3084
> URL: https://issues.apache.org/jira/browse/YARN-3084
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, webapp
> Affects Versions: 2.6.0
> Environment: Using eclipse on windows 7 (client)to run the map reduce job on the host of Hortonworks HDP 2.2 (hortonworks is on vmware version 6.0.2 build-1744117)
> Reporter: Michael Br
> Priority: Minor
> Attachments: submit-app.json, yarn-yarn-resourcemanager-sandbox.hortonworks.com.log
>
>
> Hello,
> 1. I want to run the simple Map Reduce job example (with the REST API 2.6 for yarn applications) and to calculate PI… for now it doesn’t work.
> When I use the command in the hortonworks terminal it works: “hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10”.
> But I want to submit the job with the REST API and not in the terminal as a command line. [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application]
> 2. I do succeed with other REST API requests: get state, get new application id and even kill(change state), but when I try to submit my example, the response is:
> --------------------------------------------------
> --------------------------------------------------
> The Response Header:
> Key : null ,Value : [HTTP/1.1 202 Accepted]
> Key : Date ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT]
> Key : Content-Length ,Value : [0]
> Key : Expires ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT]
> Key : Location ,Value : [http://[my port]:8088/ws/v1/cluster/apps/application_1421661392788_0038]
> Key : Content-Type ,Value : [application/json]
> Key : Server ,Value : [Jetty(6.1.26.hwx)]
> Key : Pragma ,Value : [no-cache, no-cache]
> Key : Cache-Control ,Value : [no-cache]
> The Respone Body:
> Null (No Response)
> --------------------------------------------------
> --------------------------------------------------
> 3. I need help with the http request body filling. I am doing a POST http request and I know that I am doing it right (in java).
> 4. I think the problem is in the request body.
> 5. I used this guy’s answer to help me build my map reduce example xml but it does not work: [http://hadoop-forum.org/forum/general-hadoop-discussion/miscellaneous/2136-how-can-i-run-mapreduce-job-by-rest-api].
> 6. What am I missing? (the description is not clear to me in the submit section of the rest api 2.6)
> 7. Does someone have an xml example for using a simple MR job?
> 8. Thanks! Here is the XML file I am using for the request body:
> --------------------------------------------------
> --------------------------------------------------
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <application-submission-context>
> <application-id>application_1421661392788_0038</application-id>
> <application-name>test_21_1</application-name>
> <queue>default</queue>
> <priority>3</priority>
> <am-container-spec>
> <environment>
> <entry>
> <key>CLASSPATH</key>
> <value>/usr/hdp/2.2.0.0-2041/hadoop/conf<CPS>/usr/hdp/2.2.0.0-2041/hadoop/lib/*<CPS>/usr/hdp/2.2.0.0-2041/hadoop/.//*<CPS>/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./<CPS>/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*<CPS>/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//*<CPS>/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*<CPS>/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//*<CPS>/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*<CPS>/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//*<CPS><CPS>/usr/share/java/mysql-connector-java-5.1.17.jar<CPS>/usr/share/java/mysql-connector-java.jar<CPS>/usr/hdp/current/hadoop-mapreduce-client/*<CPS>/usr/hdp/current/tez-client/*<CPS>/usr/hdp/current/tez-client/lib/*<CPS>/etc/tez/conf/<CPS>/usr/hdp/2.2.0.0-2041/tez/*<CPS>/usr/hdp/2.2.0.0-2041/tez/lib/*<CPS>/etc/tez/conf</value>
> </entry>
> </environment>
> <commands>
> <command>hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10</command>
> </commands>
> </am-container-spec>
> <unmanaged-AM>false</unmanaged-AM>
> <max-app-attempts>2</max-app-attempts>
> <resource>
> <memory>1024</memory>
> <vCores>1</vCores>
> </resource>
> <application-type>MAPREDUCE</application-type>
> <keep-containers-across-application-attempts>false</keep-containers-across-application-attempts>
> <application-tags>
> <tag>Michael</tag>
> <tag>PI example</tag>
> </application-tags>
> </application-submission-context>
> --------------------------------------------------
> --------------------------------------------------
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)