Mapreduce job submission via knox

[Rajesh Chandramohan <ra...@yahoo.com>]

Hi,
 To test mapreduce job submission via knox is that we  can do only via WebHCAT or Oozie.?
 Then regarding accessing Yarn proxy information, if our yarn Resource manager is exposing only https , how we can use via knox. Is that we need to add the certificate information of Resourcemanager to knox to make it work.  Also job submission via yarn rest-api is supported only in Hadoop-2.6 and above right.
<service>        <role>RESOURCEMANAGER</role>        <url>https://host-rm-1.vm.com:50030/ws</url>    </service>
 curl -ik --verbose --negotiate  -u : 'https://host:8443/gateway/sandbox/resourcemanager/v1/cluster' Thanks\Rajesh

Re: Mapreduce job submission via knox

[Kevin Minder <ke...@hortonworks.com>]

Hi Rajesh,

I don't believe I understand the core of your question just yet.  Here is some hopefully helpful advice to progress the conversation.

You can use Oozie and WebHCat to submit MapReduce jobs via Knox.
I believe you can also use RESOURCEMANAGER but Sumit would know better.

When accessing services via Knox you typically don't authenticate via Kerberos.  The role of Knox is mosts cases is to shield the client from Kerberos and allow the use of more common authentication schemes such as BasicAuth.  Knox then does the authentication (e.g. via LDAP) and propagates the user's identity to the services using a trusted proxy model.  This makes the curl command you used look like the command shown below.  You will be prompted for the password.

curl -iku your-username 'https://host:8443/gateway/sandbox/resourcemanager/v1/cluster'

The traffic between Knox and the ResourceManager is probably not setup with SSL in your env and would therefore not require certification configuration.  The Knox->Service communication can be setup with SSL/TLS and only requires that the urls in the knox topology files (e.g. conf/topologies/default.xml) be changed from http to https urls as long as the service SSL certificates are not self-signed.  If they are self-signed then the public keys from those self signed certificates need to be added to the truststore of the JVM that is used to run knox.

Kevin.

From: Rajesh Chandramohan <ra...@yahoo.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, Rajesh Chandramohan <ra...@yahoo.com>>
Date: Monday, February 15, 2016 at 7:09 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, Kevin Minder <ke...@hortonworks.com>>
Subject: Mapreduce job submission via knox

Hi,

 To test mapreduce job submission via knox is that we  can do only via WebHCAT or Oozie.?

 Then regarding accessing Yarn proxy information, if our yarn Resource manager is exposing only https , how we can use via knox. Is that we need to add the certificate information of Resourcemanager to knox to make it work.  Also job submission via yarn rest-api is supported only in Hadoop-2.6 and above right.

<service>
        <role>RESOURCEMANAGER</role>
        <url>https://host-rm-1.vm.com:50030/ws</url>
    </service>


 curl -ik --verbose --negotiate  -u : 'https://host:8443/gateway/sandbox/resourcemanager/v1/cluster'


Thanks
\Rajesh