[jira] [Comment Edited] (YARN-6586) YARN to facilitate HTTPS in AM web server

["Robert Kanter (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508286#comment-16508286 ] 

Robert Kanter edited comment on YARN-6586 at 6/11/18 4:19 PM:
--------------------------------------------------------------

I've written up a design document and created a PoC patch that aims to solve this problem.  The details can be found in the design doc, but the basic idea is that the RM will generate certificates that the AMs can use.  

The PoC shows that the design described in the doc will work, it's just a bit rough around the edges and is missing some things like RM HA support, configs, etc.


Please take a look!


was (Author: rkanter):
I've written up a design document and created a PoC patch that aims to solve this problem.  The details can be found in the design doc, but the basic idea is that the RM will generate certificates that the AMs can use.  Please take a look!

> YARN to facilitate HTTPS in AM web server
> -----------------------------------------
>
>                 Key: YARN-6586
>                 URL: https://issues.apache.org/jira/browse/YARN-6586
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: yarn
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Haibo Chen
>            Assignee: Robert Kanter
>            Priority: Major
>         Attachments: Design Document v1.pdf, YARN-6586.poc.patch
>
>
> MR AM today does not support HTTPS in its web server, so the traffic between RMWebproxy and MR AM is in clear text.
> MR cannot easily achieve this mainly because MR AMs are untrusted by YARN. A potential solution purely within MR, similar to what Spark has implemented, is to allow users, when they enable HTTPS in MR job, to provide their own keystore file, and then the file is uploaded to distributed cache and localized for MR AM container. The configuration users need to do is complex.
> More importantly, in typical deployments, however, web browsers go through RMWebProxy to indirectly access MR AM web server. In order to support MR AM HTTPs, RMWebProxy therefore needs to trust the user-provided keystore, which is problematic.  
> Alternatively, we can add an endpoint in NM web server that acts as a proxy between AM web server and RMWebProxy. RMWebproxy, when configured to do so, will send requests in HTTPS to the NM on which the AM is running, and the NM then can communicate with the local AM web server in HTTP.   This adds one hop between RMWebproxy and AM, but both MR and Spark can use such solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org