[jira] [Commented] (YARN-8448) AM HTTPS Support

["Robert Kanter (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556364#comment-16556364 ] 

Robert Kanter commented on YARN-8448:
-------------------------------------

I've finished up a patch that implements everything described in YARN-6586, other than the RM HA support (TODO in YARN-8449) and Documentation (just filed YARN-8582 for this).  I've put the bulk of the changes here (YARN-8448.001.patch), and the MapReduce changes in MAPREDUCE-4669.

Some notes on the patch:
- Updated BouncyCastle library to a newer version and had to also change the artifact from {{bcprov-jdk16}} to {{bcprov-jdk15on}}.  I know that sounds backwards, but jdk15on is actually newer and the one we should be using (see http://bouncy-castle.1462172.n4.nabble.com/Bouncycaslte-bcprov-jdk15-vs-bcprov-jdk16-td4656252.html).
- The {{yarn.resourcemanager.application-https.policy}} property controls how the RM should handle HTTPS when talking to AMs.  It can be {{OFF}}, {{OPTIONAL}} (default), or {{REQUIRED}}.  {{OFF}} makes it behave like today, where it does nothing special.  {{OPTIONAL}} makes it generate and provide the keystore and truststore to the AM when it sees an HTTPS tracking URL, but HTTP is also still allowed.  And {{REQUIRED}} is like {{OPTIONAL}}, but it won't follow HTTP tracking URLs.
- A lot of the code around the container executors is in providing/copying/etc the keystore and truststore files.  I've largely based this on the existing way we handle the credentials (delegation tokens) file.
- When provided a keystore file, the AM will get env vars {{KEYSTORE_FILE_LOCATION}} and {{KEYSTORE_PASSWORD}}; similarly, {{TRUSTSTORE_FILE_LOCATION}} and {{TRUSTSTORE_PASSWORD}} for the truststore file.
- Due to the (ugly) way we parse arguments in the LCE, I had to add an argument that's either {{--http}} or {{--https}} to indicate if we'll be providing it the keystore and truststore files.  Otherwise, there isn't a good way to have optional arguments.
- In order to keep things simple, I piggybacked passing the keystore and truststore files and passwords via secrets in the Credentials, which is already securely passed from the RM to the NM.
- {{ProxyCAManager}} is in charge of creating the certificates, keystores, and truststores.
- When writing the unit tests, I found a number of tests that were about 80% complete in what they were testing, which I completed in addition to adding tests for my changes.
-- I also tried to simplify some things (e.g. {{TestDockerContainerRuntime}} has ~30 tests that all duplicate the code for checking the arguments, and because I changed the number of arguments, they all failed - instead of updating them all, I created a helper method)
- I'm not sure what's up with {{test-container-executor}}, but unless my environment was messed up, it doesn't work when run as {{root}}; maybe people typically run it as a normal user?  The test talks about running as {{root}} as an option, and even has a few tests that only run when running as {{root}}.  I spent some time fixing this - it now runs in all 4 user configurations described in the existing comments.
- I've tested in a real cluster with the DefaultContainerExecutor and LinuxContainerExecutor using all combinations of {{yarn.resourcemanager.application-https.policy}}, {{yarn.app.mapreduce.am.webapp.https.enabled}}, and {{yarn.app.mapreduce.am.webapp.https.client.auth}} (see MAPREDUCE-4669), and everything behaved correctly.  I haven't tested out the DockerContainerExecutor.  
-- If you want to try this out yourself in a cluster, I'd recommend also applying the MAPREDUCE-4669 patch so you have an AM that supports the changes.  You can then use {{openssl s_client -connect <am-host>:<am-web-port>}} to get SSL details.  You can also try {{curl}}.

> AM HTTPS Support
> ----------------
>
>                 Key: YARN-8448
>                 URL: https://issues.apache.org/jira/browse/YARN-8448
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>            Priority: Major
>         Attachments: YARN-8448.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org