[jira] [Commented] (REEF-1747) Workaround to allow two AMs in one YARN container

["Sergiy Matusevych (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902229#comment-15902229 ] 

Sergiy Matusevych commented on REEF-1747:
-----------------------------------------

A hacky solution that did *not* work: use maven shade plugin to pack and relocate all Hadoop libraries used by REEF into a different namespace. That would allow us to have our own Hadoop client, separated from the one used by e.g. Spark. We've tried that, and it did not work, because internally Hadoop uses reflection to instantiate certain objects (e.g. Hadoop Configuration), and that does not work when some Hadoop classes are relocated into different packages.

> Workaround to allow two AMs in one YARN container
> -------------------------------------------------
>
>                 Key: REEF-1747
>                 URL: https://issues.apache.org/jira/browse/REEF-1747
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF Runtime YARN, REEF-Runtime-YARN
>            Reporter: Sergiy Matusevych
>            Assignee: Sergiy Matusevych
>              Labels: workaround
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The current version of the YARN Java client does not allow YARN to register two Application Masters running in the same process. Technically, YARN Resource Manager does not care which process each AM runs in. However, the Java client contains a singleton {{UserGroupInformation}} that contains the user credentials. This data structure is shared by all AMs, and when REEF application tries to register the second (Unmanaged) AM, the client library presents to YARN RM the security token of the *first* AM. YARN rejects such registration request, throwing {{InvalidApplicationMasterRequestException}} _"Application Master is already registered"_ exception.
> A proper fix for this issue would be a patch for Hadoop YARN Java client; we also need a quick workaround in REEF so we can run REEF-on-REEF and Spark+REEF applications using unpatched Hadoop libraries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)