[jira] [Created] (REEF-1747) Workaround to allow two AMs in one YARN container

["Sergiy Matusevych (JIRA)" <ji...@apache.org>]

Sergiy Matusevych created REEF-1747:
---------------------------------------

             Summary: Workaround to allow two AMs in one YARN container
                 Key: REEF-1747
                 URL: https://issues.apache.org/jira/browse/REEF-1747
             Project: REEF
          Issue Type: Bug
          Components: REEF Runtime YARN, REEF-Runtime-YARN
            Reporter: Sergiy Matusevych
            Assignee: Sergiy Matusevych


The current version of the YARN Java client does not allow YARN to register two Application Masters running in the same process. Technically, YARN Resource Manager does not care which process each AM runs in. However, the Java client contains a singleton {{UserGroupInformation}} that contains the user credentials. This data structure is shared by all AMs, and when REEF application tries to register the second (Unmanaged) AM, the client library presents to YARN RM the security token of the *first* AM. YARN rejects such registration request, throwing {{InvalidApplicationMasterRequestException}} "Application Master is already registered" exception.

A proper fix for this issue would be a patch for Hadoop YARN Java client; we also need a quick workaround in REEF so we can run REEF-on-REEF and Spark+REEF applications using unpatched Hadoop libraries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)