[jira] [Commented] (YARN-4577) Enable aux services to have their own custom classpath/jar file

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240658#comment-15240658 ] 

Vinod Kumar Vavilapalli commented on YARN-4577:
-----------------------------------------------

Just caught up with the wall of comments.

+1 in general for the ApplicationClassloader based solution. Aux-services was always a hack since Chris Douglas and I originally introduced it, the better solution is to move these services as first-class apps on top of YARN, but we are where we are.

[~sjlee0],
bq. For example, if the aux service code depends on another class property (owned by the aux service) in the configuration, that will be invoked via Configuration.getClass(), and it will still use the system classloader to load that class. Then it's very likely that you'll get a ClassNotFoundException.
Sangjin, you may be missing one important thing here - unlike in the MapReduce case, there is no shared Configuration object between NodeManager and the specific aux-service implementation here. We simply do not pass in any configuration anywhere as part of the AuxService APIs - so this entire thread of reasoning about getClass() is no long a problem? If needed, we can document advising against adding Conf as part of future API changes.

bq. The thread context classloader represents another similar problem. The moment the aux service code hits a code path that does Class.forName() that loads classes via the thread context classloader, and it needs to load an aux service-related class (that is not present in the main NM classpath), you will get a ClassNotFoundException.
In addition to wrapping aux-service API calls under a class-loader, wouldn't it suffice to simply have NM make all aux-services API calls in a separate thread whose ContextClassLoader is changed to be another custom one that resolves both System classes as well as AuxServices classes?

> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>
>                 Key: YARN-4577
>                 URL: https://issues.apache.org/jira/browse/YARN-4577
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-4577.1.patch, YARN-4577.2.patch, YARN-4577.20160119.1.patch, YARN-4577.20160204.patch, YARN-4577.3.patch, YARN-4577.3.rebase.patch, YARN-4577.4.patch
>
>
> Right now, users have to add their jars to the NM classpath directly, thus put them on the system classloader. But if multiple versions of the plugin are present on the classpath, there is no control over which version actually gets loaded. Or if there are any conflicts between the dependencies introduced by the auxiliary service and the NM itself, they can break the NM, the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that is different from the system classloader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)