[jira] [Comment Edited] (TWILL-217) AppMaster launcher should include eventHandler dependencies and nothing else from application

["Martin Serrano (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TWILL-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881207#comment-15881207 ] 

Martin Serrano edited comment on TWILL-217 at 2/23/17 8:31 PM:
---------------------------------------------------------------

Seems then that we will need some way to get the list of all org.apache.twill.api.* classes off of the classpath in order to do this dependency walk:

* We could do this ourselves with some custom code.
* Guava 14 introduced such a utility: [ClassPath|https://github.com/google/guava/wiki/ReflectionExplained#classpath].  What is the history around using guava 13 versus later libraries?
* Could sort of cheat, and do something like what {{YarnTwillRunnerService}} does but restrict for the api package:
{code:language=java}
    // Find all the classpaths for Twill classes. It is used for class filtering when building application jar
    // in the YarnTwillPreparer
    Dependencies.findClassDependencies(getClass().getClassLoader(), new ClassAcceptor() {
      @Override
      public boolean accept(String className, URL classUrl, URL classPathUrl) {
        if (!className.startsWith("org.apache.twill.")) {
          return false;
        }
        twillClassPaths.add(classPathUrl);
        return true;
      }
    }, getClass().getName());
{code}

Thoughts?


was (Author: mserrano):
Seems then that we will need some way to get the list of all org.apache.twill.api.* classes off of the classpath in order to do this dependency walk:

* We could do this ourselves with some custom code.
* Guava 14 introduced such a utility: [ClassPath|https://github.com/google/guava/wiki/ReflectionExplained#classpath].  What is the history around using guava 13 versus later libraries?
* Could sort of cheat, and do something like what {{YarnTwillRunnerService}} does but just for the api package:
{code:language=java}
    // Find all the classpaths for Twill classes. It is used for class filtering when building application jar
    // in the YarnTwillPreparer
    Dependencies.findClassDependencies(getClass().getClassLoader(), new ClassAcceptor() {
      @Override
      public boolean accept(String className, URL classUrl, URL classPathUrl) {
        if (!className.startsWith("org.apache.twill.")) {
          return false;
        }
        twillClassPaths.add(classPathUrl);
        return true;
      }
    }, getClass().getName());
{code}

Thoughts?

> AppMaster launcher should include eventHandler dependencies and nothing else from application
> ---------------------------------------------------------------------------------------------
>
>                 Key: TWILL-217
>                 URL: https://issues.apache.org/jira/browse/TWILL-217
>             Project: Apache Twill
>          Issue Type: Improvement
>          Components: yarn
>    Affects Versions: 0.9.0
>            Reporter: Martin Serrano
>            Assignee: Martin Serrano
>
> Currently the launcher for the appmaster includes the application.jar libraries.  This is to support user code that adds an EventHandler.  The application may have many dependencies and including them in the appmaster classpath can lead to otherwise inaddressable incompatibilities.
> In my case, something in my application's large dependency graph was interfering with the Kafka server operation.  I was not able to determine what it was but tweaking the appmaster loader to not include my application jars fixed the issue.
> Instead the bundler that creates the twill.jar should include the EventHandler extension (if any) as an explicit dependency.  In this way, only the jars needed to support the event handler will be on the twill classpath.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)