[jira] [Commented] (REEF-55) Delay the creation and upload of the Global JAR in the YARN runtime Driver

["Markus Weimer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232396#comment-14232396 ] 

Markus Weimer commented on REEF-55:
-----------------------------------

Can someone please look at this PR? Please run the tests on an actual YARN cluster, as this only changes things in the YARN runtime.

> Delay the creation and upload of the Global JAR in the YARN runtime Driver
> --------------------------------------------------------------------------
>
>                 Key: REEF-55
>                 URL: https://issues.apache.org/jira/browse/REEF-55
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF-Runtime-YARN
>            Reporter: Markus Weimer
>            Assignee: Markus Weimer
>              Labels: Performance
>
> {{EvaluatorSetupHelper}} is constructed as part of the YARN support in the Driver. The constructor of this class calls the {{.setup()}} which in turn (1) creates a JAR file with the global files for (future) Evaluator submissions and (2) Uploads it to the job folder on (H)DFS. 
> This adds significant delay in the construction of the object graph for the Driver. Hence, the code that creates and uploads the JAR file should be moved to a later stage of the computation, e.g. upon first call of {{getGlobalResources()}}. 
> Also, it would be wise to move the code into a {{Callable}} or {{Runnable}} such that we can move it to a thread as a further optimization after this one is done. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)