You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Markus Weimer (JIRA)" <ji...@apache.org> on 2015/01/22 18:50:36 UTC
[jira] [Reopened] (REEF-55) Delay the creation and upload of the
Global JAR in the YARN runtime Driver
[ https://issues.apache.org/jira/browse/REEF-55?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Weimer reopened REEF-55:
-------------------------------
> Delay the creation and upload of the Global JAR in the YARN runtime Driver
> --------------------------------------------------------------------------
>
> Key: REEF-55
> URL: https://issues.apache.org/jira/browse/REEF-55
> Project: REEF
> Issue Type: Improvement
> Components: REEF-Runtime-YARN
> Reporter: Markus Weimer
> Assignee: Markus Weimer
> Labels: Performance
> Fix For: 0.11-incubating
>
>
> {{EvaluatorSetupHelper}} is constructed as part of the YARN support in the Driver. The constructor of this class calls the {{.setup()}} which in turn (1) creates a JAR file with the global files for (future) Evaluator submissions and (2) Uploads it to the job folder on (H)DFS.
> This adds significant delay in the construction of the object graph for the Driver. Hence, the code that creates and uploads the JAR file should be moved to a later stage of the computation, e.g. upon first call of {{getGlobalResources()}}.
> Also, it would be wise to move the code into a {{Callable}} or {{Runnable}} such that we can move it to a thread as a further optimization after this one is done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)