[jira] [Resolved] (REEF-1180) Split REEFClient submission parameter file into per-application and per-job

["Andrew Chung (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/REEF-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Chung resolved REEF-1180.
--------------------------------
    Resolution: Implemented

> Split REEFClient submission parameter file into per-application and per-job
> ---------------------------------------------------------------------------
>
>                 Key: REEF-1180
>                 URL: https://issues.apache.org/jira/browse/REEF-1180
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF.NET, REEF.NET Client
>            Reporter: Andrew Chung
>            Assignee: Andrew Chung
>
> Loosely defining the terms used here:
> Application: A REEF application.
> Job: Submissions of REEF applications to the RM framework.
> In other words, there can be many REEF jobs, but there is only one REEF application, from the RM framework point of view.
> Currently, we package both application and job parameters into the same Avro parameter file. We should break the file down into two parameter files -- one for application and another for job, such that we can generate a package once and re-use it many times as long as the user supplies the job parameters and the package (or the location of the package).
> In the YARN case, the zip file will be uploaded to YARN as an ARCHIVE resource. The zip file will contain the Avro application parameter file, while the Avro job parameter file will be uploaded to YARN as a FILE resource. This will allow the user to generate only the Avro file for the job, and will be able to reuse the package, provided that their general application parameters do not change.
> On YARN resource localization, the Avro job file will be localized outside of the "reef" folder, while the zip file will create the "reef" folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)