[jira] [Updated] (REEF-429) Implement IMRU on Group Communications

["Markus Weimer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/REEF-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Weimer updated REEF-429:
-------------------------------
    Assignee: Dhruv Mahajan  (was: Markus Weimer)

> Implement IMRU on Group Communications
> --------------------------------------
>
>                 Key: REEF-429
>                 URL: https://issues.apache.org/jira/browse/REEF-429
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF.NET
>            Reporter: Markus Weimer
>            Assignee: Dhruv Mahajan
>
> h2. Client-Side input
> The user of the IMRU API will have to provide:
>   * A REEF Runtime Configuration. This will be used to submit the IMRU Driver.
>   * Configurations for the IMRU components: Map, Reduce and Update function as well as the needed Codec configurations.
>   * A Configuration that allows us to instantiate an instance of the (yet to be defined) {{IInputProviderDriver}} API.
>   * A Configuration for the yet to instantiate the (yet to be defined) {{IOutputProviderDriver}}
> h2. Driver side
> h3. Constructor
>   *  Use {{IInputProviderDriver}} to get partition information:
>     * Number of partitions
>     * Partition ID
>     * Partition locations (not immediately, but we will want this in the future)
>   * Use that information to configure Group Communications with:
>     * Data Broadcast and Reduce
>     * Control Broadcast (this is to indicate to the Mappers when to close)
> h3. OnNext(DriverStarted)
>   * Request the Evaluators as per the data provider's information
>   * Set a timeout for this request to be satisfied. If it is not within that time, exit the Driver.
> h3. OnNext(AllocatedEvaluator)
>   * For the {{UpdateFunction}}:
>     * Assemble the Service Configurations for group communication and data output
>     * Submit the {{UpdateTask}}
>   * For the {{MapFunction}}
>     * Assemble the Service Configurations for group communication and data input
>     * Submit the {{MapTask}}
> h3. OnNext(CompletedTask) and OnNext(CompletedEvaluator)
>   * Keep track of all Tasks and make sure that they all exit cleanly
>   * Set a timer when the first task completes and fail when not all tasks complete by the end of the timer.
> h3. OnNext(FailedTask) and OnNext(FailedEvaluator)
>   * Fail the Driver (for now)
> h2. UpdateTask
> h3. Constructor
>   * Establish and validate data output
>   * Establish and validate group communications
> h3. Call() main loop
>   * Use the {{IUpdateFunction}} instance to determine whether there is a next iteration
>   * If yes:
>     * Send control message to the mappers, followed by the data message
>   * If no:
>     * Send the ending control message to the mappers
>   * If the {{IUpdateFunction}} provided output, send it.
> h2. MapTask
> h3. Constructor
>   * Establish and validate data input
>   * Establish and validate group communications
> h3. Call() main loop
>   * Wait for control message.
>   * If one more iteration, call the {{IMapFunction}} with the data sent on the data broadcast.
>   * If the computation is to end, exit the loop.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)