[jira] [Comment Edited] (REEF-429) Implement IMRU on Group Communications

["Markus Weimer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642157#comment-14642157 ] 

Markus Weimer edited comment on REEF-429 at 7/27/15 11:53 PM:
--------------------------------------------------------------

I see. Let's discuss tomorrow. The effort of one over another does not seem to be too much to me unless I am ignoring something.
Dhruv



was (Author: dkm2110):
I see. Let's discuss tomorrow. The effort of one over another does not seem to be too much to me unless I am ignoring something.
Dhruv

Sent by Outlook<http://taps.io/outlookmobile> for Android



On Sun, Jul 26, 2015 at 6:07 PM -0700, "Markus Weimer (JIRA)" <ji...@apache.org>> wrote:


    [ https://issues.apache.org/jira/browse/REEF-429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642153#comment-14642153 ]

Markus Weimer commented on REEF-429:
------------------------------------

The way I was envisioning this to work (for now) was to have a {{PartitionedDataSet}} as one of the parameters to the IMRU Driver. It would then inform the IMRU scheduler about how many Evaluators are needed, what amount of memory they need and where the underlying data is. In a future world of actually having a scheduler for multiple stages, that partitioned data set would be provided by the upstream stage.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


> Implement IMRU on Group Communications
> --------------------------------------
>
>                 Key: REEF-429
>                 URL: https://issues.apache.org/jira/browse/REEF-429
>             Project: REEF
>          Issue Type: Sub-task
>          Components: REEF.NET
>            Reporter: Markus Weimer
>            Assignee: Markus Weimer
>
> h2. Client-Side input
> The user of the IMRU API will have to provide:
>   * A REEF Runtime Configuration. This will be used to submit the IMRU Driver.
>   * Configurations for the IMRU components: Map, Reduce and Update function as well as the needed Codec configurations.
>   * A Configuration that allows us to instantiate an instance of the (yet to be defined) {{IInputProviderDriver}} API.
>   * A Configuration for the yet to instantiate the (yet to be defined) {{IOutputProviderDriver}}
> h2. Driver side
> h3. Constructor
>   *  Use {{IInputProviderDriver}} to get partition information:
>     * Number of partitions
>     * Partition ID
>     * Partition locations (not immediately, but we will want this in the future)
>   * Use that information to configure Group Communications with:
>     * Data Broadcast and Reduce
>     * Control Broadcast (this is to indicate to the Mappers when to close)
> h3. OnNext(DriverStarted)
>   * Request the Evaluators as per the data provider's information
>   * Set a timeout for this request to be satisfied. If it is not within that time, exit the Driver.
> h3. OnNext(AllocatedEvaluator)
>   * For the {{UpdateFunction}}:
>     * Assemble the Service Configurations for group communication and data output
>     * Submit the {{UpdateTask}}
>   * For the {{MapFunction}}
>     * Assemble the Service Configurations for group communication and data input
>     * Submit the {{MapTask}}
> h3. OnNext(CompletedTask) and OnNext(CompletedEvaluator)
>   * Keep track of all Tasks and make sure that they all exit cleanly
>   * Set a timer when the first task completes and fail when not all tasks complete by the end of the timer.
> h3. OnNext(FailedTask) and OnNext(FailedEvaluator)
>   * Fail the Driver (for now)
> h2. UpdateTask
> h3. Constructor
>   * Establish and validate data output
>   * Establish and validate group communications
> h3. Call() main loop
>   * Use the {{IUpdateFunction}} instance to determine whether there is a next iteration
>   * If yes:
>     * Send control message to the mappers, followed by the data message
>   * If no:
>     * Send the ending control message to the mappers
>   * If the {{IUpdateFunction}} provided output, send it.
> h2. MapTask
> h3. Constructor
>   * Establish and validate data input
>   * Establish and validate group communications
> h3. Call() main loop
>   * Wait for control message.
>   * If one more iteration, call the {{IMapFunction}} with the data sent on the data broadcast.
>   * If the computation is to end, exit the loop.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)