[jira] [Commented] (REEF-1791) Implement reef-runtime-spark

["Saikat Kanjilal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197873#comment-16197873 ] 

Saikat Kanjilal commented on REEF-1791:
---------------------------------------

- Based on discussions with [~motus] I have:
1) Implemented SparkDataLoadingRequestBuilder and SparkRunner inside new reef-runtime-spark module
2) Implemented a driver in reef-examples called DataLoadingREEFOnSpark which invokes the two classes in number 1
3) Had to use anonymous functions and not lambda expressions since we have not moved to Java8 in the codebase yet

Currently we are using flatmap function to run the reef driver process within spark across all the partitions


Ready to test this now, will send cr later in the week for this first cut implementation.

> Implement reef-runtime-spark
> ----------------------------
>
>                 Key: REEF-1791
>                 URL: https://issues.apache.org/jira/browse/REEF-1791
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>         Attachments: REEF_1791_Workflow.jpg, file-1.jpeg, file.jpeg
>
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
>
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or {{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll need to run .NET Tasks on Executors as well.
> h3. P.S. Here's a REEF Wiki page with more details: *[Spark+REEF integration|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=73636401]*



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)