[jira] [Commented] (REEF-1791) Implement reef-runtime-spark

["Saikat Kanjilal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243385#comment-16243385 ] 

Saikat Kanjilal commented on REEF-1791:
---------------------------------------

Current status:
1) Still researching how to implement the launching of the Evaluators from inside the SparkRunner
2) Removed extraneous files from PR
3) Added SparkRuntimeConfiguration, for now it will mimic something similar to the following:  https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala and all configs can be done in the DataLoaderREEFOnSpark class


Stay tuned while the investigation for 1 continues.

> Implement reef-runtime-spark
> ----------------------------
>
>                 Key: REEF-1791
>                 URL: https://issues.apache.org/jira/browse/REEF-1791
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>         Attachments: REEF_1791_Workflow.jpg, file-1.jpeg, file.jpeg
>
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
>
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or {{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll need to run .NET Tasks on Executors as well.
> h3. P.S. Here's a REEF Wiki page with more details: *[Spark+REEF integration|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=73636401]*



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)