[jira] [Comment Edited] (REEF-1791) Implement reef-runtime-spark

["Saikat Kanjilal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092410#comment-16092410 ] 

Saikat Kanjilal edited comment on REEF-1791 at 7/19/17 12:40 AM:
-----------------------------------------------------------------

[~motus] can you take a look at this pull request, I have done the following since our last discussion:
1) Removed a bunch of code that is not needed in this module
2) Added a custom listener which extends the SparkListener interface and allows us to listen for the spark events coming from spark within the Reef app
3) I also moved the example code from the src/test location to the src/main location.
I will now start removing the yarn related configs from the example code

Pull request updated here: https://github.com/apache/reef/pull/1324/files


was (Author: kanjilal):
[~motus] can you take a look at this pull request, I have done the following since our last discussion:
1) Removed a bunch of code that is not needed in this module
2) Added a custom listener which extends the SparkListener interface and allows us to listen for the spark events coming from spark within the Reef app
3) I also moved the example code from the src/test location to the src/main location.
I will now start removing the yarn related configs from the example code

> Implement reef-runtime-spark
> ----------------------------
>
>                 Key: REEF-1791
>                 URL: https://issues.apache.org/jira/browse/REEF-1791
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>         Attachments: file-1.jpeg, file.jpeg
>
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
>
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or {{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll need to run .NET Tasks on Executors as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)