[jira] [Commented] (REEF-1791) Implement reef-runtime-spark

["Saikat Kanjilal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990132#comment-15990132 ] 

Saikat Kanjilal commented on REEF-1791:
---------------------------------------

[~motus] The work on this has begun, here's where the code lives:  https://github.com/skanjila/reef/tree/reef-1791

I have done the following:
1) Created a new maven project called reef-runtime-spark
2) Imported for now spark-core and spark-sql into the pom file
3) Took all of the reef-runtime-mesos code and brought that over or now, renamed all the mesos classes to spark
4) Added some stubbed out empty unit tests and called them Sparkxxx where xxx stands for client , driver etc


I will now take a deep dive into the design and add that design to this JIRA.   Let me know your thoughts to this approach. 

> Implement reef-runtime-spark
> ----------------------------
>
>                 Key: REEF-1791
>                 URL: https://issues.apache.org/jira/browse/REEF-1791
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
>
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or {{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll need to run .NET Tasks on Executors as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)