[jira] [Commented] (REEF-1791) Implement reef-runtime-spark

["Saikat Kanjilal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065616#comment-16065616 ] 

Saikat Kanjilal commented on REEF-1791:
---------------------------------------

[~markus.weimer][~motus] a few observations on deeper inspection:

1) As I perused through the example code we've written for reef on spark it seems that it uses the scala ARM library which does automatic resource management, please correct me if I'm wrong but I think the reef-runtime-spark should use the resource management enabled by yarn and not this library, am I correct here or the two pieces are different
2) The example simply prints out hello world as a reef task, I'd actually like to invoke one of the spark ML API's from reef as a success criteria for this (case in point linear regression or logistic regression), this would of course involve some research into the exact events that reef needs to listen for
3) It seems that the exampl works inside a look where it invokes methods on the client and then calls reef.run in an inner loop, I dont think this is how the runtime should run, in fact I propose a cleaner approach where the reef-runtime-spark gets a handle to the client based on the driver configuration and then invokes reef.run on this client


Thoughts?


> Implement reef-runtime-spark
> ----------------------------
>
>                 Key: REEF-1791
>                 URL: https://issues.apache.org/jira/browse/REEF-1791
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>         Attachments: file-1.jpeg, file.jpeg
>
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
>
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or {{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll need to run .NET Tasks on Executors as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)