[jira] [Comment Edited] (OODT-700) Create a Spark Runner for the Wengine

["Michael Starch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OODT-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012432#comment-14012432 ] 

Michael Starch edited comment on OODT-700 at 5/29/14 3:14 PM:
--------------------------------------------------------------

The clear question here is: is the "Runner" mechanism the best place for this multiplexing of different styles of running jobs, or is the resource manager?  Opinions?


was (Author: starchmd@umich.edu):
The clear question here is: is the "Runner" mechanism the best place for this multiplexing of different styles of running jobs, or is the resource manager?

> Create a Spark Runner for the Wengine
> -------------------------------------
>
>                 Key: OODT-700
>                 URL: https://issues.apache.org/jira/browse/OODT-700
>             Project: OODT
>          Issue Type: Sub-task
>          Components: file manager, resource manager, workflow manager
>            Reporter: Michael Starch
>
> Spark is a useful technology for processing of streams and running map-reduce within the memory of a cluster.  Thus it is important to fold it into the streaming augmentation of OODT.
> Currently, the wengine runner extension point seems to be the best place to put this.  In order for the project to be fully backwards compatible, some processing point in OODT must allow multiplexing of jobs (some are run on mesos via resource manager, some are run in Spark, etc.).



--
This message was sent by Atlassian JIRA
(v6.2#6252)