You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Robert Kanter (JIRA)" <ji...@apache.org> on 2016/03/02 19:28:20 UTC
[jira] [Commented] (OOZIE-2477) Oozie Spark Node to support
Standalone and Mesos Deployment modes.
[ https://issues.apache.org/jira/browse/OOZIE-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176167#comment-15176167 ]
Robert Kanter commented on OOZIE-2477:
--------------------------------------
We've seen a number of problems with the Spark Action, largely due to classpath and other issues because we're running the SparkSubmit class in the Java Action. I'm all for refactoring it to use a REST API. Thanks for looking into this.
Here's some early feedback/questions based on the doc:
# We should ensure that yarn-client, yarn-cluster, and local modes still work (and don't require existing workflows to be updated or anything). In other words, we need to make sure this change is backwards compatible.
# Who receives the REST API calls? I'm more familiar with Yarn than Standalone and Mesos, but with used with Yarn, the only daemon I'm aware of is the Spark History Server.
# I asked someone more familiar with Spark about this internal REST API, and while he hasn't been following it too closely, he was concerned that it wasn't really designed with external clients in mind. Do we know how stable it is? If Oozie runs a lot of Spark Actions, will they hold up?
# Do we need to worry about the REST API changing between Spark versions? From my brief look at the Spark JIRA, it sounds like part of the idea was to ensure compatibility between Spark versions, so it shouldn't change; but we should look into what guarantees are made here.
> Oozie Spark Node to support Standalone and Mesos Deployment modes.
> ------------------------------------------------------------------
>
> Key: OOZIE-2477
> URL: https://issues.apache.org/jira/browse/OOZIE-2477
> Project: Oozie
> Issue Type: Improvement
> Reporter: Ahmed Kamal
> Labels: Spark
>
> I'm interested in extending the current spark node to support them and contributing this to the project. An initial design document is proposed here
> https://docs.google.com/document/d/12uf3B6VMgp_sI4sUiOwcmiMLTgLb5kL2weM7T0cgZNk/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)