You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jacek Lewandowski (JIRA)" <ji...@apache.org> on 2015/10/30 08:18:27 UTC
[jira] [Comment Edited] (SPARK-11402) Allow to define a custom
driver runner and executor runner
[ https://issues.apache.org/jira/browse/SPARK-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982030#comment-14982030 ]
Jacek Lewandowski edited comment on SPARK-11402 at 10/30/15 7:18 AM:
---------------------------------------------------------------------
That's right, this is not only introduction of pluggable executor and driver runners. However, at least some of the changes are related to coding practises, which i believe are good:
- {{WorkerPage}} does use neither {{ExecutorRunner}} nor {{DriverRunner}}, but traits which expose only the methods to read information
- There is a functionality related to executor and driver runners which is common to both of them. By introducing interfaces I made this explicit - from scheduling point of view, either executor or driver are some services which consume resources so as far as scheduling is concerned, they should be generalised and handled uniformly
- the number of arguments which were passed to runner constructors grew to ridiculous numbers so the related settings were collected in dedicates case classes - those which represent the worker configuration and those which represent the process configuration
- fixed some inconsistency - the driver main directory was created inside the driver runner while executor directory was created in the worker - this was unified
Do you think the aforementioned changes are wrong? I don't believe they increase the maintenance and support effort because the refactoring i proposed is towards better code reuse, separation of concerns and encapsulation. The ability to plug a custom runner implementation is just a small part of this whole patch (and afaik using factories is also a good practice).
was (Author: jlewandowski):
That's right, this is not only introduction of pluggable executor and driver runners. However, at least some of the changes are related to coding practises, which i believe are good:
- {{WorkerPage}} does use neither {{ExecutorRunner}} nor {{DriverRunner}}, but traits which expose only the methods to read information
- There is a functionality related to executor and driver runners which is common to both of them. By introducing interfaces I made this explicit - from scheduling point of view, either executor or driver are some services which consume resources so as far as scheduling is concerned, they should be generalised and handled in uniformly
- the number of arguments which were passed to runner constructors grew to ridiculous numbers so the related settings were collected in dedicates case classes - those which represent the worker configuration and those which represent the process configuration
- fixed some inconsistency - the driver main directory was created inside the driver runner while executor directory was created in the worker - this was unified
Do you think the aforementioned changes are wrong? I don't believe they increase the maintenance and support effort because the refactoring i proposed is towards better code reuse, separation of concerns and encapsulation. The ability to plug a custom runner implementation is just a small part of this whole patch (and afaik using factories is also a good practice).
> Allow to define a custom driver runner and executor runner
> ----------------------------------------------------------
>
> Key: SPARK-11402
> URL: https://issues.apache.org/jira/browse/SPARK-11402
> Project: Spark
> Issue Type: Improvement
> Components: Deploy, Spark Core
> Reporter: Jacek Lewandowski
> Priority: Minor
>
> {{DriverRunner}} and {{ExecutorRunner}} are used by Spark Worker in standalone mode to spawn driver and executor processes respectively. When integrating Spark with some environments, it would be useful to allow providing a custom implementation of those components.
> The idea is simple - provide factory class names for driver and executor runners in Worker configuration. By default, the current implementations are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org