You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2017/12/12 22:08:00 UTC
[jira] [Commented] (SPARK-22765) Create a new executor allocation
scheme based on that of MR
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288353#comment-16288353 ]
Sean Owen commented on SPARK-22765:
-----------------------------------
I'm not clear why this needs another allocation scheme. You say dynamic allocation has overhead at runtime -- yes -- and M/R doesn't because it's static. So why not disable dynamic allocation? Things you're identifying as "problems" are just because Spark is a generalization; you can write a bunch of independent 2-stage map-reduce jobs if you want. Killing idle executors is the point of dynamic allocation, not a problem.
I don't see any detail on how this differs from anything else in Spark.
> Create a new executor allocation scheme based on that of MR
> -----------------------------------------------------------
>
> Key: SPARK-22765
> URL: https://issues.apache.org/jira/browse/SPARK-22765
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 1.6.0
> Reporter: Xuefu Zhang
>
> Many users migrating their workload from MR to Spark find a significant resource consumption hike (i.e, SPARK-22683). While this might not be a concern for users that are more performance centric, for others conscious about cost, such hike creates a migration obstacle. This situation can get worse as more users are moving to cloud.
> Dynamic allocation make it possible for Spark to be deployed in multi-tenant environment. With its performance-centric design, its inefficiency has also unfortunately shown up, especially when compared with MR. Thus, it's believed that MR-styled scheduler still has its merit. Based on our research, the inefficiency associated with dynamic allocation comes in many aspects such as executor idling out, bigger executors, many stages (rather than 2 stages only in MR) in a spark job, etc.
> Rather than fine tuning dynamic allocation for efficiency, the proposal here is to add a new, efficiency-centric scheduling scheme based on that of MR. Such a MR-based scheme can be further enhanced and be more adapted to Spark execution model. This alternative is expected to offer good performance improvement (compared to MR) still with similar to or even better efficiency than MR.
> Inputs are greatly welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org