You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Travis Hegner (JIRA)" <ji...@apache.org> on 2017/03/23 17:17:41 UTC
[jira] [Commented] (SPARK-15176) Job Scheduling Within Application
Suffers from Priority Inversion
[ https://issues.apache.org/jira/browse/SPARK-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938813#comment-15938813 ]
Travis Hegner commented on SPARK-15176:
---------------------------------------
I'd like to have this feature available as well for a slightly different use case.
Currently we have an application that reconciles data from a relational database. It would be ideal to place any job within this app that touches the RDB into a resource capped pool ("maxShare" was what I instinctively looked for), to prevent overloading the remote RDB. The app has many other stages of data manipulations which would benefit from utilizing a larger share of the cluster, without the risk of hurting the RDB.
Limiting the number of executors and cores per executor works to prevent overloading the RDB, but also limits the entire application, causing it to run slower, and leaving potential idle resources on the table. Currently, the only other option I can think of would be to split the RDB pull into a separate app that persists the data to disk temporarily. This seems it would make the overall process take even more time, however.
Is anyone still working this issue? Should I look at porting the existing PR to 2.1.x branch?
> Job Scheduling Within Application Suffers from Priority Inversion
> -----------------------------------------------------------------
>
> Key: SPARK-15176
> URL: https://issues.apache.org/jira/browse/SPARK-15176
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.6.1
> Reporter: Nick White
>
> Say I have two pools, and N cores in my cluster:
> * I submit a job to one, which has M >> N tasks
> * N of the M tasks are scheduled
> * I submit a job to the second pool - but none of its tasks get scheduled until a task from the other pool finishes!
> This can lead to unbounded denial-of-service for the second pool - regardless of `minShare` or `weight` settings. Ideally Spark would support a pre-emption mechanism, or an upper bound on a pool's resource usage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org