You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Imran Rashid (JIRA)" <ji...@apache.org> on 2018/08/13 15:15:00 UTC
[jira] [Commented] (SPARK-24954) Fail fast on job submit if run a
barrier stage with dynamic resource allocation enabled
[ https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578419#comment-16578419 ]
Imran Rashid commented on SPARK-24954:
--------------------------------------
I was thinking more about this, and realized you'll also run into issues w/ static allocation, and a multi-tenant cluster with pre-emption. You might keep gaining resources and having them taken away from you.
I'm not sure if there is anything we can really do about that, though, other than telling users to not run w/ preemption, though its hard to tell whether maybe they have things configured in a way they'll never get pre-empted.
> Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-24954
> URL: https://issues.apache.org/jira/browse/SPARK-24954
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Jiang Xingbo
> Assignee: Jiang Xingbo
> Priority: Blocker
> Fix For: 2.4.0
>
>
> Since we explicitly listed "Support running barrier stage with dynamic resource allocation" a Non-Goal in the design doc, we shall fail fast on job submit if running a barrier stage with dynamic resource allocation enabled, to avoid some confusing behaviors (can refer to SPARK-24942 for some examples).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org