You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mario Briggs (JIRA)" <ji...@apache.org> on 2016/10/18 18:49:58 UTC
[jira] [Comment Edited] (SPARK-17917) Convert 'Initial job has not
accepted any resources..' logWarning to a SparkListener event
[ https://issues.apache.org/jira/browse/SPARK-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574639#comment-15574639 ]
Mario Briggs edited comment on SPARK-17917 at 10/18/16 6:49 PM:
----------------------------------------------------------------
>>
I don't have a strong feeling on this partly because I'm not sure what the action then is – kill the job?
<<
Here is an example - Lets say i am using a notebook and kicked off some spark actions that dont' get executors because user/org/group quota's of executors have been exhausted. These events can be used by the notebook implementor to then surface the issue to the user via a UI update on that cell itself, maybe even additionally query the user/org/group quota's, show which apps are using up the quota's etc and allow the user to take what action required (kill the other jobs, just wait on this job etc). Therefore not looking to define in anyway on the event, what the set of actions can be, since that would be very implementation specific.
>>
Maybe, I suppose it will be a little tricky to define what the event is here
<<
Where you referring to the actual arguments of the event method. I can give a shot at defining and then look for feedback
was (Author: mariobriggs):
>>
I don't have a strong feeling on this partly because I'm not sure what the action then is – kill the job?
<<
Here is an example - Lets say i am using a notebook and kicked off some spark actions that dont' get executors because user/org/group quota's of executors have been exhausted. These events can be used by the notebook implementor to then surface the issue to the user via a UI update on that cell itself, maybe even additionally query the user/org/group quota's show which apps are using up the quota's etc and allow the user to take what action required (kill the other jobs, just wait on this job etc). Therefore not looking to define in anyway on the event, what the set of actions can be, since that would be very implementation specific.
>>
Maybe, I suppose it will be a little tricky to define what the event is here
<<
Where you referring to the actual arguments of the event method. I can give a shot at defining and then look for feedback
> Convert 'Initial job has not accepted any resources..' logWarning to a SparkListener event
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-17917
> URL: https://issues.apache.org/jira/browse/SPARK-17917
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Mario Briggs
> Priority: Minor
>
> When supporting Spark on a multi-tenant shared large cluster with quotas per tenant, often a submitted taskSet might not get executors because quotas have been exhausted (or) resources unavailable. In these situations, firing a SparkListener event instead of just logging the issue (as done currently at https://github.com/apache/spark/blob/9216901d52c9c763bfb908013587dcf5e781f15b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L192), would give applications/listeners an opportunity to handle this more appropriately as needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org