You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2019/03/01 14:39:00 UTC
[jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware
scheduling
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781728#comment-16781728 ]
Thomas Graves commented on SPARK-27005:
---------------------------------------
It seems like we are mixing gpu's as static resource vs generic one as accelerators. Perhaps we should consider making it generic but then having kind of a #define for gpu's so it can be consistent for users but if they want other generic ones they work as well.
So for instance of calling it *spark.task.gpus* we could call it spark.task.accelerator.[resource] to make that part generic and then we could define gpu as a known type so that its easier for users ot use the same name there.
I also assume in a few of the places you say search for gpu requirements you mean search for generic accelerator requirements - like in the *Expand RDD/Stage to support GPU* section?
If you are using a separate queue for the tasks with accelerator needs, does it still go through locality checks? Can you expand upon exactly where the queue is you are proposing adding? The scheduler goes through the resourceOffer code and down into resourceOfferSingleTaskSet for each locality level and then into the dequeueTask where it finally pulls from a queue, is this dequeueTask where you are proposing adding a new queue?
> Design sketch: Accelerator-aware scheduling
> -------------------------------------------
>
> Key: SPARK-27005
> URL: https://issues.apache.org/jira/browse/SPARK-27005
> Project: Spark
> Issue Type: Story
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Xingbo Jiang
> Priority: Major
>
> This task is to outline a design sketch for the accelerator-aware scheduling SPIP discussion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org