You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zhzhan <gi...@git.apache.org> on 2016/10/19 01:26:50 UTC
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
GitHub user zhzhan opened a pull request:
https://github.com/apache/spark/pull/15541
[SPARK-17637][Scheduler]Packed scheduling for Spark tasks across executors
## What changes were proposed in this pull request?
Restructure the code and implement two new task assigner.
PackedAssigner: try to allocate tasks to the executors with least available cores, so that spark can release reserved executors when dynamic allocation is enabled.
BalancedAssigner: try to allocate tasks to the executors with more available cores in order to balance the workload across all executors.
By default, the original round robin assigner is used.
We test a pipeline, and new PackedAssigner save around 45% regarding the reserved cpu and memory with dynamic allocation enabled.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Both unit test in TaskSchedulerImplSuite and manual tests in production pipeline.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhzhan/spark TaskAssigner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15541.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15541
----
commit 75cdd1a77a227fa492a09e93794d4ea7be8a020f
Author: Zhan Zhang <zh...@fb.com>
Date: 2016-10-19T01:20:48Z
TaskAssigner to support different scheduling algorithms
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84616347
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+
+ def assignTask(task: TaskDescription, cpu: Int): Unit = {
+ tasks += task
+ coresAvailable -= cpu
+ assert(coresAvailable >= 0)
+ }
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes multiple rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext/next() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, TaskScheduler calls offerAccepted() to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, after task assignment is done, TaskScheduler invokes the function tasks to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] sealed abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): TaskAssigner = {
+ this.cpuPerTask = cpuPerTask
+ this
+ }
+
+ /** The currently assigned offers. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /**
+ * Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ * By default, offers is randomly shuffled to avoid always placing tasks on the same set of
+ * workers.
+ */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ *
+ * @param isAccepted whether TaskScheduler assigns a task to current offer.
+ */
+ def offerAccepted(isAccepted: Boolean): Unit
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ val className = {
+ val name = assignerMap.get(assignerName.toLowerCase())
+ name.getOrElse {
+ logWarning(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ roundrobin
+ }
+ }
+ // The className is valid. No need to catch exceptions.
+ logInfo(s"Constructing TaskAssigner as $className")
+ Utils.classForName(className).getConstructor().newInstance().asInstanceOf[TaskAssigner]
+ .withCpuPerTask(cpuPerTask = conf.getInt("spark.task.cpus", 1))
+ }
+}
+
+/**
+ * Assigns the task to workers with available cores in roundrobin manner.
--- End diff --
`in a roundrobin manner`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67428/consoleFull)** for PR 15541 at commit [`a820e96`](https://github.com/apache/spark/commit/a820e96284f1d9108ef62cd3ef55171ebd47e08f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591169
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. There are three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
+ with randomness is used, which tries to allocate task to workers with available cores in
--- End diff --
`is used, which tries to allocate task` -> is used to allocate tasks`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590747
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
--- End diff --
`getNext()` -> `next()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84621076
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -250,24 +251,24 @@ private[spark] class TaskSchedulerImpl(
private def resourceOfferSingleTaskSet(
taskSet: TaskSetManager,
maxLocality: TaskLocality,
- shuffledOffers: Seq[WorkerOffer],
- availableCpus: Array[Int],
- tasks: IndexedSeq[ArrayBuffer[TaskDescription]]) : Boolean = {
+ taskAssigner: TaskAssigner) : Boolean = {
var launchedTask = false
- for (i <- 0 until shuffledOffers.size) {
- val execId = shuffledOffers(i).executorId
- val host = shuffledOffers(i).host
- if (availableCpus(i) >= CPUS_PER_TASK) {
+ taskAssigner.init()
+ while (taskAssigner.hasNext) {
+ var isAccepted = false
+ val currentOffer = taskAssigner.next()
+ val execId = currentOffer.workOffer.executorId
+ val host = currentOffer.workOffer.host
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK) {
try {
for (task <- taskSet.resourceOffer(execId, host, maxLocality)) {
- tasks(i) += task
+ currentOffer.assignTask(task, CPUS_PER_TASK)
val tid = task.taskId
taskIdToTaskSetManager(tid) = taskSet
taskIdToExecutorId(tid) = execId
executorIdToTaskCount(execId) += 1
- availableCpus(i) -= CPUS_PER_TASK
- assert(availableCpus(i) >= 0)
--- End diff --
@viirya The assert will not fail even in the legacy code, because taskSet.resourceOffer(execId, host, maxLocality) return an Option. The for loop at most run once.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15541
@rxin Would you like to take a look and let you know if you have any concern? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67395/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995649
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
--- End diff --
This feels like something which is set once for object generation and not supposed to be mutated later. I am not able to come up with better ways to do that. (Maybe pass it as a constructor param ?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591137
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. There are three task
--- End diff --
`There are three task` -> `Three task`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591073
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -250,24 +251,26 @@ private[spark] class TaskSchedulerImpl(
private def resourceOfferSingleTaskSet(
taskSet: TaskSetManager,
maxLocality: TaskLocality,
- shuffledOffers: Seq[WorkerOffer],
- availableCpus: Array[Int],
- tasks: IndexedSeq[ArrayBuffer[TaskDescription]]) : Boolean = {
+ taskAssigner: TaskAssigner) : Boolean = {
var launchedTask = false
- for (i <- 0 until shuffledOffers.size) {
- val execId = shuffledOffers(i).executorId
- val host = shuffledOffers(i).host
- if (availableCpus(i) >= CPUS_PER_TASK) {
+ taskAssigner.init()
+ while (taskAssigner.hasNext) {
+ var assigned = false
+ val currentOffer = taskAssigner.next()
+ val execId = currentOffer.workOffer.executorId
+ val host = currentOffer.workOffer.host
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK) {
try {
for (task <- taskSet.resourceOffer(execId, host, maxLocality)) {
- tasks(i) += task
+ currentOffer.tasks += task
val tid = task.taskId
taskIdToTaskSetManager(tid) = taskSet
taskIdToExecutorId(tid) = execId
executorIdToTaskCount(execId) += 1
- availableCpus(i) -= CPUS_PER_TASK
- assert(availableCpus(i) >= 0)
+ currentOffer.coresAvailable -= CPUS_PER_TASK
+ assert(currentOffer.coresAvailable >= 0)
--- End diff --
Could we add a function in `class OfferState` for including the following three lines of codes?
```Scala
currentOffer.tasks += task
currentOffer.coresAvailable -= CPUS_PER_TASK
assert(currentOffer.coresAvailable >= 0)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72506/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998771
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
--- End diff --
Use `val` and call `clear` in `init` below?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r85985739
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+
+ def assignTask(task: TaskDescription, cpu: Int): Unit = {
+ if (coresAvailable < cpu) {
+ throw new SparkException(s"Available cores are less than cpu per task" +
+ s" ($coresAvailable < $cpu)")
+ }
+ tasks += task
+ coresAvailable -= cpu
+ }
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes multiple rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext/next() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, TaskScheduler calls offerAccepted() to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, after task assignment is done, TaskScheduler invokes the function tasks to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] sealed abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): TaskAssigner = {
+ this.cpuPerTask = cpuPerTask
+ this
+ }
+
+ /** The currently assigned offers. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /**
+ * Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ * By default, offers is randomly shuffled to avoid always placing tasks on the same set of
+ * workers.
+ */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ *
+ * @param isAccepted whether TaskScheduler assigns a task to current offer.
+ */
+ def offerAccepted(isAccepted: Boolean): Unit
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ val className = {
+ val name = assignerMap.get(assignerName.toLowerCase())
+ name.getOrElse {
+ logWarning(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ roundrobin
+ }
+ }
+ // The className is valid. No need to catch exceptions.
+ logInfo(s"Constructing TaskAssigner as $className")
+ Utils.classForName(className).getConstructor().newInstance().asInstanceOf[TaskAssigner]
+ .withCpuPerTask(cpuPerTask = conf.getInt("spark.task.cpus", 1))
+ }
+}
+
+/**
+ * Assigns the task to workers with available cores in a roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var currentOfferIndex = 0
+
+ override def init(): Unit = {
+ currentOfferIndex = 0
+ }
+
+ override def hasNext: Boolean = currentOfferIndex < offer.size
+
+ override def next(): OfferState = {
+ offer(currentOfferIndex)
+ }
+
+ override def offerAccepted(isAccepted: Boolean): Unit = {
+ currentOfferIndex += 1
+ }
+}
+
+/**
+ * Assigns the task to workers with the most available cores. In other words, BalancedAssigner tries
+ * to distribute the task across workers in a balanced way. Potentially, it may alleviate the
+ * workers' memory pressure as less tasks running on the same workers, which also indicates that
+ * the task itself can make use of more computation resources, e.g., hyper-thread, across clusters.
+ */
+class BalancedAssigner extends TaskAssigner {
--- End diff --
These two assigner may behave similar in practice. The difference is that the balanced assigner tries to distribute the work load more aggressively.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83997070
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var idx = 0
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ idx = 0
+ sorted = offer.filter(_.coresAvailable >= CPUS_PER_TASK).sortBy(_.coresAvailable)
+ }
+
+ override def hasNext: Boolean = idx < sorted.size
+
+ override def getNext(): OfferState = {
+ currentOffer = sorted(idx)
--- End diff --
will change to make it similar to Iterator.next() method and add comments with similar comments to the Iterator.next()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15541
One concern I have is the cluster of machines with different codes. It is possibly if you construct your cluster with solution like ec2 and buy different kinds of nodes. In this case, the balance assigner would first consume machines with higher cores, and packed assigner would consume machines with less codes. I don't know if this is an issue to most of you. Even it is, this might not be able to solve in this change and we can consider this later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995590
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
--- End diff --
- nit: use `protected`
- nit: Why camel case ? This looks like an regular class var and not a constant. I know you want this to be treated as a constant but probably should have better guarding against that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84621271
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -250,24 +251,24 @@ private[spark] class TaskSchedulerImpl(
private def resourceOfferSingleTaskSet(
taskSet: TaskSetManager,
maxLocality: TaskLocality,
- shuffledOffers: Seq[WorkerOffer],
- availableCpus: Array[Int],
- tasks: IndexedSeq[ArrayBuffer[TaskDescription]]) : Boolean = {
+ taskAssigner: TaskAssigner) : Boolean = {
var launchedTask = false
- for (i <- 0 until shuffledOffers.size) {
- val execId = shuffledOffers(i).executorId
- val host = shuffledOffers(i).host
- if (availableCpus(i) >= CPUS_PER_TASK) {
+ taskAssigner.init()
+ while (taskAssigner.hasNext) {
+ var isAccepted = false
+ val currentOffer = taskAssigner.next()
+ val execId = currentOffer.workOffer.executorId
+ val host = currentOffer.workOffer.host
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK) {
try {
for (task <- taskSet.resourceOffer(execId, host, maxLocality)) {
- tasks(i) += task
+ currentOffer.assignTask(task, CPUS_PER_TASK)
val tid = task.taskId
taskIdToTaskSetManager(tid) = taskSet
taskIdToExecutorId(tid) = execId
executorIdToTaskCount(execId) += 1
- availableCpus(i) -= CPUS_PER_TASK
- assert(availableCpus(i) >= 0)
--- End diff --
@zhzhan ah, thanks for explanation! I don't notice it returns an Option. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83999827
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
--- End diff --
Do we need `reset()`? Looks like we only need `init()`. As we will call `init` before each assignment, it should be complete in resetting the status to initial.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67863/consoleFull)** for PR 15541 at commit [`a820e96`](https://github.com/apache/spark/commit/a820e96284f1d9108ef62cd3ef55171ebd47e08f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84421949
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
--- End diff --
Is this class private to `scheduler`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995272
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
--- End diff --
How about this?
```Scala
val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
val className = assignerMap.getOrElse(assignerName.toLowerCase(), roundrobin)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995463
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
--- End diff --
It might be better to put these into separate points and not as a paragraph. Also, I am not sure what the protocol is about putting details like method names in the doc. As things stand, it will serve good for people trying to read the code but as the codebase evolves, things might get out of sync if this comment is not updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84160226
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
--- End diff --
@zhzhan Let is preserve the behavior - for any application using this assigner, all tasksets will be executed based on the ordering of offers (both with and without good locality info).
The impact can be fairly non trivial - which is why shuffle'ing was initially added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84225583
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invoke the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests Whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ */
+ def offerAccepted(assigned: Boolean): Unit
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ val className = {
+ val name = assignerMap.get(assignerName.toLowerCase())
+ name.getOrElse {
+ logWarning(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ roundrobin
+ }
+ }
+ // The className is valid. No need to catch exceptions.
+ logInfo(s"Constructing TaskAssigner as $className")
+ val assigner = Utils.classForName(className)
+ .getConstructor().newInstance().asInstanceOf[TaskAssigner]
+ assigner.withCpuPerTask(cpuPerTask = conf.getInt("spark.task.cpus", 1))
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var currentOfferIndex = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ currentOfferIndex = 0
+ }
+
+ override def hasNext: Boolean = currentOfferIndex < offer.size
+
+ override def next(): OfferState = {
+ offer(currentOfferIndex)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ currentOfferIndex += 1
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores. It other words, BalancedAssigner tries
+ * to distribute the task across workers in a balanced way. Potentially, it may alleviate the
+ * workers' memory pressure as less tasks running on the same workers, which also indicates that
+ * the task itself can make use of more computation resources, e.g., hyper-thread, across clusters.
+ */
+class BalancedAssigner extends TaskAssigner {
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+ private val maxHeap: PriorityQueue[OfferState] = new PriorityQueue[OfferState]()
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ maxHeap.clear()
+ offer.filter(_.coresAvailable >= cpuPerTask).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def next(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= cpuPerTask && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for jobs, which over allocate resources than they need.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var currentOfferIndex = 0
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ currentOfferIndex = 0
+ sorted = offer.filter(_.coresAvailable >= cpuPerTask).sortBy(_.coresAvailable)
--- End diff --
@viirya Agree, good point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84119714
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
--- End diff --
After talking to several other people, they don't feel the shuffle is strongly needed. @mridulm If you don't mind, I will remove it in my next patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995189
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
--- End diff --
nit: `requested` => `requests`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97708/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4231/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15541
Sure will take a look in the next couple of days to get this into 2.1 if possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84154979
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
--- End diff --
@zhzhan Can you elaborate what the concern with shuffle'ing are ?
There were various reasons why we started shuffling offers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84129486
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -109,6 +108,85 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with B
assert(!failedTaskSet)
}
+ test("Scheduler does not always schedule tasks on the same workers") {
+ val taskScheduler = setupScheduler()
+ roundrobin(taskScheduler)
+ }
+
+ test("User can specify the roundrobin task assigner") {
+ val taskScheduler = setupScheduler(("spark.scheduler.taskAssigner", "RoUndrObin"))
+ roundrobin(taskScheduler)
+ }
+
+ test("Fallback to roundrobin when the task assigner provided is not valid") {
+ val taskScheduler = setupScheduler("spark.scheduler.taskAssigner" -> "invalid")
+ roundrobin(taskScheduler)
+ }
+
+ test("Scheduler balance the assignment to the worker with more free cores") {
--- End diff --
Will change the test case name to make it more explicit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84000016
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
--- End diff --
It sounds correct. However, I don't think it has real effect. Once the cores are the same, meaning no task gets assigned in previous run. So it doesn't matter if we begin with different order of offers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r85924864
--- Diff: docs/configuration.md ---
@@ -1350,6 +1350,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. Three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
--- End diff --
Nit: I suggest double quote the keywords "roundrobin", "packed", and "balanced" in this paragraph. E.g. `the "balanced" task assigner` sounds better to me than `the balanced task assigner`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998311
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -109,6 +108,85 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with B
assert(!failedTaskSet)
}
+ test("Scheduler does not always schedule tasks on the same workers") {
+ val taskScheduler = setupScheduler()
+ roundrobin(taskScheduler)
+ }
+
+ test("User can specify the roundrobin task assigner") {
+ val taskScheduler = setupScheduler(("spark.scheduler.taskAssigner", "RoUndrObin"))
+ roundrobin(taskScheduler)
+ }
+
+ test("Fallback to roundrobin when the task assigner provided is not valid") {
+ val taskScheduler = setupScheduler("spark.scheduler.taskAssigner" -> "invalid")
+ roundrobin(taskScheduler)
+ }
+
+ test("Scheduler balance the assignment to the worker with more free cores") {
--- End diff --
I think you should structure this so that tests for the same task assigner are packed together. Right now, one has the look into each test case to figure out which policy is being tested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998058
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var idx = 0
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ idx = 0
+ sorted = offer.filter(_.coresAvailable >= CPUS_PER_TASK).sortBy(_.coresAvailable)
+ }
+
+ override def hasNext: Boolean = idx < sorted.size
+
+ override def getNext(): OfferState = {
+ currentOffer = sorted(idx)
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable < CPUS_PER_TASK || !assigned) {
--- End diff --
If the current offer is rejected, it is not valid for the current taskset, (probably due to locality restriction). Each scheduling algorithm has to respect the locality restriction, and in the meantime provide next available offer to the taskset.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84222924
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invoke the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests Whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ */
+ def offerAccepted(assigned: Boolean): Unit
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ val className = {
+ val name = assignerMap.get(assignerName.toLowerCase())
+ name.getOrElse {
+ logWarning(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ roundrobin
+ }
+ }
+ // The className is valid. No need to catch exceptions.
+ logInfo(s"Constructing TaskAssigner as $className")
+ val assigner = Utils.classForName(className)
+ .getConstructor().newInstance().asInstanceOf[TaskAssigner]
+ assigner.withCpuPerTask(cpuPerTask = conf.getInt("spark.task.cpus", 1))
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var currentOfferIndex = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ currentOfferIndex = 0
+ }
+
+ override def hasNext: Boolean = currentOfferIndex < offer.size
+
+ override def next(): OfferState = {
+ offer(currentOfferIndex)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ currentOfferIndex += 1
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores. It other words, BalancedAssigner tries
+ * to distribute the task across workers in a balanced way. Potentially, it may alleviate the
+ * workers' memory pressure as less tasks running on the same workers, which also indicates that
+ * the task itself can make use of more computation resources, e.g., hyper-thread, across clusters.
+ */
+class BalancedAssigner extends TaskAssigner {
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+ private val maxHeap: PriorityQueue[OfferState] = new PriorityQueue[OfferState]()
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ maxHeap.clear()
+ offer.filter(_.coresAvailable >= cpuPerTask).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def next(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= cpuPerTask && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for jobs, which over allocate resources than they need.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var currentOfferIndex = 0
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ currentOfferIndex = 0
+ sorted = offer.filter(_.coresAvailable >= cpuPerTask).sortBy(_.coresAvailable)
--- End diff --
Sounds like the shuffling reason of `BalancedAssigner` can be applied here too? If no shuffling, this ensures that initial sorted offers will be randomized when cores are the same, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15541
@rxin @gatorsmile Can you please take a look, and kindly provide your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67924/consoleFull)** for PR 15541 at commit [`b06de5e`](https://github.com/apache/spark/commit/b06de5ed0c285f01230f7182bf53676a9f6de74e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83999756
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
--- End diff --
Your concern is valid. Each scheduling algorithms has its pros and cons, and which one is chosen depends on user's requirement. We mainly want to use this to save reserved resources combined with dynamic allocation. In our pipeline, we didn't observe the problem. If it happens, we need to investigate the memory allocation part to see whether it has problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97724/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #97724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97724/consoleFull)** for PR 15541 at commit [`ada2a45`](https://github.com/apache/spark/commit/ada2a45b43b9722bae99cf578b544195e69a1625).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995119
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
--- End diff --
"this worker" ? `this` wont represent a worker here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67290/consoleFull)** for PR 15541 at commit [`945d623`](https://github.com/apache/spark/commit/945d623701813d148e6c2763bc439dc4606af296).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591093
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -305,12 +309,8 @@ private[spark] class TaskSchedulerImpl(
hostsByRack.getOrElseUpdate(rack, new HashSet[String]()) += o.host
}
}
+ taskAssigner.construct(offers)
- // Randomly shuffle offers to avoid always placing tasks on the same set of workers.
--- End diff --
Could we add this comment to [the code in `construct`](https://github.com/zhzhan/spark/blob/dd2b2077430bbb07047e928d20c1ad8fe940827a/core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala#L79)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84619023
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+
+ def assignTask(task: TaskDescription, cpu: Int): Unit = {
+ tasks += task
+ coresAvailable -= cpu
+ assert(coresAvailable >= 0)
+ }
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes multiple rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext/next() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, TaskScheduler calls offerAccepted() to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, after task assignment is done, TaskScheduler invokes the function tasks to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] sealed abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): TaskAssigner = {
+ this.cpuPerTask = cpuPerTask
--- End diff --
You mean cpuPerTask >= 1? I don't think we need this check.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591196
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. There are three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
+ with randomness is used, which tries to allocate task to workers with available cores in
+ roundrobin manner. The packed task assigner tries to allocate tasks to workers with the least
+ free cores, resulting in tasks assigned to few workers, which may help driver to release the
+ reserved idle workers when dynamic allocation(spark.dynamicAllocation.enabled) is enabled.
+ The balanced task assigner tries to assign tasks across workers in a balance way (allocating
--- End diff --
`balance ` -> `balanced `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84618189
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+
+ def assignTask(task: TaskDescription, cpu: Int): Unit = {
+ tasks += task
+ coresAvailable -= cpu
+ assert(coresAvailable >= 0)
+ }
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes multiple rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext/next() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, TaskScheduler calls offerAccepted() to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, after task assignment is done, TaskScheduler invokes the function tasks to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] sealed abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): TaskAssigner = {
+ this.cpuPerTask = cpuPerTask
--- End diff --
Do we need to add sanity check for `cpuPerTask`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67428/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67224 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67224/consoleFull)** for PR 15541 at commit [`e81b279`](https://github.com/apache/spark/commit/e81b2796f7b0c0c4a52d7345f14e0ab4e0e143b3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83997822
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var idx = 0
--- End diff --
nit: `idx` => `currentOfferIndex`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84422647
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invoke the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests Whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
--- End diff --
`@return` is not aligned with the line above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67418/consoleFull)** for PR 15541 at commit [`6b29002`](https://github.com/apache/spark/commit/6b29002c29fecdbe32159dd0d31f53716630de46).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590773
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
--- End diff --
`After` -> `after`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84002480
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -109,6 +108,85 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext with B
assert(!failedTaskSet)
}
+ test("Scheduler does not always schedule tasks on the same workers") {
+ val taskScheduler = setupScheduler()
+ roundrobin(taskScheduler)
+ }
+
+ test("User can specify the roundrobin task assigner") {
+ val taskScheduler = setupScheduler(("spark.scheduler.taskAssigner", "RoUndrObin"))
+ roundrobin(taskScheduler)
+ }
+
+ test("Fallback to roundrobin when the task assigner provided is not valid") {
+ val taskScheduler = setupScheduler("spark.scheduler.taskAssigner" -> "invalid")
+ roundrobin(taskScheduler)
+ }
+
+ test("Scheduler balance the assignment to the worker with more free cores") {
--- End diff --
Can you please clarify?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15541
To be honest, I find the current API pretty weird (it is some stateful object that has to be reset every time). I suspect you designed this API by just abstracting out the logic you wanted to change from the existing implementation, but that doesn't necessarily lead to intuitive apis. It's been a while since I last checked the scheduler code, so it'd take me a while to page back in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84002685
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
--- End diff --
My concern is that if we do not use reset, the assigner has to keep internal resources until next time, but it is not big overhead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #69015 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69015/consoleFull)** for PR 15541 at commit [`ada2a45`](https://github.com/apache/spark/commit/ada2a45b43b9722bae99cf578b544195e69a1625).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67224/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15541
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4256/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan closed the pull request at:
https://github.com/apache/spark/pull/15541
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67924/consoleFull)** for PR 15541 at commit [`b06de5e`](https://github.com/apache/spark/commit/b06de5ed0c285f01230f7182bf53676a9f6de74e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995317
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
--- End diff --
nit: `different the locality restrictions` => `different locality restrictions`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84000957
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var idx = 0
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ idx = 0
+ sorted = offer.filter(_.coresAvailable >= CPUS_PER_TASK).sortBy(_.coresAvailable)
+ }
+
+ override def hasNext: Boolean = idx < sorted.size
+
+ override def getNext(): OfferState = {
+ currentOffer = sorted(idx)
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable < CPUS_PER_TASK || !assigned) {
--- End diff --
same as before. Shouldn't you add it to the heap any ways despite of what `assigned` is set to ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995208
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
--- End diff --
nit: `Tracking` => `Tracks`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998358
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,19 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. By default, roundrobin
--- End diff --
nit: create a list for each policy and explain inline instead of saying `former`, `latter` below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3570/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67157/consoleFull)** for PR 15541 at commit [`75cdd1a`](https://github.com/apache/spark/commit/75cdd1a77a227fa492a09e93794d4ea7be8a020f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15541
Accidentally, I deleted all my comments. You might need to check the emails to find all my comments. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67157/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84619879
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -250,24 +251,24 @@ private[spark] class TaskSchedulerImpl(
private def resourceOfferSingleTaskSet(
taskSet: TaskSetManager,
maxLocality: TaskLocality,
- shuffledOffers: Seq[WorkerOffer],
- availableCpus: Array[Int],
- tasks: IndexedSeq[ArrayBuffer[TaskDescription]]) : Boolean = {
+ taskAssigner: TaskAssigner) : Boolean = {
var launchedTask = false
- for (i <- 0 until shuffledOffers.size) {
- val execId = shuffledOffers(i).executorId
- val host = shuffledOffers(i).host
- if (availableCpus(i) >= CPUS_PER_TASK) {
+ taskAssigner.init()
+ while (taskAssigner.hasNext) {
+ var isAccepted = false
+ val currentOffer = taskAssigner.next()
+ val execId = currentOffer.workOffer.executorId
+ val host = currentOffer.workOffer.host
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK) {
try {
for (task <- taskSet.resourceOffer(execId, host, maxLocality)) {
- tasks(i) += task
+ currentOffer.assignTask(task, CPUS_PER_TASK)
val tid = task.taskId
taskIdToTaskSetManager(tid) = taskSet
taskIdToExecutorId(tid) = execId
executorIdToTaskCount(execId) += 1
- availableCpus(i) -= CPUS_PER_TASK
- assert(availableCpus(i) >= 0)
--- End diff --
Thanks @viirya for the comments. Actually I was thinking removing the check although it is part of the legacy code. Now the check is moved into OfferState, which makes more sense. IMHO, typically the assertion should never fail. But from the OfferState's perspective, it should guarantee such restriction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84158910
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
--- End diff --
@mridulm I am not sure how much the shuffle can impact the scheduling, and thus don't have strong opinion on this.
@viirya Even if the cores are the same, it does not mean that "no task gets assigned in previous run". Shuffling does take effect here. For example, the previous round may be (5, 4, 3), and one core is allocated, then the current round would be (4, ,4 3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67395/consoleFull)** for PR 15541 at commit [`dd2b207`](https://github.com/apache/spark/commit/dd2b2077430bbb07047e928d20c1ad8fe940827a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83996351
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
--- End diff --
This deserves to be a warn
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83999715
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
--- End diff --
Shouldn't you add it to the heap any ways despite of what `assigned` is set to ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998916
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -60,7 +58,7 @@ private[spark] class TaskSchedulerImpl(
def this(sc: SparkContext) = this(sc, sc.conf.get(config.MAX_TASK_FAILURES))
val conf = sc.conf
-
+ private val taskAssigner: TaskAssigner = TaskAssigner.init(conf)
--- End diff --
nit: extra space in `private val`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r85882792
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+
+ def assignTask(task: TaskDescription, cpu: Int): Unit = {
+ if (coresAvailable < cpu) {
+ throw new SparkException(s"Available cores are less than cpu per task" +
+ s" ($coresAvailable < $cpu)")
+ }
+ tasks += task
+ coresAvailable -= cpu
+ }
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes multiple rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext/next() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, TaskScheduler calls offerAccepted() to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, after task assignment is done, TaskScheduler invokes the function tasks to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] sealed abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): TaskAssigner = {
+ this.cpuPerTask = cpuPerTask
+ this
+ }
+
+ /** The currently assigned offers. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /**
+ * Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ * By default, offers is randomly shuffled to avoid always placing tasks on the same set of
+ * workers.
+ */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ *
+ * @param isAccepted whether TaskScheduler assigns a task to current offer.
+ */
+ def offerAccepted(isAccepted: Boolean): Unit
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ val className = {
+ val name = assignerMap.get(assignerName.toLowerCase())
+ name.getOrElse {
+ logWarning(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
--- End diff --
i'd actually fail Spark if it cannot be constructed -- otherwise it is easier to make mistakes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/15541
@rxin @zhzhan Is there any chance that we could reach any consensus on the API design and move forward with this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #97780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97780/consoleFull)** for PR 15541 at commit [`ada2a45`](https://github.com/apache/spark/commit/ada2a45b43b9722bae99cf578b544195e69a1625).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591178
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. There are three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
+ with randomness is used, which tries to allocate task to workers with available cores in
+ roundrobin manner. The packed task assigner tries to allocate tasks to workers with the least
+ free cores, resulting in tasks assigned to few workers, which may help driver to release the
--- End diff --
`few` -> `fewer`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84616602
--- Diff: docs/configuration.md ---
@@ -1350,6 +1350,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. Three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
+ with randomness is used to allocate task to workers with available cores in a
--- End diff --
`allocate task` -> `allocate tasks`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591185
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. There are three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
+ with randomness is used, which tries to allocate task to workers with available cores in
+ roundrobin manner. The packed task assigner tries to allocate tasks to workers with the least
+ free cores, resulting in tasks assigned to few workers, which may help driver to release the
+ reserved idle workers when dynamic allocation(spark.dynamicAllocation.enabled) is enabled.
+ The balanced task assigner tries to assign tasks across workers in a balance way (allocating
+ tasks to workers with most free cores).
--- End diff --
`most` -> `the most`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84423237
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invoke the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests Whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ */
+ def offerAccepted(assigned: Boolean): Unit
--- End diff --
I remembered that in your original PR, there is a resource release method. Do you still need it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84591160
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. There are three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
+ with randomness is used, which tries to allocate task to workers with available cores in
+ roundrobin manner. The packed task assigner tries to allocate tasks to workers with the least
--- End diff --
`in roundrobin manner` -> `in a round-robin manner `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69015/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998964
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -250,24 +248,26 @@ private[spark] class TaskSchedulerImpl(
private def resourceOfferSingleTaskSet(
taskSet: TaskSetManager,
maxLocality: TaskLocality,
- shuffledOffers: Seq[WorkerOffer],
- availableCpus: Array[Int],
- tasks: IndexedSeq[ArrayBuffer[TaskDescription]]) : Boolean = {
+ taskAssigner: TaskAssigner) : Boolean = {
var launchedTask = false
- for (i <- 0 until shuffledOffers.size) {
- val execId = shuffledOffers(i).executorId
- val host = shuffledOffers(i).host
- if (availableCpus(i) >= CPUS_PER_TASK) {
+ taskAssigner.init()
+ while(taskAssigner.hasNext) {
--- End diff --
nit: space after `while`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67224/consoleFull)** for PR 15541 at commit [`e81b279`](https://github.com/apache/spark/commit/e81b2796f7b0c0c4a52d7345f14e0ab4e0e143b3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4191/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67418/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #97708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97708/consoleFull)** for PR 15541 at commit [`ada2a45`](https://github.com/apache/spark/commit/ada2a45b43b9722bae99cf578b544195e69a1625).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998540
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
--- End diff --
As you will put offers into the `PriorityQueue`, is it still necessary to do shuffling?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590978
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ */
+ def offerAccepted(assigned: Boolean): Unit
--- End diff --
`assigned ` is a very critical parameter. Could you explain it in the function description (the code comment)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67863/consoleFull)** for PR 15541 at commit [`a820e96`](https://github.com/apache/spark/commit/a820e96284f1d9108ef62cd3ef55171ebd47e08f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590835
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
--- End diff --
Here, the implementation of this function does not guarantee the offer assignment is final or not. Thus, maybe we can change the description to `The currently assigned offers`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #72506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72506/consoleFull)** for PR 15541 at commit [`ada2a45`](https://github.com/apache/spark/commit/ada2a45b43b9722bae99cf578b544195e69a1625).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67157/consoleFull)** for PR 15541 at commit [`75cdd1a`](https://github.com/apache/spark/commit/75cdd1a77a227fa492a09e93794d4ea7be8a020f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class OfferState(val workOffer: WorkerOffer) `
* `class RoundRobinAssigner extends TaskAssigner `
* `class BalancedAssigner extends TaskAssigner `
* `class PackedAssigner extends TaskAssigner `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84422357
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invoke the init()
--- End diff --
`invoke` -> `invokes`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83996326
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
--- End diff --
nit: `idx` => `currentOfferIndex`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83996865
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
--- End diff --
I like the explanation here because it tells the implications of using this assignment strategy. Can you also comment that for other impls ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67924/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15541
@rxin Thanks for the feedback regarding the TaskAssigner API. The current API is designed based on the current logic of TaskSchedulerImp, where the scheduler takes many rounds to assign the tasks for each task set. I have not figured out a better way yet. Any suggestions are welcome.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67418/consoleFull)** for PR 15541 at commit [`6b29002`](https://github.com/apache/spark/commit/6b29002c29fecdbe32159dd0d31f53716630de46).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590664
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
--- End diff --
`a number of` -> `multiple`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67290/consoleFull)** for PR 15541 at commit [`945d623`](https://github.com/apache/spark/commit/945d623701813d148e6c2763bc439dc4606af296).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67290/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83996692
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
--- End diff --
Whats the reason behind doing a shuffle here ? You later populate a max heap from `offer` so the ordering is lost.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r85956857
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -305,12 +307,8 @@ private[spark] class TaskSchedulerImpl(
hostsByRack.getOrElseUpdate(rack, new HashSet[String]()) += o.host
}
}
+ taskAssigner.construct(offers)
--- End diff --
The comments of the `resourceOffers` method shoud be updated. It still says `We fill each node with tasks in a round-robin manner so that tasks are balanced across the cluster.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84424034
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invoke the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests Whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ */
+ def offerAccepted(assigned: Boolean): Unit
--- End diff --
Based on the review comments, we do not need it anymore. The resource will be released in the init method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67395/consoleFull)** for PR 15541 at commit [`dd2b207`](https://github.com/apache/spark/commit/dd2b2077430bbb07047e928d20c1ad8fe940827a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #72506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72506/consoleFull)** for PR 15541 at commit [`ada2a45`](https://github.com/apache/spark/commit/ada2a45b43b9722bae99cf578b544195e69a1625).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84002236
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
--- End diff --
If it is rejected, it is not valid for this round of assignment for this specific task set anymore. Because it means it is not valid for all tasks in the task set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590948
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ */
+ def offerAccepted(assigned: Boolean): Unit
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ val className = {
+ val name = assignerMap.get(assignerName.toLowerCase())
+ name.getOrElse {
+ logWarning(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ roundrobin
+ }
+ }
+ // The className is valid. No need to catch exceptions.
+ logInfo(s"Constructing TaskAssigner as $className")
+ val assigner = Utils.classForName(className)
+ .getConstructor().newInstance().asInstanceOf[TaskAssigner]
+ assigner.withCpuPerTask(cpuPerTask = conf.getInt("spark.task.cpus", 1))
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var currentOfferIndex = 0
+
+ override def init(): Unit = {
+ currentOfferIndex = 0
+ }
+
+ override def hasNext: Boolean = currentOfferIndex < offer.size
+
+ override def next(): OfferState = {
+ offer(currentOfferIndex)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ currentOfferIndex += 1
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores. In other words, BalancedAssigner tries
+ * to distribute the task across workers in a balanced way. Potentially, it may alleviate the
+ * workers' memory pressure as less tasks running on the same workers, which also indicates that
+ * the task itself can make use of more computation resources, e.g., hyper-thread, across clusters.
+ */
+class BalancedAssigner extends TaskAssigner {
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+ private val maxHeap: PriorityQueue[OfferState] = new PriorityQueue[OfferState]()
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ maxHeap.clear()
+ offer.filter(_.coresAvailable >= cpuPerTask).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def next(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= cpuPerTask && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for jobs, which over allocate resources than they need.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
--- End diff --
`sorted ` -> `sortedOffer`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15541
LGTM except a few minor comments. Could @rxin do the final check?
Thank you for your work! @zhzhan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83996049
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
--- End diff --
Right now user is bounded to use only one of the 3 impls provided. You can make this extensible so that power users can use their own custom implementation by just put the jar is driver class path and setting the config to the desired class name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83998150
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
--- End diff --
`This assigner can potentially reduce the resource reservation for jobs which overallocate resources than they need.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #69015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69015/consoleFull)** for PR 15541 at commit [`ada2a45`](https://github.com/apache/spark/commit/ada2a45b43b9722bae99cf578b544195e69a1625).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15541
**[Test build #67428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67428/consoleFull)** for PR 15541 at commit [`a820e96`](https://github.com/apache/spark/commit/a820e96284f1d9108ef62cd3ef55171ebd47e08f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/15541
LGTM. I have a question for the legacy code, but not related to this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83996232
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var idx = 0
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ idx = 0
+ sorted = offer.filter(_.coresAvailable >= CPUS_PER_TASK).sortBy(_.coresAvailable)
+ }
+
+ override def hasNext: Boolean = idx < sorted.size
+
+ override def getNext(): OfferState = {
+ currentOffer = sorted(idx)
--- End diff --
If I just read this class, my first question is how we can ensure this will not be out of boundary? We need to leave a comment to explain this. Or add a safety check for avoiding any bug we could add in the future
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84422908
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invoke the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ protected var offer: Seq[OfferState] = _
+ protected var cpuPerTask = 1
+
+ protected def withCpuPerTask(cpuPerTask: Int): Unit = {
+ this.cpuPerTask = cpuPerTask
+ }
+
+ /** The final assigned offer returned to TaskScheduler. */
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ /** Invoked at the beginning of resource offering to construct the offer with the workoffers. */
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ /** Invoked at each round of Taskset assignment to initialize the internal structure. */
+ def init(): Unit
+
+ /**
+ * Tests Whether there is offer available to be used inside of one round of Taskset assignment.
+ * @return `true` if a subsequent call to `next` will yield an element,
+ * `false` otherwise.
+ */
+ def hasNext: Boolean
+
+ /**
+ * Produces next worker offer based on the task assignment strategy.
+ * @return the next available offer, if `hasNext` is `true`,
+ * undefined behavior otherwise.
+ */
+ def next(): OfferState
+
+ /**
+ * Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ * the assigner can decide whether the current worker is valid for the next offering.
+ */
+ def offerAccepted(assigned: Boolean): Unit
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ val className = {
+ val name = assignerMap.get(assignerName.toLowerCase())
+ name.getOrElse {
+ logWarning(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ roundrobin
+ }
+ }
+ // The className is valid. No need to catch exceptions.
+ logInfo(s"Constructing TaskAssigner as $className")
+ val assigner = Utils.classForName(className)
+ .getConstructor().newInstance().asInstanceOf[TaskAssigner]
+ assigner.withCpuPerTask(cpuPerTask = conf.getInt("spark.task.cpus", 1))
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var currentOfferIndex = 0
+
+ override def init(): Unit = {
+ currentOfferIndex = 0
+ }
+
+ override def hasNext: Boolean = currentOfferIndex < offer.size
+
+ override def next(): OfferState = {
+ offer(currentOfferIndex)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ currentOfferIndex += 1
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores. It other words, BalancedAssigner tries
--- End diff --
`It` -> `In`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590726
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
--- End diff --
`hasNext()` -> `hasNext`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84422079
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * perform task assignment given available workers, it first sorts the candidate tasksets,
--- End diff --
`perform` -> `performs`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15541
@gatorsmile I didn't see your new comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83999452
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
--- End diff --
As this assigner will try to pack as more as possible tasks into the same worker, the concern would be the increasing memory pressure on the worker. Do you have experienced such issue in your practical usage?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84424097
--- Diff: docs/configuration.md ---
@@ -1342,6 +1342,20 @@ Apart from these, the following properties are also available, and may be useful
Should be greater than or equal to 1. Number of allowed retries = this value - 1.
</td>
</tr>
+<tr>
+ <td><code>spark.scheduler.taskAssigner</code></td>
+ <td>roundrobin</td>
+ <td>
+ The strategy of how to allocate tasks among workers with free cores. There are three task
+ assigners (roundrobin, packed, and balanced) are supported currently. By default, roundrobin
+ with randomness is used, which tries to allocate task to workers with available cores in
+ roundrobin manner.The packed task assigner tries to allocate tasks to workers with the least
--- End diff --
missed space between . and `The`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84590702
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracks the current state of the workers with available cores and assigned task list. */
+private[scheduler] class OfferState(val workOffer: WorkerOffer) {
+ /** The current remaining cores that can be allocated to tasks. */
+ var coresAvailable: Int = workOffer.cores
+ /** The list of tasks that are assigned to this WorkerOffer. */
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, when TaskScheduler
+ * performs task assignment given available workers, it first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows.
+ *
+ * First, TaskScheduler invokes construct() of TaskAssigner to initialize the its internal
+ * worker states at the beginning of resource offering.
+ *
+ * Second, before each round of task assignment for a taskset, TaskScheduler invokes the init()
+ * of TaskAssigner to initialize the data structure for the round.
+ *
+ * Third, when performing real task assignment, hasNext()/getNext() is used by TaskScheduler
+ * to check the worker availability and retrieve current offering from TaskAssigner.
+ *
+ * Fourth, then offerAccepted is used by TaskScheduler to notify the TaskAssigner so that
+ * TaskAssigner can decide whether the current offer is valid or not for the next request.
+ *
+ * Fifth, After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information.
+ */
+
+private[scheduler] abstract class TaskAssigner {
--- End diff --
`abstract class` -> `sealed abstract class`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67863/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r83995415
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
--- End diff --
one line?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15541
@rxin Can you please take a look, and let me know if you have any concern?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84002353
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.PriorityQueue
+import scala.util.Random
+
+import org.apache.spark.internal.{config, Logging}
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+/** Tracking the current state of the workers with available cores and assigned task list. */
+class OfferState(val workOffer: WorkerOffer) {
+ // The current remaining cores that can be allocated to tasks.
+ var coresAvailable: Int = workOffer.cores
+ // The list of tasks that are assigned to this worker.
+ val tasks = new ArrayBuffer[TaskDescription](coresAvailable)
+}
+
+/**
+ * TaskAssigner is the base class for all task assigner implementations, and can be
+ * extended to implement different task scheduling algorithms.
+ * Together with [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]], TaskAssigner
+ * is used to assign tasks to workers with available cores. Internally, TaskScheduler, requested
+ * to perform task assignment given available workers, first sorts the candidate tasksets,
+ * and then for each taskset, it takes a number of rounds to request TaskAssigner for task
+ * assignment with different the locality restrictions until there is either no qualified
+ * workers or no valid tasks to be assigned.
+ *
+ * TaskAssigner is responsible to maintain the worker availability state and task assignment
+ * information. The contract between [[org.apache.spark.scheduler.TaskScheduler TaskScheduler]]
+ * and TaskAssigner is as follows. First, TaskScheduler invokes construct() of TaskAssigner to
+ * initialize the its internal worker states at the beginning of resource offering. Before each
+ * round of task assignment for a taskset, TaskScheduler invoke the init() of TaskAssigner to
+ * initialize the data structure for the round. When performing real task assignment,
+ * hasNext()/getNext() is used by TaskScheduler to check the worker availability and retrieve
+ * current offering from TaskAssigner. Then offerAccepted is used by TaskScheduler to notify
+ * the TaskAssigner so that TaskAssigner can decide whether the current offer is valid or not for
+ * the next request. After task assignment is done, TaskScheduler invokes the tasks() to
+ * retrieve all the task assignment information, and eventually, invokes reset() method so that
+ * TaskAssigner can cleanup its internal maintained resources.
+ */
+
+private[scheduler] abstract class TaskAssigner {
+ var offer: Seq[OfferState] = _
+ var CPUS_PER_TASK = 1
+
+ def withCpuPerTask(CPUS_PER_TASK: Int): Unit = {
+ this.CPUS_PER_TASK = CPUS_PER_TASK
+ }
+
+ // The final assigned offer returned to TaskScheduler.
+ final def tasks: Seq[ArrayBuffer[TaskDescription]] = offer.map(_.tasks)
+
+ // Invoked at the beginning of resource offering to construct the offer with the workoffers.
+ def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = workOffer.map(o => new OfferState(o))
+ }
+
+ // Invoked at each round of Taskset assignment to initialize the internal structure.
+ def init(): Unit
+
+ // Whether there is offer available to be used inside of one round of Taskset assignment.
+ def hasNext: Boolean
+
+ // Returned the next assigned offer based on the task assignment strategy.
+ def getNext(): OfferState
+
+ // Invoked by the TaskScheduler to indicate whether the current offer is accepted or not so that
+ // the assigner can decide whether the current worker is valid for the next offering.
+ def offerAccepted(assigned: Boolean): Unit
+
+ // Invoked at the end of resource offering to release internally maintained resources.
+ // Subclass is responsible to release its own private resources.
+ def reset(): Unit = {
+ offer = null
+ }
+}
+
+object TaskAssigner extends Logging {
+ private val roundrobin = classOf[RoundRobinAssigner].getCanonicalName
+ private val packed = classOf[PackedAssigner].getCanonicalName
+ private val balanced = classOf[BalancedAssigner].getCanonicalName
+ private val assignerMap: Map[String, String] =
+ Map("roundrobin" -> roundrobin,
+ "packed" -> packed,
+ "balanced" -> balanced)
+
+ def init(conf: SparkConf): TaskAssigner = {
+ val assignerName = conf.get(config.SPARK_SCHEDULER_TASK_ASSIGNER.key, "roundrobin")
+ .toLowerCase()
+ val className = assignerMap.getOrElse(assignerName, roundrobin)
+ val CPUS_PER_TASK = conf.getInt("spark.task.cpus", 1)
+ val assigner = try {
+ logInfo(s"Constructing an assigner as $className")
+ Utils.classForName(className).getConstructor()
+ .newInstance().asInstanceOf[TaskAssigner]
+ } catch {
+ case _: Throwable =>
+ logInfo(s"$assignerName cannot be constructed, fallback to default $roundrobin.")
+ new RoundRobinAssigner()
+ }
+ assigner.withCpuPerTask(CPUS_PER_TASK)
+ assigner
+ }
+}
+
+/**
+ * Assign the task to workers with available cores in roundrobin manner.
+ */
+class RoundRobinAssigner extends TaskAssigner {
+ private var idx = 0
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ override def init(): Unit = {
+ idx = 0
+ }
+
+ override def hasNext: Boolean = idx < offer.size
+
+ override def getNext(): OfferState = {
+ offer(idx)
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ idx += 1
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ idx = 0
+ }
+}
+
+/**
+ * Assign the task to workers with the most available cores.
+ */
+class BalancedAssigner extends TaskAssigner {
+ private var maxHeap: PriorityQueue[OfferState] = _
+ private var currentOffer: OfferState = _
+
+ override def construct(workOffer: Seq[WorkerOffer]): Unit = {
+ offer = Random.shuffle(workOffer.map(o => new OfferState(o)))
+ }
+
+ implicit val ord: Ordering[OfferState] = new Ordering[OfferState] {
+ def compare(x: OfferState, y: OfferState): Int = {
+ return Ordering[Int].compare(x.coresAvailable, y.coresAvailable)
+ }
+ }
+
+ override def init(): Unit = {
+ maxHeap = new PriorityQueue[OfferState]()
+ offer.filter(_.coresAvailable >= CPUS_PER_TASK).foreach(maxHeap.enqueue(_))
+ }
+
+ override def hasNext: Boolean = maxHeap.nonEmpty
+
+ override def getNext(): OfferState = {
+ currentOffer = maxHeap.dequeue()
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable >= CPUS_PER_TASK && assigned) {
+ maxHeap.enqueue(currentOffer)
+ }
+ }
+
+ override def reset(): Unit = {
+ super.reset
+ maxHeap = null
+ currentOffer = null
+ }
+}
+
+/**
+ * Assign the task to workers with the least available cores. In other words, PackedAssigner tries
+ * to schedule tasks to fewer workers. As a result, there will be idle workers without any tasks
+ * assigned if more than required workers are reserved. If the dynamic allocator is enabled,
+ * these idle workers will be released by driver. The released resources can then be allocated to
+ * other jobs by underling resource manager. This assigner can potentially reduce the resource
+ * reservation for a job.
+ */
+class PackedAssigner extends TaskAssigner {
+ private var sorted: Seq[OfferState] = _
+ private var idx = 0
+ private var currentOffer: OfferState = _
+
+ override def init(): Unit = {
+ idx = 0
+ sorted = offer.filter(_.coresAvailable >= CPUS_PER_TASK).sortBy(_.coresAvailable)
+ }
+
+ override def hasNext: Boolean = idx < sorted.size
+
+ override def getNext(): OfferState = {
+ currentOffer = sorted(idx)
+ currentOffer
+ }
+
+ override def offerAccepted(assigned: Boolean): Unit = {
+ if (currentOffer.coresAvailable < CPUS_PER_TASK || !assigned) {
--- End diff --
Similar to the reason above, if the offer is rejected, we have to move forward
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org