You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/18 17:43:22 UTC
[GitHub] [spark] holdenk opened a new pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
holdenk opened a new pull request #34650:
URL: https://github.com/apache/spark/pull/34650
### What changes were proposed in this pull request?
Keep track of and communicate to the listener bus how long we are waiting for execs to be allocated from the underlying cluster manager.
### Why are the changes needed?
Sometimes the cluster manager may choke or otherwise not be able to allocate resources and we don't have a good way of detecting this situation making it difficult for the user to debug and tell apart from Spark not scaling up correctly.
### Does this PR introduce _any_ user-facing change?
New field in the listener bus message for when a executor is allocated.
### How was this patch tested?
New unit test in the listener suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973229794
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973194334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973281561
**[Test build #145403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145403/testReport)** for PR 34650 at commit [`33d3918`](https://github.com/apache/spark/commit/33d3918edf9ab6ec7430144dfe17f45791bfa087).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973156390
**[Test build #145402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145402/testReport)** for PR 34650 at commit [`a1ce1fd`](https://github.com/apache/spark/commit/a1ce1fd4bdea0b6755d65168feaf997b33d309ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk closed pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
holdenk closed pull request #34650:
URL: https://github.com/apache/spark/pull/34650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973144066
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-992997139
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-993008960
**[Test build #146146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146146/testReport)** for PR 34650 at commit [`6371b7a`](https://github.com/apache/spark/commit/6371b7a32fdd2a3fd5e4c4ed0df8b5f33393e8c0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973291905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145403/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973162274
**[Test build #145403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145403/testReport)** for PR 34650 at commit [`33d3918`](https://github.com/apache/spark/commit/33d3918edf9ab6ec7430144dfe17f45791bfa087).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-993027902
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146146/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-993027902
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146146/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
holdenk commented on a change in pull request #34650:
URL: https://github.com/apache/spark/pull/34650#discussion_r768111424
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -776,16 +801,55 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
val resourceProfileToNumExecutors = resourceProfileIdToNumExecutors.map { case (rpid, num) =>
(scheduler.sc.resourceProfileManager.resourceProfileFromId(rpid), num)
}
+ val oldResourceProfileToNumExecutors = requestedTotalExecutorsPerResourceProfile.map {
+ case (rp, num) =>
+ (rp.id, num)
+ }.toMap
Review comment:
Good point
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorInfo.scala
##########
@@ -72,6 +82,6 @@ class ExecutorInfo(
override def hashCode(): Int = {
val state = Seq(executorHost, totalCores, logUrlMap, attributes, resourcesInfo,
resourceProfileId)
- state.map(_.hashCode()).foldLeft(0)((a, b) => 31 * a + b)
+ state.filter(_ != null).map(_.hashCode()).foldLeft(0)((a, b) => 31 * a + b)
Review comment:
This is semi-related, it's changed so that we can compare executorinfo objects during testing which may have components that are null.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -257,9 +263,27 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
.resourceProfileFromId(resourceProfileId).getNumSlotsPerAddress(rName, conf)
(info.name, new ExecutorResourceInfo(info.name, info.addresses, numParts))
}
+ // If we've requested the executor figure out when we did.
+ val reqTs: Option[Long] = {
+ execRequestTimes.get(resourceProfileId).flatMap {
Review comment:
Yeah good point
##########
File path: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala
##########
@@ -1744,7 +1744,8 @@ class ExecutorAllocationManagerSuite extends SparkFunSuite {
id: String,
rp: ResourceProfile): Unit = {
val cores = rp.getExecutorCores.getOrElse(1)
- val execInfo = new ExecutorInfo("host1", cores, Map.empty, Map.empty, Map.empty, rp.id)
+ val execInfo = new ExecutorInfo("host1", cores, Map.empty, Map.empty, Map.empty, rp.id,
+ None, None)
Review comment:
That's a good point, let me back out the unrelated test changes (and that will also ensure the constructor change captured what was desired).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34650:
URL: https://github.com/apache/spark/pull/34650#discussion_r753420475
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -257,9 +263,27 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
.resourceProfileFromId(resourceProfileId).getNumSlotsPerAddress(rName, conf)
(info.name, new ExecutorResourceInfo(info.name, info.addresses, numParts))
}
+ // If we've requested the executor figure out when we did.
+ val reqTs: Option[Long] = {
+ execRequestTimes.get(resourceProfileId).flatMap {
Review comment:
`synchronized` ?
##########
File path: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala
##########
@@ -1744,7 +1744,8 @@ class ExecutorAllocationManagerSuite extends SparkFunSuite {
id: String,
rp: ResourceProfile): Unit = {
val cores = rp.getExecutorCores.getOrElse(1)
- val execInfo = new ExecutorInfo("host1", cores, Map.empty, Map.empty, Map.empty, rp.id)
+ val execInfo = new ExecutorInfo("host1", cores, Map.empty, Map.empty, Map.empty, rp.id,
+ None, None)
Review comment:
For the current PR state, we dont need changes to ExecutorAllocationManagerSuite, JsonProtocolSuite or ExecutorMonitorSuite given the constructor ?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -776,16 +801,55 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
val resourceProfileToNumExecutors = resourceProfileIdToNumExecutors.map { case (rpid, num) =>
(scheduler.sc.resourceProfileManager.resourceProfileFromId(rpid), num)
}
+ val oldResourceProfileToNumExecutors = requestedTotalExecutorsPerResourceProfile.map {
+ case (rp, num) =>
+ (rp.id, num)
+ }.toMap
val response = synchronized {
this.requestedTotalExecutorsPerResourceProfile.clear()
this.requestedTotalExecutorsPerResourceProfile ++= resourceProfileToNumExecutors
this.numLocalityAwareTasksPerResourceProfileId = numLocalityAwareTasksPerResourceProfileId
this.rpHostToLocalTaskCount = hostToLocalTaskCount
+ updateExecRequestTimes(oldResourceProfileToNumExecutors, resourceProfileIdToNumExecutors)
doRequestTotalExecutors(requestedTotalExecutorsPerResourceProfile.toMap)
}
defaultAskTimeout.awaitResult(response)
}
+ private def updateExecRequestTimes(oldProfile: Map[Int, Int], newProfile: Map[Int, Int]): Unit = {
+ newProfile.map {
+ case (k, v) =>
+ val delta = v - oldProfile.getOrElse(k, 0)
+ if (delta != 0) {
+ updateExecRequestTime(k, delta)
+ }
+ }
+ }
+
+ private def updateExecRequestTime(profileId: Int, delta: Int) = {
+ val times = execRequestTimes.getOrElseUpdate(profileId, Queue[(Int, Long)]())
+ if (delta > 0) {
+ // Add the request to the end, constant time op
+ times += ((delta, System.currentTimeMillis()))
+ } else if (delta < 0) {
+ // Consume as if |delta| had been allocated
+ var c = -delta
+ try {
+ while (c > 0) {
+ val h = times.dequeue
+ if (h._1 > c) {
+ // Prepend updated first req to times, constant time op
+ ((h._1 - c, h._2)) +=: times
+ } else {
+ c = c - h._1
+ }
+ }
+ } catch {
+ case e: Exception => // Ignore
Review comment:
avoid this with `!times.isEmpty` in while loop ?
How/Why would it happen btw ?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorInfo.scala
##########
@@ -72,6 +82,6 @@ class ExecutorInfo(
override def hashCode(): Int = {
val state = Seq(executorHost, totalCores, logUrlMap, attributes, resourcesInfo,
resourceProfileId)
- state.map(_.hashCode()).foldLeft(0)((a, b) => 31 * a + b)
+ state.filter(_ != null).map(_.hashCode()).foldLeft(0)((a, b) => 31 * a + b)
Review comment:
This should not need to change, right ? Or unrelated bugfix ?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -776,16 +801,55 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
val resourceProfileToNumExecutors = resourceProfileIdToNumExecutors.map { case (rpid, num) =>
(scheduler.sc.resourceProfileManager.resourceProfileFromId(rpid), num)
}
+ val oldResourceProfileToNumExecutors = requestedTotalExecutorsPerResourceProfile.map {
+ case (rp, num) =>
+ (rp.id, num)
+ }.toMap
Review comment:
Move this into the `synchronized` block ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-992958654
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973241342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973108419
**[Test build #145399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145399/testReport)** for PR 34650 at commit [`cee2b17`](https://github.com/apache/spark/commit/cee2b17d27b5c5a750b2d629e4b44f279ded781a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973108419
**[Test build #145399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145399/testReport)** for PR 34650 at commit [`cee2b17`](https://github.com/apache/spark/commit/cee2b17d27b5c5a750b2d629e4b44f279ded781a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973194329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973193881
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973118297
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973241342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973244532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-992928178
**[Test build #146146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146146/testReport)** for PR 34650 at commit [`6371b7a`](https://github.com/apache/spark/commit/6371b7a32fdd2a3fd5e4c4ed0df8b5f33393e8c0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
holdenk commented on a change in pull request #34650:
URL: https://github.com/apache/spark/pull/34650#discussion_r768106412
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -776,16 +801,55 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
val resourceProfileToNumExecutors = resourceProfileIdToNumExecutors.map { case (rpid, num) =>
(scheduler.sc.resourceProfileManager.resourceProfileFromId(rpid), num)
}
+ val oldResourceProfileToNumExecutors = requestedTotalExecutorsPerResourceProfile.map {
+ case (rp, num) =>
+ (rp.id, num)
+ }.toMap
val response = synchronized {
this.requestedTotalExecutorsPerResourceProfile.clear()
this.requestedTotalExecutorsPerResourceProfile ++= resourceProfileToNumExecutors
this.numLocalityAwareTasksPerResourceProfileId = numLocalityAwareTasksPerResourceProfileId
this.rpHostToLocalTaskCount = hostToLocalTaskCount
+ updateExecRequestTimes(oldResourceProfileToNumExecutors, resourceProfileIdToNumExecutors)
doRequestTotalExecutors(requestedTotalExecutorsPerResourceProfile.toMap)
}
defaultAskTimeout.awaitResult(response)
}
+ private def updateExecRequestTimes(oldProfile: Map[Int, Int], newProfile: Map[Int, Int]): Unit = {
+ newProfile.map {
+ case (k, v) =>
+ val delta = v - oldProfile.getOrElse(k, 0)
+ if (delta != 0) {
+ updateExecRequestTime(k, delta)
+ }
+ }
+ }
+
+ private def updateExecRequestTime(profileId: Int, delta: Int) = {
+ val times = execRequestTimes.getOrElseUpdate(profileId, Queue[(Int, Long)]())
+ if (delta > 0) {
+ // Add the request to the end, constant time op
+ times += ((delta, System.currentTimeMillis()))
+ } else if (delta < 0) {
+ // Consume as if |delta| had been allocated
+ var c = -delta
+ try {
+ while (c > 0) {
+ val h = times.dequeue
+ if (h._1 > c) {
+ // Prepend updated first req to times, constant time op
+ ((h._1 - c, h._2)) +=: times
+ } else {
+ c = c - h._1
+ }
+ }
+ } catch {
+ case e: Exception => // Ignore
Review comment:
If someone choses something other than Spark to manage the scale-up/scale-down of resources, or a straggler shows up that we assumed wasn't going to show up so we duplicated the request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-992928178
**[Test build #146146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146146/testReport)** for PR 34650 at commit [`6371b7a`](https://github.com/apache/spark/commit/6371b7a32fdd2a3fd5e4c4ed0df8b5f33393e8c0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-992997139
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973118220
**[Test build #145399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145399/testReport)** for PR 34650 at commit [`cee2b17`](https://github.com/apache/spark/commit/cee2b17d27b5c5a750b2d629e4b44f279ded781a).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973166396
**[Test build #145402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145402/testReport)** for PR 34650 at commit [`a1ce1fd`](https://github.com/apache/spark/commit/a1ce1fd4bdea0b6755d65168feaf997b33d309ac).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973118297
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973176342
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973244532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973156390
**[Test build #145402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145402/testReport)** for PR 34650 at commit [`a1ce1fd`](https://github.com/apache/spark/commit/a1ce1fd4bdea0b6755d65168feaf997b33d309ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973198548
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973244493
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973291905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145403/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-973162274
**[Test build #145403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145403/testReport)** for PR 34650 at commit [`33d3918`](https://github.com/apache/spark/commit/33d3918edf9ab6ec7430144dfe17f45791bfa087).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34650: [WIP][SPARK-36664][CORE] Log time waiting for cluster resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34650:
URL: https://github.com/apache/spark/pull/34650#issuecomment-992994776
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org