You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/02 21:17:02 UTC
[GitHub] [spark] attilapiros opened a new pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
attilapiros opened a new pull request #31445:
URL: https://github.com/apache/spark/pull/31445
### What changes were proposed in this pull request?
Fixing identification of timed-out pending pod requests as excess requests to delete when the excess is higher than the newly created timed out requests and there is some non-timed out newly created requests too.
### Why are the changes needed?
After https://github.com/apache/spark/pull/29981 only timed out newly created requests and timed out pending requests are taken as excess request.
But there is small bug when the excess is higher than the newly created timed out requests and there is some non-timed out newly created requests as well. Because all the newly created requests are counted as excess request when items are chosen from the timed out pod pending requests.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
There is new unit test added: `SPARK-34334: correctly identify timed out pending pod requests as excess`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775941686
**[Test build #135074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135074/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775950714
**[Test build #135074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135074/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-776151962
Thanks @holdenk!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773412608
**[Test build #134882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134882/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772097275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39387/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #31445:
URL: https://github.com/apache/spark/pull/31445#discussion_r570299313
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
##########
@@ -223,14 +223,15 @@ private[spark] class ExecutorPodsAllocator(
if (knownPodCount > targetNum) {
val excess = knownPodCount - targetNum
+ val newlyCreatedToDelete = newlyCreatedExecutorsForRpId
+ .filter { case (_, (_, createTime)) =>
+ currentTime - createTime > executorIdleTimeout
+ }.keys.take(excess).toList
val knownPendingToDelete = currentPendingExecutors
.filter(x => isExecutorIdleTimedOut(x._2, currentTime))
+ .take(excess - newlyCreatedToDelete.size)
.map { case (id, _) => id }
- .take(excess - newlyCreatedExecutorsForRpId.size)
Review comment:
Before this PR this line caused the error in the new unit test.
As some `newlyCreatedExecutorsForRpId` might be not timed out and in place of those we should choose timed out pending executor pod requests.
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
##########
@@ -223,14 +223,15 @@ private[spark] class ExecutorPodsAllocator(
if (knownPodCount > targetNum) {
val excess = knownPodCount - targetNum
+ val newlyCreatedToDelete = newlyCreatedExecutorsForRpId
+ .filter { case (_, (_, createTime)) =>
+ currentTime - createTime > executorIdleTimeout
+ }.keys.take(excess).toList
val knownPendingToDelete = currentPendingExecutors
.filter(x => isExecutorIdleTimedOut(x._2, currentTime))
+ .take(excess - newlyCreatedToDelete.size)
.map { case (id, _) => id }
- .take(excess - newlyCreatedExecutorsForRpId.size)
Review comment:
Before this PR this line caused the error in the new unit test.
As some `newlyCreatedExecutorsForRpId` might be not timed out and in place of those we should choose timed out pending executor pod requests.
So the `.size` must be called on the filtered `newlyCreatedExecutors` which only contains the timed out elements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773684367
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39479/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773613284
**[Test build #134892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134892/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773632693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134892/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775555758
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135049/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772153751
cc @dongjoon-hyun @HyukjinKwon @holdenk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773492547
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39466/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775536734
**[Test build #135049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775966357
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39656/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775580867
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39632/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772035874
**[Test build #134799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134799/testReport)** for PR 31445 at commit [`cdccf7e`](https://github.com/apache/spark/commit/cdccf7e863544d170bfa54b8b29ad143c7244716).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775536734
**[Test build #135049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773499545
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39466/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773499545
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39466/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775509482
LGTM pending CI.
Jenkins failure is unrelated but to be safe, Jenkins retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773632693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134892/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772035874
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773417127
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772035874
**[Test build #134799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134799/testReport)** for PR 31445 at commit [`cdccf7e`](https://github.com/apache/spark/commit/cdccf7e863544d170bfa54b8b29ad143c7244716).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775941686
**[Test build #135074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135074/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772070950
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-776134382
Merged to the current dev & 3.1 branches.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775555758
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135049/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773619383
**[Test build #134892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134892/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773404117
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772070950
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134799/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772035874
**[Test build #134799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134799/testReport)** for PR 31445 at commit [`cdccf7e`](https://github.com/apache/spark/commit/cdccf7e863544d170bfa54b8b29ad143c7244716).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772070950
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134799/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773596593
jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #31445:
URL: https://github.com/apache/spark/pull/31445
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775926667
jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773404117
**[Test build #134882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134882/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775580867
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39632/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775969691
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39656/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772156485
Thanks for working on this :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773404117
**[Test build #134882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134882/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775568472
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39632/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775975085
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775552827
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39632/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773684367
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39479/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772070950
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772046076
**[Test build #134799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134799/testReport)** for PR 31445 at commit [`cdccf7e`](https://github.com/apache/spark/commit/cdccf7e863544d170bfa54b8b29ad143c7244716).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-772097275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39387/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #31445:
URL: https://github.com/apache/spark/pull/31445#discussion_r570299313
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
##########
@@ -223,14 +223,15 @@ private[spark] class ExecutorPodsAllocator(
if (knownPodCount > targetNum) {
val excess = knownPodCount - targetNum
+ val newlyCreatedToDelete = newlyCreatedExecutorsForRpId
+ .filter { case (_, (_, createTime)) =>
+ currentTime - createTime > executorIdleTimeout
+ }.keys.take(excess).toList
val knownPendingToDelete = currentPendingExecutors
.filter(x => isExecutorIdleTimedOut(x._2, currentTime))
+ .take(excess - newlyCreatedToDelete.size)
.map { case (id, _) => id }
- .take(excess - newlyCreatedExecutorsForRpId.size)
Review comment:
Before this PR this line caused the error in the new unit test.
As some `newlyCreatedExecutorsForRpId` might be not timed out and in place of those we should choose timed out pending executor pod requests.
So the `.size` must be called on the filtered `newlyCreatedExecutors` which only contains the timed out elements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775541886
**[Test build #135049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-776131110
Out of disk space error on the Jenkins machine is unrelated (`failed to copy files: failed to copy directory: Error processing tar file(exit status 1): write /hadoop-client-api-3.2.2.jar: no space left on device`), I'm going to merge this bug fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773613284
**[Test build #134892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134892/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #31445:
URL: https://github.com/apache/spark/pull/31445#discussion_r570299313
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
##########
@@ -223,14 +223,15 @@ private[spark] class ExecutorPodsAllocator(
if (knownPodCount > targetNum) {
val excess = knownPodCount - targetNum
+ val newlyCreatedToDelete = newlyCreatedExecutorsForRpId
+ .filter { case (_, (_, createTime)) =>
+ currentTime - createTime > executorIdleTimeout
+ }.keys.take(excess).toList
val knownPendingToDelete = currentPendingExecutors
.filter(x => isExecutorIdleTimedOut(x._2, currentTime))
+ .take(excess - newlyCreatedToDelete.size)
.map { case (id, _) => id }
- .take(excess - newlyCreatedExecutorsForRpId.size)
Review comment:
Before this PR this line caused the error in the new unit test.
As some `newlyCreatedExecutorsForRpId` might be not timed out and in place of those we should choose timed out pending executor pod requests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773417127
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773404117
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773596593
jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773430805
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39466/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773417127
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134882/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-773417127
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134882/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31445:
URL: https://github.com/apache/spark/pull/31445#issuecomment-775975085
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org