You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/09 21:49:13 UTC
[GitHub] [spark] holdenk opened a new pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
holdenk opened a new pull request #30693:
URL: https://github.com/apache/spark/pull/30693
### What changes were proposed in this pull request?
Check that the pod state is not pending or running even if there is a deletion timestamp.
### Why are the changes needed?
This can occur when the pod state and deletion timestamp are not updated by etcd in sync & we get a pod snapshot during an inconsistent view.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual testing with local version of Minikube on an overloaded computer that caused out of sync updates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30693:
URL: https://github.com/apache/spark/pull/30693#discussion_r539709918
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
##########
@@ -93,7 +93,9 @@ object ExecutorPodsSnapshot extends Logging {
(
pod.getStatus == null ||
pod.getStatus.getPhase == null ||
- pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "terminating"
+ (pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "terminating" &&
+ pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "running" &&
+ pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "pending")
Review comment:
Is this exhaustive for that cases?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742091840
**[Test build #132518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132518/testReport)** for PR 30693 at commit [`d60ec66`](https://github.com/apache/spark/commit/d60ec66e60a2ba23efcaa6497573958dfab38d27).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742221514
**[Test build #132538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132538/testReport)** for PR 30693 at commit [`e44710d`](https://github.com/apache/spark/commit/e44710d6807337e993678dbea80b79985147d8b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742235755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132538/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742109641
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37118/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743111338
**[Test build #132646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132646/testReport)** for PR 30693 at commit [`a077ec3`](https://github.com/apache/spark/commit/a077ec3f1fa740935a388a9e2282c3351fc17ca4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742114634
**[Test build #132520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132520/testReport)** for PR 30693 at commit [`d60ec66`](https://github.com/apache/spark/commit/d60ec66e60a2ba23efcaa6497573958dfab38d27).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742092008
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132518/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742120501
**[Test build #132520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132520/testReport)** for PR 30693 at commit [`d60ec66`](https://github.com/apache/spark/commit/d60ec66e60a2ba23efcaa6497573958dfab38d27).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #30693:
URL: https://github.com/apache/spark/pull/30693
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742254964
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37140/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742086147
**[Test build #132518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132518/testReport)** for PR 30693 at commit [`d60ec66`](https://github.com/apache/spark/commit/d60ec66e60a2ba23efcaa6497573958dfab38d27).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743465098
Now that we've fixed the R issue we've got a successful run :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
holdenk commented on a change in pull request #30693:
URL: https://github.com/apache/spark/pull/30693#discussion_r539824745
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
##########
@@ -93,7 +93,9 @@ object ExecutorPodsSnapshot extends Logging {
(
pod.getStatus == null ||
pod.getStatus.getPhase == null ||
- pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "terminating"
+ (pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "terminating" &&
+ pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "running" &&
+ pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "pending")
Review comment:
No, but these are the cases which if it's deleted timestamp it might not be deleted. For example if it's in a failed state it's ok if its detected as deleted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742725193
**[Test build #132585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132585/testReport)** for PR 30693 at commit [`a077ec3`](https://github.com/apache/spark/commit/a077ec3f1fa740935a388a9e2282c3351fc17ca4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742156543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37122/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743138257
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37249/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742221514
**[Test build #132538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132538/testReport)** for PR 30693 at commit [`e44710d`](https://github.com/apache/spark/commit/e44710d6807337e993678dbea80b79985147d8b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742114634
**[Test build #132520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132520/testReport)** for PR 30693 at commit [`d60ec66`](https://github.com/apache/spark/commit/d60ec66e60a2ba23efcaa6497573958dfab38d27).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742092008
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742235755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132538/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742237440
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37140/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743122367
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132646/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742113829
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742319930
Ur, it seems to have a bug. One of the three failures has the following log.
```
Waiting for dynamic alloc
20/12/10 05:17:42 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 172.17.0.5:40669 in memory (size: 5.4 KiB, free: 593.9 MiB)
20/12/10 05:17:42 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:43 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:44 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:46 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:47 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:48 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:49 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:50 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:51 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:52 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:53 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:54 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:55 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:56 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:57 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:58 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:17:59 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:18:00 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:18:01 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:18:02 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:18:03 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:18:04 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
20/12/10 05:18:05 INFO ExecutorPodsAllocator: Deleting 3 excess pod requests (5,4,3).
" did not contain "Finished waiting, stopping Spark" The application did not complete, driver log did not contain str Finished waiting, stopping Spark. (KubernetesSuite.scala:403)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743104330
**[Test build #132646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132646/testReport)** for PR 30693 at commit [`a077ec3`](https://github.com/apache/spark/commit/a077ec3f1fa740935a388a9e2282c3351fc17ca4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742247370
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37140/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742119047
SPARK-33727 is filed for the K8s IT failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743101706
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743125902
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37249/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742749156
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742225121
**[Test build #132538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132538/testReport)** for PR 30693 at commit [`e44710d`](https://github.com/apache/spark/commit/e44710d6807337e993678dbea80b79985147d8b5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742732405
**[Test build #132585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132585/testReport)** for PR 30693 at commit [`a077ec3`](https://github.com/apache/spark/commit/a077ec3f1fa740935a388a9e2282c3351fc17ca4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743148826
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37249/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742156543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37122/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742753902
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742108534
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37118/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743104330
**[Test build #132646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132646/testReport)** for PR 30693 at commit [`a077ec3`](https://github.com/apache/spark/commit/a077ec3f1fa740935a388a9e2282c3351fc17ca4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742725193
**[Test build #132585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132585/testReport)** for PR 30693 at commit [`a077ec3`](https://github.com/apache/spark/commit/a077ec3f1fa740935a388a9e2282c3351fc17ca4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742753902
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742104620
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37118/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743122367
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132646/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742254964
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37140/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-743148826
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37249/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30693: [SPARK-33716][K8S] Fix potential race condition during pod termination
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30693:
URL: https://github.com/apache/spark/pull/30693#issuecomment-742086147
**[Test build #132518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132518/testReport)** for PR 30693 at commit [`d60ec66`](https://github.com/apache/spark/commit/d60ec66e60a2ba23efcaa6497573958dfab38d27).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org