You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/24 09:21:32 UTC
[GitHub] [spark] slothspot opened a new pull request #34095: [WIP][SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
slothspot opened a new pull request #34095:
URL: https://github.com/apache/spark/pull/34095
### What changes were proposed in this pull request?
Don't open watch when not needed
### Why are the changes needed?
In spark-submit, we currently open a pod watch for any spark submission. If WAIT_FOR_APP_COMPLETION is false, we then immediately ignore the result of the watcher and break out of the watcher.
When submitting spark applications at scale, this is a source of operational pain, since opening the watch relies on opening a websocket, which tends to run into subtle networking issues around negotiating the websocket connection.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Standard tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926715550
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48116/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926749721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48116/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926701646
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143604/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] slothspot commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
slothspot commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-927602354
@dongjoon-hyun my JIRA id is `dmytrom`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34095: [WIP][SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926479829
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34095: [WIP][SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926479829
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34095: [WIP][SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926677874
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926701646
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143604/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926679754
**[Test build #143604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143604/testReport)** for PR 34095 at commit [`da1e11a`](https://github.com/apache/spark/commit/da1e11a3c1994db55971c59c9ab26a80fac43339).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926749721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48116/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] slothspot commented on pull request #34095: [WIP][SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
slothspot commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926484602
@dongjoon-hyun what do you think about such approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926824286
What is your Apache Jira ID, @slothspot ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #34095:
URL: https://github.com/apache/spark/pull/34095
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926690711
**[Test build #143604 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143604/testReport)** for PR 34095 at commit [`da1e11a`](https://github.com/apache/spark/commit/da1e11a3c1994db55971c59c9ab26a80fac43339).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926749615
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48116/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34095: [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34095:
URL: https://github.com/apache/spark/pull/34095#issuecomment-926679754
**[Test build #143604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143604/testReport)** for PR 34095 at commit [`da1e11a`](https://github.com/apache/spark/commit/da1e11a3c1994db55971c59c9ab26a80fac43339).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org