You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/25 16:54:51 UTC
[GitHub] [spark] onursatici opened a new pull request #27695:
[SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
onursatici opened a new pull request #27695: [SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
`spark.driver.cores` configuration is used to set the amount of parallelism in kubernetes cluster mode drivers. Previously the amount of parallelism in the drivers were the number of cores in the host when running on java 8 or older, or the maximum of driver containers resource requests and limits when running on java 9 or newer. This will enable users to specify `spark.driver.cores` to set parallelism, and specify `spark.kubernetes.driver.requests.cores` to limit the resource requests of the driver container, effectively decoupling the two
### Why are the changes needed?
Drivers submitted in kubernetes cluster mode set the parallelism of various components like `RpcEnv`, `MemoryManager`, `BlockManager` from inferring the number of available cores by calling `Runtime.getRuntime().availableProcessors()`. By using this, spark applications running on java 8 or older incorrectly get the total number of cores in the host, [ignoring the cgroup limits set by kubernetes](https://bugs.openjdk.java.net/browse/JDK-6515172). Java 9 and newer runtimes do not have this problem.
Orthogonal to this, it is currently not possible to decouple resource limits on the driver container with the amount of parallelism of the various network and memory components listed above.
### Does this PR introduce any user-facing change?
Yes. Previously the amount of parallelism in kubernetes cluster mode submitted drivers were the number of cores in the host when running on java 8 or older, or the maximum of driver containers resource requests and limits when running on java 9 or newer. Now the value of `spark.driver.cores` is used.
### How was this patch tested?
happy to add tests if my proposal looks reasonable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604051121
**[Test build #120369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120369/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613799273
@onursatici . I'm still not sure about this approach.
- First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use this kind of old JDKs (JDK 8u120 or older). We have reached the milestone of JDK11 at Apache Spark 3.0.0.
- Second, even in old JDKs, this original YARN code is made by the following PR for `Netty`. In K8s environment, the total number of available cores is the same with the requested amount.
- https://github.com/apache/spark/pull/21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)
The last possibility which I can guess is that you want to have bigger parallelism on the small container. Is that your case? Could you give us more concrete example where this PR is beneficial?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-590961257
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604262384
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25107/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604666017
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604666024
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120434/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591136331
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604004318
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25078/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-603983643
**[Test build #120369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120369/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604014618
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25078/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-603983643
**[Test build #120369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120369/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604051616
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120369/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613799273
@onursatici . I'm still not sure about this approach.
- First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use this kind of old JDKs (JDK 8u120 or older). We have reached the milestone of JDK11 at Apache Spark 3.0.0.
- Second, even in old JDKs, this original YARN code is made by the following PR for `Netty`. In K8s environment, the total number of available cores is the same with the requested amount.
- https://github.com/apache/spark/pull/21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)
The last possibility which I can guess is that you want to have bigger parallelism on the small container. Is that your case? Could you give us more concrete example which this PR is beneficial?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591136306
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23681/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-594823234
@onursatici . One thing I'm thinking is the deprecation of `8u120` and the older versions at 3.0.0. Until now, `3.0.0-preview2` gave some early warning (not official deprecation yet) for `8u91` and the older versions like the following.
- https://spark.apache.org/docs/3.0.0-preview2/
> Java 8 prior to version 8u92 support is deprecated as of Spark 3.0.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695: [SPARK-30949][K8S]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-590961257
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591188300
**[Test build #118933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118933/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] onursatici commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
onursatici commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-603876593
@jiangxb1987 do you recommend to do that in this PR? I think changing the stand-alone driver core count behaviour would broaden the scope of this PR such that it might warrant a separate discussion.
Does this change make sense for k8s? Any blockers @dongjoon-hyun?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604240942
**[Test build #120398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120398/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-600222788
Hi, @jiangxb1987 . What do you mean by **this**? This PR or this old JDK bug?
> Since this also affects Standalone cluster, I'd suggest we only exclude Mesos backend in the case match.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604014608
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613799273
@onursatici . I'm still not sure about this approach.
- First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use this kind of old JDKs (JDK 8u120 or older). We have reached the milestone of JDK11 at Apache Spark 3.0.0.
- Second, even in old JDKs, this original YARN code is made by the following PR for `Netty`. In K8s environment, the available number of core is the same with the requested amount.
- https://github.com/apache/spark/pull/21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)
So, do you want to have bigger parallelism on the small container?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591197574
Ping @holdenk since the flakiness of `Test basic decommissioning` is observed again.
```
- Test basic decommissioning *** FAILED ***
The code passed to eventually never returned normally. Attempted 121 times over 2.01174358735 minutes. Last failure message: "++ id -u
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-603980696
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604665150
**[Test build #120434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120434/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604051604
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604239855
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604618955
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25142/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-603980359
It's fine if you want to focus on the k8s behaviour here, I can submit another PR to fix the Standalone backend after this is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591098027
**[Test build #118933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118933/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604265510
**[Test build #120398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120398/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-594823234
@onursatici . One thing I'm thinking is the deprecation of `8u120` and the older at 3.0.0. Until now, `3.0.0-preview2` gave some early warning (not official deprecation yet) for `8u92` like the following.
- https://spark.apache.org/docs/3.0.0-preview2/
> Java 8 prior to version 8u92 support is deprecated as of Spark 3.0.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591098027
**[Test build #118933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118933/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613799273
@onursatici . I'm still not sure about this approach.
- First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use old JDKs (JDK 8u120 or older).
- Second, even in old JDKs, this original YARN code is made by the following PR for `Netty`. In K8s environment, the available number of core is the same with the requested amount.
- https://github.com/apache/spark/pull/21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)
So, do you want to have bigger parallelism on the small container?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591136341
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23681/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591189167
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591136331
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604255288
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25107/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
holdenk commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-592826519
In my experience the K8s tests have all been flaky but I’ll dig into them & decom as well this coming week
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604014618
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25078/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604051616
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120369/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613799273
@onursatici . I'm still not sure about this approach.
- First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use this kind of old JDKs (JDK 8u120 or older). We have reached the milestone of JDK11 at Apache Spark 3.0.0.
- Second, even in old JDKs, this original YARN code is made by the following PR for `Netty`. In K8s environment, the total number of available cores is the same with the requested amount.
- https://github.com/apache/spark/pull/21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)
So, do you want to have bigger parallelism on the small container?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604575780
**[Test build #120434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120434/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604603358
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25142/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591136341
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23681/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591118896
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23681/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-600222788
Hi, @jiangxb1987 . What do you mean by **this**? This PR or this old JDK bug?
> Since **this** also affects Standalone cluster, I'd suggest we only exclude Mesos backend in the case match.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604619016
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25142/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-592820588
Gentle ping, @onursatici .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604262384
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25107/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-615905091
Gentle ping, @onursatici .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604014579
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25078/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604265748
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604051604
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613799273
@onursatici . I'm still not sure about this approach.
- First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use this kind of old JDKs (JDK 8u120 or older). We have reached the milestone of JDK11 at Apache Spark 3.0.0.
- Second, even in old JDKs, this original YARN code is made by the following PR for `Netty`. In K8s environment, the total number of available cores is the same with the requested amount.
- https://github.com/apache/spark/pull/21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)
The last possibility which I can guess is that you want to have bigger parallelism on the small container. Is that your case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604014608
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604619001
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591189159
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613799273
@onursatici . I'm still not sure about this approach.
- First, K8s environment is different from on-prem environment. I don't think the user of Apache Spark 3.1.0 will use this kind of old JDKs (JDK 8u120 or older). We have reached the milestone of JDK11 at Apache Spark 3.0.0.
- Second, this original YARN code is made by the following PR for `Netty`. In K8s environment, the total number of available cores is the same with the requested amount.
- https://github.com/apache/spark/pull/21885 ([SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations)
The last possibility which I can guess is that you want to have bigger parallelism on the small container. Is that your case? Could you give us more concrete example where this PR is beneficial?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604666017
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-593723047
IIUC this issue also affects Standalone cluster mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604572117
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604666024
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120434/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-614910556
How do you think about the above comment, @onursatici ? I'm wondering your opinion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] onursatici commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
onursatici commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-594652279
@dongjoon-hyun, what are the next steps? Does it look fine from your perspective?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591189167
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-600214613
Since this also affects Standalone cluster, I'd suggest we only exclude Mesos backend in the case match.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-590960508
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] onursatici commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
onursatici commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-613415263
@dongjoon-hyun do you mind taking a look at this? I have revised the PR description
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604240942
**[Test build #120398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120398/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604265755
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120398/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604575780
**[Test build #120434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120434/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffb1111b230cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591189159
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27695: [SPARK-30949][K8S] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-591095081
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604265755
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120398/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604619016
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25142/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604262379
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-600223798
I mean the JDK bug mentioned in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604619001
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE]
decouple requests and parallelism on kubernetes drivers
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604262367
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25107/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604262379
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27695:
[SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes
drivers
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
URL: https://github.com/apache/spark/pull/27695#issuecomment-604265748
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org