You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liyinan926 <gi...@git.apache.org> on 2018/01/31 21:34:30 UTC
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
GitHub user liyinan926 opened a pull request:
https://github.com/apache/spark/pull/20460
[SPARK-23285][K8S] Allow fractional values for spark.executor.cores
## What changes were proposed in this pull request?
K8s treats CPU as a "compressible resource" and can actually assign millicpus to individual containers, e.g., 0.1 or 100m. In https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala#L94, we already parse `spark.executor.cores` as a double value. This PR simply bypasses the check for integral values for the property in Kubernetes mode.
## How was this patch tested?
Manual tests.
@foxish @vanzin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liyinan926/spark-k8s master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20460.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20460
----
commit a97187016c7f0711f71da6cb5b1a66ac791e843e
Author: Yinan Li <li...@...>
Date: 2018-01-31T21:29:12Z
[SPARK-23285][K8S] Allow fractional values for spark.executor.cores
K8s treats CPU as a "compressible resource" and can actually assign millicpus to individual containers, e.g., 0.1 or 100m. In https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala#L94, we already parse `spark.executor.cores` as a double value. This PR simply bypasses the check for integral values for the property in Kubernetes mode.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86995 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86995/testReport)** for PR 20460 at commit [`476221d`](https://github.com/apache/spark/commit/476221d0e4cb9f797a85f133aaba5236844e0fc5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
Agreed. It seems much more complicated than expected, considering the interaction between `spark.executor.cores` and `spark.task.cpus` and the implication for each scheduler backend. Will do a more thorough investigation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86890/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20460#discussion_r165360148
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -267,7 +267,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
&& Try(JavaUtils.byteStringAsBytes(executorMemory)).getOrElse(-1L) <= 0) {
SparkSubmit.printErrorAndExit("Executor Memory cores must be a positive number")
}
- if (executorCores != null && Try(executorCores.toInt).getOrElse(-1) <= 0) {
+ if (executorCores != null && Try(executorCores.toDouble).getOrElse(-1d) <= 0d) {
--- End diff --
Although I think "0.0" is more readable than "0d", don't bother changing it here. Just an aside
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
This may require more changes than it appears to be, agree we may need more tests to ensure it don't break anything, so +1 on target it to 2.4 instead of 2.3
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86992/testReport)** for PR 20460 at commit [`dc7483a`](https://github.com/apache/spark/commit/dc7483a03d959b2099da0b1cbea177fe767e870e).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class WorkerOffer(executorId: String, host: String, cores: Double)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/529/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86949/testReport)** for PR 20460 at commit [`44e489e`](https://github.com/apache/spark/commit/44e489eca6c22254fa57ff6f4750e3af85c43e34).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86995/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86998/testReport)** for PR 20460 at commit [`b21e864`](https://github.com/apache/spark/commit/b21e8640142f2f22d169bf461238d34009cc503d).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/527/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86998/testReport)** for PR 20460 at commit [`b21e864`](https://github.com/apache/spark/commit/b21e8640142f2f22d169bf461238d34009cc503d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86890 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86890/testReport)** for PR 20460 at commit [`a971870`](https://github.com/apache/spark/commit/a97187016c7f0711f71da6cb5b1a66ac791e843e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/442/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20460
I would suggest to bring out a discussion or even a design on dev mail list before doing such ground changing. This may affect not only dynamic allocation, but also scheduler. It is better to collect all the feedbacks (especially those who works on scheduler side).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20460
I think here (https://github.com/apache/spark/blob/032c11b83f0d276bf8085992229b8c598f02798a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L117) should also be fixed.
Besides, I think "spark.task.cpus" should also be fixed, right? BTW, looks like dynamic allocation can not be well supported if cores are fractional.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20460
Jenkins, retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86992/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86894/testReport)** for PR 20460 at commit [`712a5ff`](https://github.com/apache/spark/commit/712a5ff9e973b5eb0a970c3653aa48b6742969a3).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/527/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20460#discussion_r165221502
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -267,7 +267,11 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
&& Try(JavaUtils.byteStringAsBytes(executorMemory)).getOrElse(-1L) <= 0) {
SparkSubmit.printErrorAndExit("Executor Memory cores must be a positive number")
}
- if (executorCores != null && Try(executorCores.toInt).getOrElse(-1) <= 0) {
+ // Kubernetes mode allows fractional values for spark.executor.cores so bypass this check in
+ // the Kubernetes mode.
+ if (!master.startsWith("k8s")
+ && executorCores != null
+ && Try(executorCores.toInt).getOrElse(-1) <= 0) {
--- End diff --
If the intent here is just to check that it's positive, then just try to get it as a double and check it. This doesn't require making the logic dependent on a particular resource manager. Of course it relaxes the condition from what it is now, which would reject input like "1.5". But it would reject it not with a graceful exit but an exception, so I kind of think it's not what this code intends to catch anyway right now. (It would cause an error pretty quickly in YARN et al anyway)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #4088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4088/testReport)** for PR 20460 at commit [`d9805c3`](https://github.com/apache/spark/commit/d9805c3e4d4795f866e72f3c30f8ca29db90761d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/488/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
Posted by foxish <gi...@git.apache.org>.
Github user foxish commented on a diff in the pull request:
https://github.com/apache/spark/pull/20460#discussion_r165195110
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -267,7 +267,11 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
&& Try(JavaUtils.byteStringAsBytes(executorMemory)).getOrElse(-1L) <= 0) {
SparkSubmit.printErrorAndExit("Executor Memory cores must be a positive number")
}
- if (executorCores != null && Try(executorCores.toInt).getOrElse(-1) <= 0) {
+ // Kubernetes mode allows fractional values for spark.executor.cores so bypass this check in
+ // the Kubernetes mode.
+ if (!master.startsWith("k8s")
+ && executorCores != null
+ && Try(executorCores.toInt).getOrElse(-1) <= 0) {
--- End diff --
should this be nested `if..()` statements?
That's more readable IMO but not sure what's considered more idiomatic here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86993/testReport)** for PR 20460 at commit [`2d61782`](https://github.com/apache/spark/commit/2d617820da00a587cdc25238c1ff64cd3c19fcf3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
@jiangxb1987 @felixcheung @jerryshao this PR has been updated to allow both `spark.executor.cores` and `spark.task.cpus` to have fractional values. Please take a look at the updated PR description. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #4088 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4088/testReport)** for PR 20460 at commit [`d9805c3`](https://github.com/apache/spark/commit/d9805c3e4d4795f866e72f3c30f8ca29db90761d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/448/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
Just realized that the update I did still won't work when dynamic resource allocation is enabled. So please ignore the update. This is definitely much more impactful than I thought.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/462/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20460
> I'd target this 2.3 & master. Waiting for tests
@felixcheung is it too risky to target to 2.3, this is a fundamental behavior change. We should make sure k8s could well use fractional cores, in the mean time we should also guard other cluster manager to use it. I think we need more tests on it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86995 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86995/testReport)** for PR 20460 at commit [`476221d`](https://github.com/apache/spark/commit/476221d0e4cb9f797a85f133aaba5236844e0fc5).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86992/testReport)** for PR 20460 at commit [`dc7483a`](https://github.com/apache/spark/commit/dc7483a03d959b2099da0b1cbea177fe767e870e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/464/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/488/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86998/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/443/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86894 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86894/testReport)** for PR 20460 at commit [`712a5ff`](https://github.com/apache/spark/commit/712a5ff9e973b5eb0a970c3653aa48b6742969a3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86993/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
Agreed. This is a fundamental change to the way Spark handles task scheduling, task parallelism, and dynamic resource allocation, etc., and it impacts every scheduler backends. I'm closing this PR for now. Thanks for reviewing and giving feedbacks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86894/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86915/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/442/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86915/testReport)** for PR 20460 at commit [`d9805c3`](https://github.com/apache/spark/commit/d9805c3e4d4795f866e72f3c30f8ca29db90761d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/462/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
Do we also want to update the comment of `SPARK_EXECUTOR_CORES` in `spark-env.sh` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86890 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86890/testReport)** for PR 20460 at commit [`a971870`](https://github.com/apache/spark/commit/a97187016c7f0711f71da6cb5b1a66ac791e843e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by zenglian <gi...@git.apache.org>.
Github user zenglian commented on the issue:
https://github.com/apache/spark/pull/20460
I suggest to introduce a new property `spark.kubernetes.coreUnit = cpu | mcpu` (cpu by default), which would affect k8s only.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86949/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/446/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86993/testReport)** for PR 20460 at commit [`2d61782`](https://github.com/apache/spark/commit/2d617820da00a587cdc25238c1ff64cd3c19fcf3).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/490/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/532/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/447/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86895/testReport)** for PR 20460 at commit [`d9805c3`](https://github.com/apache/spark/commit/d9805c3e4d4795f866e72f3c30f8ca29db90761d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 closed the pull request at:
https://github.com/apache/spark/pull/20460
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86949/testReport)** for PR 20460 at commit [`44e489e`](https://github.com/apache/spark/commit/44e489eca6c22254fa57ff6f4750e3af85c43e34).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
Yea, we should either move the check to elsewhere or modify it, to avoid potential failure.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/535/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
@jiangxb1987 it seems `spark.cores.max` only applies to standalone and Mesos coarse-grained modes. In k8s mode, `spark.cores.max` is not applicable. I think the check really should not be here, but rather pushed to specific modes where this applies.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
Actually I think this may fail some check (though may not throw exceptions) for instance this one: https://github.com/apache/spark/blob/9678941f54ebc5db935ed8d694e502086e2a31c0/core/src/main/scala/org/apache/spark/SparkConf.scala#L553
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86895/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20460
ah, sounds like this is more impactful than we thought
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86895/testReport)** for PR 20460 at commit [`d9805c3`](https://github.com/apache/spark/commit/d9805c3e4d4795f866e72f3c30f8ca29db90761d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20460#discussion_r165223500
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -267,7 +267,11 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
&& Try(JavaUtils.byteStringAsBytes(executorMemory)).getOrElse(-1L) <= 0) {
SparkSubmit.printErrorAndExit("Executor Memory cores must be a positive number")
}
- if (executorCores != null && Try(executorCores.toInt).getOrElse(-1) <= 0) {
+ // Kubernetes mode allows fractional values for spark.executor.cores so bypass this check in
+ // the Kubernetes mode.
+ if (!master.startsWith("k8s")
+ && executorCores != null
+ && Try(executorCores.toInt).getOrElse(-1) <= 0) {
--- End diff --
Makes sense, done.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
@jiangxb1987 fixed the check in https://github.com/apache/spark/pull/20460/commits/44e489eca6c22254fa57ff6f4750e3af85c43e34.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86915/testReport)** for PR 20460 at commit [`d9805c3`](https://github.com/apache/spark/commit/d9805c3e4d4795f866e72f3c30f8ca29db90761d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20460
I'd target this 2.3 & master. Waiting for tests
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/446/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20460#discussion_r165223463
--- Diff: docs/configuration.md ---
@@ -1220,14 +1220,15 @@ Apart from these, the following properties are also available, and may be useful
<tr>
<td><code>spark.executor.cores</code></td>
<td>
- 1 in YARN mode, all the available cores on the worker in
+ 1 in YARN and Kubernetes modes, all the available cores on the worker in
standalone and Mesos coarse-grained modes.
</td>
<td>
The number of cores to use on each executor.
In standalone and Mesos coarse-grained modes, for more detail, see
- <a href="spark-standalone.html#Executors Scheduling">this description</a>.
+ <a href="spark-standalone.html#Executors Scheduling">this description</a>. In Kubernetes mode,
+ a fractional value can be used, e.g., 0.1 or 100m.
--- End diff --
Oh, right. Fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/530/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20460: [SPARK-23285][K8S] Allow fractional values for sp...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20460#discussion_r165221518
--- Diff: docs/configuration.md ---
@@ -1220,14 +1220,15 @@ Apart from these, the following properties are also available, and may be useful
<tr>
<td><code>spark.executor.cores</code></td>
<td>
- 1 in YARN mode, all the available cores on the worker in
+ 1 in YARN and Kubernetes modes, all the available cores on the worker in
standalone and Mesos coarse-grained modes.
</td>
<td>
The number of cores to use on each executor.
In standalone and Mesos coarse-grained modes, for more detail, see
- <a href="spark-standalone.html#Executors Scheduling">this description</a>.
+ <a href="spark-standalone.html#Executors Scheduling">this description</a>. In Kubernetes mode,
+ a fractional value can be used, e.g., 0.1 or 100m.
--- End diff --
100m isn't fractional though?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org