You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liyinan926 <gi...@git.apache.org> on 2017/12/13 21:13:32 UTC
[GitHub] spark pull request #19972: [SPARK-22778][Kubernetes] Added the missing servi...
GitHub user liyinan926 opened a pull request:
https://github.com/apache/spark/pull/19972
[SPARK-22778][Kubernetes] Added the missing service metadata for KubernetesClusterManager
## What changes were proposed in this pull request?
This PR added the missing service metadata for `KubernetesClusterManager`. Without the metadata, the service loader couldn't load `KubernetesClusterManager`, and caused the driver to fail to create a `ExternalClusterManager`, as being reported in SPARK-22778. The PR also changed the `k8s:` prefix used to `k8s://`, which is what existing Spark on k8s users are familiar and used to.
## How was this patch tested?
Manual testing verified that the fix resolved the issue in SPARK-22778.
/cc k8s-big-data SIG members & contributors: @mccheah @foxish @ash211 @ssuchter @varunkatta @kimoonkim @erikerlandson @tnachen @ifilonenko @liyinan926
/cc @vanzin @felixcheung @jiangxb1987
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liyinan926/spark-k8s fix-22778
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19972.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19972
----
commit 48c4b0a6f31b3d05151219f55f8ac545a8236102
Author: Yinan Li <li...@gmail.com>
Date: 2017-12-13T21:05:41Z
[SPARK-22778][Kubernetes] Added the missing service metadata for KubernetesClusterManager
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19972
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19972
**[Test build #84888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84888/testReport)** for PR 19972 at commit [`c91c9a6`](https://github.com/apache/spark/commit/c91c9a606bb00ea57c50fe27081273e0d5a8bcee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by foxish <gi...@git.apache.org>.
Github user foxish commented on the issue:
https://github.com/apache/spark/pull/19972
LGTM.
Will independently verify the fix as well. Thanks @mccheah and @liyinan926 for debugging.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19972
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19972
**[Test build #84878 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84878/testReport)** for PR 19972 at commit [`48c4b0a`](https://github.com/apache/spark/commit/48c4b0a6f31b3d05151219f55f8ac545a8236102).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19972
**[Test build #84878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84878/testReport)** for PR 19972 at commit [`48c4b0a`](https://github.com/apache/spark/commit/48c4b0a6f31b3d05151219f55f8ac545a8236102).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19972: [SPARK-22778][Kubernetes] Added the missing servi...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19972#discussion_r156994588
--- Diff: resource-managers/kubernetes/core/src/main/resources/META-INF/services/org.apache.spark.scheduler.ExternalClusterManager ---
@@ -0,0 +1 @@
+org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager
--- End diff --
Done.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19972: [SPARK-22778][Kubernetes] Added the missing servi...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19972#discussion_r156876310
--- Diff: resource-managers/kubernetes/core/src/main/resources/META-INF/services/org.apache.spark.scheduler.ExternalClusterManager ---
@@ -0,0 +1 @@
+org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager
--- End diff --
nit: add an extra empty line below this
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19972
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19972: [SPARK-22778][Kubernetes] Added the missing servi...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19972#discussion_r156875950
--- Diff: resource-managers/kubernetes/core/src/main/resources/META-INF/services/org.apache.spark.scheduler.ExternalClusterManager ---
@@ -0,0 +1 @@
+org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager
--- End diff --
What's this line for?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19972
**[Test build #84920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84920/testReport)** for PR 19972 at commit [`a2e6396`](https://github.com/apache/spark/commit/a2e6396e4b7e1654aac0489b67777090cf59c4dc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19972
One of the tests failures looks legitimate...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19972
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84920/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19972
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19972
@vanzin Yes, fixed in https://github.com/apache/spark/pull/19972/commits/c91c9a606bb00ea57c50fe27081273e0d5a8bcee.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19972
@vanzin @felixcheung @jiangxb1987 Can one of you help take a look and merge? This is fixing a blocker issue for Kubernetes mode. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19972
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84888/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19972
**[Test build #84920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84920/testReport)** for PR 19972 at commit [`a2e6396`](https://github.com/apache/spark/commit/a2e6396e4b7e1654aac0489b67777090cf59c4dc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19972
Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19972: [SPARK-22778][Kubernetes] Added the missing servi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19972
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19972
**[Test build #84888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84888/testReport)** for PR 19972 at commit [`c91c9a6`](https://github.com/apache/spark/commit/c91c9a606bb00ea57c50fe27081273e0d5a8bcee).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19972: [SPARK-22778][Kubernetes] Added the missing service meta...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19972
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84878/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org