You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gaborgsomogyi <gi...@git.apache.org> on 2018/02/21 14:25:54 UTC
[GitHub] spark pull request #20652: [SPARK-23476][SPARK-SHELL] Generate secret in loc...
GitHub user gaborgsomogyi opened a pull request:
https://github.com/apache/spark/pull/20652
[SPARK-23476][SPARK-SHELL] Generate secret in local mode when authentication on
## What changes were proposed in this pull request?
If spark is run with "spark.authenticate=true", then it will fail to start in local mode.
This PR generates secret in local mode when authentication on.
## How was this patch tested?
Modified existing unit test.
Manually started spark-shell.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gaborgsomogyi/spark SPARK-23476
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20652.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20652
----
commit cf35ff3e9bcbb95ec6f5755af964c2ae4de5b463
Author: Gabor Somogyi <ga...@...>
Date: 2018-02-21T10:23:49Z
[SPARK-23476][SPARK-SHELL] Generate secret in local mode when authentication on
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87604/testReport)** for PR 20652 at commit [`3f49371`](https://github.com/apache/spark/commit/3f49371759c58af4277e1b8825b5c05f376b3ef0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87619/testReport)** for PR 20652 at commit [`e01feef`](https://github.com/apache/spark/commit/e01feeff30ed226d638055fed0238ef0e1c0c69e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87601/testReport)** for PR 20652 at commit [`3f49371`](https://github.com/apache/spark/commit/3f49371759c58af4277e1b8825b5c05f376b3ef0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87619/testReport)** for PR 20652 at commit [`e01feef`](https://github.com/apache/spark/commit/e01feeff30ed226d638055fed0238ef0e1c0c69e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87593/testReport)** for PR 20652 at commit [`cf35ff3`](https://github.com/apache/spark/commit/cf35ff3e9bcbb95ec6f5755af964c2ae4de5b463).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20652
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20652
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87593 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87593/testReport)** for PR 20652 at commit [`cf35ff3`](https://github.com/apache/spark/commit/cf35ff3e9bcbb95ec6f5755af964c2ae4de5b463).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87604/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20652: [SPARK-23476][SPARK-SHELL] Generate secret in loc...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20652#discussion_r169797492
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -529,7 +529,8 @@ private[spark] class SecurityManager(
return
}
- if (sparkConf.get(SparkLauncher.SPARK_MASTER, null) != "yarn") {
+ val master = sparkConf.get(SparkLauncher.SPARK_MASTER, "")
--- End diff --
Nice catch! I've enhanced the test and fixed the issue.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/20652
cc @vanzin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20652
Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20652
LGTM, but the title should say `[core]` (not spark-shell).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20652: [SPARK-23476][CORE] Generate secret in local mode...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20652#discussion_r169996395
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala ---
@@ -440,23 +440,41 @@ class SecurityManagerSuite extends SparkFunSuite with ResetSystemProperties {
assert(keyFromEnv === new SecurityManager(conf2).getSecretKey())
}
- test("secret key generation in yarn mode") {
- val conf = new SparkConf()
- .set(NETWORK_AUTH_ENABLED, true)
- .set(SparkLauncher.SPARK_MASTER, "yarn")
- val mgr = new SecurityManager(conf)
-
- UserGroupInformation.createUserForTesting("authTest", Array()).doAs(
- new PrivilegedExceptionAction[Unit]() {
- override def run(): Unit = {
- mgr.initializeAuth()
- val creds = UserGroupInformation.getCurrentUser().getCredentials()
- val secret = creds.getSecretKey(SecurityManager.SECRET_LOOKUP_KEY)
- assert(secret != null)
- assert(new String(secret, UTF_8) === mgr.getSecretKey())
+ test("secret key generation") {
+ Seq(
+ ("yarn", true),
+ ("local", true),
+ ("local[*]", true),
+ ("local[1,2]", true),
--- End diff --
nit: `local[1,2]` -> `local[1, 2]`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20652: [SPARK-23476][SPARK-SHELL] Generate secret in loc...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20652#discussion_r169736571
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -529,7 +529,8 @@ private[spark] class SecurityManager(
return
}
- if (sparkConf.get(SparkLauncher.SPARK_MASTER, null) != "yarn") {
+ val master = sparkConf.get(SparkLauncher.SPARK_MASTER, "")
--- End diff --
I'm not sure this will work for local-cluster. Try `local-cluster[2, 1, 1024]` as the master and see what happens.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87619/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20652: [SPARK-23476][CORE] Generate secret in local mode...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20652
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87601/testReport)** for PR 20652 at commit [`3f49371`](https://github.com/apache/spark/commit/3f49371759c58af4277e1b8825b5c05f376b3ef0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87601/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20652
**[Test build #87604 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87604/testReport)** for PR 20652 at commit [`3f49371`](https://github.com/apache/spark/commit/3f49371759c58af4277e1b8825b5c05f376b3ef0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][CORE] Generate secret in local mode when a...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/20652
lgtm
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20652: [SPARK-23476][CORE] Generate secret in local mode...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20652#discussion_r170003945
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala ---
@@ -440,23 +440,41 @@ class SecurityManagerSuite extends SparkFunSuite with ResetSystemProperties {
assert(keyFromEnv === new SecurityManager(conf2).getSecretKey())
}
- test("secret key generation in yarn mode") {
- val conf = new SparkConf()
- .set(NETWORK_AUTH_ENABLED, true)
- .set(SparkLauncher.SPARK_MASTER, "yarn")
- val mgr = new SecurityManager(conf)
-
- UserGroupInformation.createUserForTesting("authTest", Array()).doAs(
- new PrivilegedExceptionAction[Unit]() {
- override def run(): Unit = {
- mgr.initializeAuth()
- val creds = UserGroupInformation.getCurrentUser().getCredentials()
- val secret = creds.getSecretKey(SecurityManager.SECRET_LOOKUP_KEY)
- assert(secret != null)
- assert(new String(secret, UTF_8) === mgr.getSecretKey())
+ test("secret key generation") {
+ Seq(
+ ("yarn", true),
+ ("local", true),
+ ("local[*]", true),
+ ("local[1,2]", true),
--- End diff --
Fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20652: [SPARK-23476][SPARK-SHELL] Generate secret in local mode...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20652
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87593/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org