You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by vanzin <gi...@git.apache.org> on 2015/10/24 01:38:02 UTC
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/9257
[SPARK-11073] [core] [yarn] Remove akka dependency in secret key generation.
Use standard JDK APIs for that (with a little help from Guava). Most of the
changes here are in test code, since there were no tests specific to that
part of the code.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-11073
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9257.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9257
----
commit 98761ddef9204639f9be8a9c518bccc0f4e0c800
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2015-10-23T23:33:10Z
[SPARK-11073] [core] [yarn] Remove akka dependency in secret key generation.
Use standard JDK APIs for that (with a little help from Guava). Most of the
changes here are in test code, since there were no tests specific to that
part of the code.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151841537
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150732059
cc @tgravescs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r43000107
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
--- End diff --
Better to use Byte.SIZE rather then hardcoded 8.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151791971
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44512/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151760692
**[Test build #44512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44512/consoleFull)** for PR 9257 at commit [`c6cde97`](https://github.com/apache/spark/commit/c6cde972a9743b938e34e2765d2b3fb8c5d35c93).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150868552
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150854057
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-152867212
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r43012664
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
+ val secret = new Array[Byte](length)
+ rnd.nextBytes(secret)
+
+ val cookie = HashCodes.fromBytes(secret).toString()
+ SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY, cookie)
+ cookie
} else {
- logDebug("getSecretKey: yarn mode, secret key from credentials is null")
--- End diff --
They didn't feel particularly useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r43012812
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
--- End diff --
akka hardcodes the secret size at 32 bytes; I meant to keep the configuration private, since I don't expect people to change this. But you never know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151855499
+1 pending Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151841517
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150867602
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44304/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150873147
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150867571
**[Test build #44304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44304/consoleFull)** for PR 9257 at commit [`c6c9823`](https://github.com/apache/spark/commit/c6c98233a2e7e888a6cf28e39ac4993d013e62b9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150856249
**[Test build #44304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44304/consoleFull)** for PR 9257 at commit [`c6c9823`](https://github.com/apache/spark/commit/c6c98233a2e7e888a6cf28e39ac4993d013e62b9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150717680
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150873148
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44303/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150867601
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150868461
not sure what failed. retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r43251769
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
+ val secret = new Array[Byte](length)
+ rnd.nextBytes(secret)
+
+ val cookie = HashCodes.fromBytes(secret).toString()
+ SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY, cookie)
+ cookie
} else {
- logDebug("getSecretKey: yarn mode, secret key from credentials is null")
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150752727
**[Test build #44283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44283/consoleFull)** for PR 9257 at commit [`98761dd`](https://github.com/apache/spark/commit/98761ddef9204639f9be8a9c518bccc0f4e0c800).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150878473
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151843133
**[Test build #44528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44528/consoleFull)** for PR 9257 at commit [`b4a29bf`](https://github.com/apache/spark/commit/b4a29bf56dbee1e60d36df8d2272e7bfc8794f3b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150878474
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44305/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-152878679
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151880856
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44528/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151760113
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150868548
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150742433
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9257
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r43251680
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
+ val secret = new Array[Byte](length)
+ rnd.nextBytes(secret)
+
+ val cookie = HashCodes.fromBytes(secret).toString()
+ SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY, cookie)
+ cookie
} else {
- logDebug("getSecretKey: yarn mode, secret key from credentials is null")
--- End diff --
I'd prefer to see this one left. Otherwise there is no easy way to see what its doing for the secret. In general I'm against removing debug stuff unless its really really noisy. This should only be printed once and can be useful debugging user settings or issues with secrets.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151791817
**[Test build #44512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44512/consoleFull)** for PR 9257 at commit [`c6cde97`](https://github.com/apache/spark/commit/c6cde972a9743b938e34e2765d2b3fb8c5d35c93).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r43003681
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
--- End diff --
Also do we want to document this config or did you mean it to be internal for now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150752926
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44283/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150878447
**[Test build #44305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44305/consoleFull)** for PR 9257 at commit [`c6c9823`](https://github.com/apache/spark/commit/c6c98233a2e7e888a6cf28e39ac4993d013e62b9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150854052
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150855200
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150717944
**[Test build #44270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44270/consoleFull)** for PR 9257 at commit [`98761dd`](https://github.com/apache/spark/commit/98761ddef9204639f9be8a9c518bccc0f4e0c800).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151760102
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150734094
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44270/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r42999614
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
--- End diff --
does akka have a limit on the size of this? I remember the documentation being very sparse on the details.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150868861
**[Test build #44305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44305/consoleFull)** for PR 9257 at commit [`c6c9823`](https://github.com/apache/spark/commit/c6c98233a2e7e888a6cf28e39ac4993d013e62b9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150734093
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150717697
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-152878680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44771/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151791967
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-152878898
Merging this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150733922
**[Test build #44270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44270/consoleFull)** for PR 9257 at commit [`98761dd`](https://github.com/apache/spark/commit/98761ddef9204639f9be8a9c518bccc0f4e0c800).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class DefaultSource extends HadoopFsRelationProvider with DataSourceRegister `\n * `class TextOutputWriter(path: String, dataSchema: StructType, context: TaskAttemptContext)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150741629
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150854315
**[Test build #44303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44303/consoleFull)** for PR 9257 at commit [`35002a9`](https://github.com/apache/spark/commit/35002a99324570dbddcd42551e220860c5b14f39).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-152878642
**[Test build #44771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44771/consoleFull)** for PR 9257 at commit [`0878148`](https://github.com/apache/spark/commit/08781488e0bca2701d1c01d38f5f00ed96c860c3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150873124
**[Test build #44303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44303/consoleFull)** for PR 9257 at commit [`35002a9`](https://github.com/apache/spark/commit/35002a99324570dbddcd42551e220860c5b14f39).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9257#discussion_r42997294
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -365,33 +366,37 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
* we throw an exception.
*/
private def generateSecretKey(): String = {
- if (!isAuthenticationEnabled) return null
- // first check to see if the secret is already set, else generate a new one if on yarn
- val sCookie = if (SparkHadoopUtil.get.isYarnMode) {
- val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
- if (secretKey != null) {
- logDebug("in yarn mode, getting secret from credentials")
- return new Text(secretKey).toString
+ if (!isAuthenticationEnabled) {
+ null
+ } else if (SparkHadoopUtil.get.isYarnMode) {
+ // In YARN mode, the secure cookie will be created by the driver and stashed in the
+ // user's credentials, where executors can get it. The check for an array of size 0
+ // is because of the test code in YarnSparkHadoopUtilSuite.
+ val secretKey = SparkHadoopUtil.get.getSecretKeyFromUserCredentials(SECRET_LOOKUP_KEY)
+ if (secretKey == null || secretKey.length == 0) {
+ val rnd = new SecureRandom()
+ val length = sparkConf.getInt("spark.authenticate.secretBitLength", 256) / 8
+ val secret = new Array[Byte](length)
+ rnd.nextBytes(secret)
+
+ val cookie = HashCodes.fromBytes(secret).toString()
+ SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY, cookie)
+ cookie
} else {
- logDebug("getSecretKey: yarn mode, secret key from credentials is null")
--- End diff --
any reason not to leave the debug messages?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150742957
**[Test build #44283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44283/consoleFull)** for PR 9257 at commit [`98761dd`](https://github.com/apache/spark/commit/98761ddef9204639f9be8a9c518bccc0f4e0c800).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151880637
**[Test build #44528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44528/consoleFull)** for PR 9257 at commit [`b4a29bf`](https://github.com/apache/spark/commit/b4a29bf56dbee1e60d36df8d2272e7bfc8794f3b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-152868050
**[Test build #44771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44771/consoleFull)** for PR 9257 at commit [`0878148`](https://github.com/apache/spark/commit/08781488e0bca2701d1c01d38f5f00ed96c860c3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150855171
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150752923
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-151880853
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-150742469
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11073] [core] [yarn] Remove akka depend...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9257#issuecomment-152867221
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org