You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2018/06/13 06:04:12 UTC
[GitHub] spark pull request #21548: [SPARK-24518] Using Hadoop credential provider AP...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21548
[SPARK-24518] Using Hadoop credential provider API to store password
## What changes were proposed in this pull request?
Current Spark configs password in a plaintext way, like putting in the configuration file or adding as a launch arguments, sometimes such configurations like SSL password is configured by cluster admin, which should not be seen by user, but now this passwords are world readable to all the users.
Hadoop credential provider API support storing password in a secure way, in which Spark could read it in a secure way, so here propose to add support of using credential provider API to get password.
## How was this patch tested?
Existing tests and verified locally.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-24518
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21548.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21548
----
commit 575152bc01ecd9ab9fcbdef8c03d1fed714d7c53
Author: jerryshao <ss...@...>
Date: 2018-06-13T05:01:40Z
Using Hadoop credential provider API to store password
Change-Id: Ie774eeb9376f8b5d7379f1976826e12e9c529be3
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3975/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #91757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91757/testReport)** for PR 21548 at commit [`575152b`](https://github.com/apache/spark/commit/575152bc01ecd9ab9fcbdef8c03d1fed714d7c53).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21548
Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21548: [SPARK-24518][CORE] Using Hadoop credential provi...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197033939
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging {
.orElse(defaults.flatMap(_.keyStore))
val keyStorePassword = conf.getWithSubstitution(s"$ns.keyStorePassword")
+ .orElse(Option(hadoopConf.getPassword(s"$ns.keyStorePassword")).map(new String(_)))
--- End diff --
@vanzin the return value of `hadoopConf#getPassword` is char array, so there's no way to specify the charset here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21548: [SPARK-24518][CORE] Using Hadoop credential provi...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r196974952
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging {
.orElse(defaults.flatMap(_.keyStore))
val keyStorePassword = conf.getWithSubstitution(s"$ns.keyStorePassword")
+ .orElse(Option(hadoopConf.getPassword(s"$ns.keyStorePassword")).map(new String(_)))
--- End diff --
Needs charset (also in others).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #92120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92120/testReport)** for PR 21548 at commit [`c7ef15e`](https://github.com/apache/spark/commit/c7ef15e47e97e675a63444d0b66b8b8808cccf90).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518] Using Hadoop credential provider API to st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91747/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/351/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21548: [SPARK-24518][CORE] Using Hadoop credential provi...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r196976257
--- Diff: docs/security.md ---
@@ -446,6 +446,18 @@ replaced with one of the above namespaces.
</tr>
</table>
+Spark also supports retrieving `${ns}.keyPassword`, `${ns}.keyStorePassword` and `${ns}.trustStorePassword` from
+[Hadoop Credential Providers](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
+User could store password into credential file and make it accessible by different components, like:
+
+```
+hadoop credential create spark.ssl.keyPassword -value password \
+ -provider jceks://hdfs@nn1.example.com:9001/user/backup/ssl.jceks
+```
+
+In the meantime, adding configuration "hadoop.security.credential.provider.path=jceks://hdfs@nn1.example.com:9001/user/backup/ssl.jceks"
--- End diff --
Rephrase:
"To configure the location of the credential provider, set the `hadoop.security.credential.provider.path` config option in the Hadoop configuration used by Spark."
Your example also kinda looks like a Spark config (which would be "spark.hadoop.blah"), since Hadoop configs are generally in XML.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518] Using Hadoop credential provider API to st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21548: [SPARK-24518][CORE] Using Hadoop credential provi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21548
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #92129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92129/testReport)** for PR 21548 at commit [`c7ef15e`](https://github.com/apache/spark/commit/c7ef15e47e97e675a63444d0b66b8b8808cccf90).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518] Using Hadoop credential provider API to st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3967/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/346/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Jenkins, retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92164/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
> I just have an issue with your summary. It makes it sound like there's no way to securely configure these things today, and that's not true.
>
> You can have the password in the config file and have it only readable by authorized users, which is basically what the credential provider does.
>
> Or you can have the password in an env variable, and reference the env variable in the Spark config.
I see, thanks for explanation @vanzin . This might be the problem in our distribution, because we don't do such fine-grained access control of config file, also configuration file is world readable shared between different components, that's why we're seeking to use Hadoop credential provide to secure this thing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #92120 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92120/testReport)** for PR 21548 at commit [`c7ef15e`](https://github.com/apache/spark/commit/c7ef15e47e97e675a63444d0b66b8b8808cccf90).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21548: [SPARK-24518][CORE] Using Hadoop credential provi...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197327620
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging {
.orElse(defaults.flatMap(_.keyStore))
val keyStorePassword = conf.getWithSubstitution(s"$ns.keyStorePassword")
+ .orElse(Option(hadoopConf.getPassword(s"$ns.keyStorePassword")).map(new String(_)))
--- End diff --
Hi @vanzin , I checked jdk8 doc again, I don't find a String constructor which takes both char array and charset as parameters.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518] Using Hadoop credential provider API to st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #91747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91747/testReport)** for PR 21548 at commit [`575152b`](https://github.com/apache/spark/commit/575152bc01ecd9ab9fcbdef8c03d1fed714d7c53).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21548
we would definitely want to update the docs on how user would do this. I don't see a test that actually tests reading from the hadoopConf either, so we should add one.
I need to look at the hadoop api in more depth to do a full review
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #92164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92164/testReport)** for PR 21548 at commit [`1ab28d0`](https://github.com/apache/spark/commit/1ab28d0320f19de5697b7d275aa43918c19d89ee).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92120/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/85/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #92164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92164/testReport)** for PR 21548 at commit [`1ab28d0`](https://github.com/apache/spark/commit/1ab28d0320f19de5697b7d275aa43918c19d89ee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91757/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92129/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #92129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92129/testReport)** for PR 21548 at commit [`c7ef15e`](https://github.com/apache/spark/commit/c7ef15e47e97e675a63444d0b66b8b8808cccf90).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4247/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518] Using Hadoop credential provider API to st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/77/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21548: [SPARK-24518][CORE] Using Hadoop credential provi...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197502613
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging {
.orElse(defaults.flatMap(_.keyStore))
val keyStorePassword = conf.getWithSubstitution(s"$ns.keyStorePassword")
+ .orElse(Option(hadoopConf.getPassword(s"$ns.keyStorePassword")).map(new String(_)))
--- End diff --
Oh, my bad, that's a char array, not a byte array. All is good then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21548: [SPARK-24518][CORE] Using Hadoop credential provi...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21548#discussion_r197182301
--- Diff: core/src/main/scala/org/apache/spark/SSLOptions.scala ---
@@ -179,9 +185,11 @@ private[spark] object SSLOptions extends Logging {
.orElse(defaults.flatMap(_.keyStore))
val keyStorePassword = conf.getWithSubstitution(s"$ns.keyStorePassword")
+ .orElse(Option(hadoopConf.getPassword(s"$ns.keyStorePassword")).map(new String(_)))
--- End diff --
`new String` takes a charset. (In fact the constructor you're calling should be deprecated...)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4242/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #91747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91747/testReport)** for PR 21548 at commit [`575152b`](https://github.com/apache/spark/commit/575152bc01ecd9ab9fcbdef8c03d1fed714d7c53).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21548
**[Test build #91757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91757/testReport)** for PR 21548 at commit [`575152b`](https://github.com/apache/spark/commit/575152bc01ecd9ab9fcbdef8c03d1fed714d7c53).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21548
Could you update the summary so that it doesn't sound like this is an existing security issue?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/372/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21548
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
CC @vanzin @tgravescs would you please help to review, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Jenkins, retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21548: [SPARK-24518][CORE] Using Hadoop credential provider API...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21548
Thanks @tgravescs I will add the docs about how to use it. I was thinking to add a test case, but it looks like may not be easy to add one.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org