You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by subrotosanyal <gi...@git.apache.org> on 2016/06/03 15:16:00 UTC
[GitHub] spark pull request #13499: SPARK-15754 Not letting the credentials containin...
GitHub user subrotosanyal opened a pull request:
https://github.com/apache/spark/pull/13499
SPARK-15754 Not letting the credentials containing hdfs delegation tokens to be added in current user credential.
## What changes were proposed in this pull request?
The credentials are not added to the credentials of UserGroupInformation.getCurrentUser(). Further if the client has possibility to login using keytab then the updateDelegationToken thread is not started on client.
## How was this patch tested?
ran dev/run-tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/subrotosanyal/spark SPARK-15754-save-ugi-from-changing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13499.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13499
----
commit 16ce28c08e463567fcc5a66c80967923bd19193e
Author: Subroto Sanyal <ss...@datameer.com>
Date: 2016-06-03T14:58:17Z
SPARK-15754 Not letting the credentials containing hdfs delegation tokens to be added in current user credential.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13499: [SPARK-15754] [yarn] Not letting the credentials ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13499#discussion_r65764703
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---
@@ -64,7 +68,7 @@ private[spark] class YarnClientSchedulerBackend(
// SPARK-8851: In yarn-client mode, the AM still does the credentials refresh. The driver
// reads the credentials from HDFS, just like the executors and updates its own credentials
// cache.
- if (conf.contains("spark.yarn.credentials.file")) {
+ if (!conf.contains(PRINCIPAL.key) && conf.contains("spark.yarn.credentials.file")) {
--- End diff --
That's a different issue, and I'm not sure it's really an issue, given what the comment above this line says.
This code is here for when Spark is managing your kerberos credentials and delegation tokens, i.e., when you pass `--principal` and `--keytab`. In that case, as the comment above says, the AM is tasked with renewing the credentials and tokens periodically, and this code exists so that the driver (which is not running in the AM in client mode) can get the new tokens.
You're basically breaking that feature by changing this. If your app is managing the kerberos login, you'd never pass `--principal` and `--keytab` (or the equivalent settings) to Spark, so you wouldn't run into this problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13499
**[Test build #59958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59958/consoleFull)** for PR 13499 at commit [`3a4c04a`](https://github.com/apache/spark/commit/3a4c04a6c57851512e836d247b0c97e647aa86cd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13499: [SPARK-15754] [yarn] Not letting the credentials ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13499
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: SPARK-15754 Not letting the credentials containing hdfs ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13499
**[Test build #59938 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59938/consoleFull)** for PR 13499 at commit [`16ce28c`](https://github.com/apache/spark/commit/16ce28c08e463567fcc5a66c80967923bd19193e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13499
**[Test build #59958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59958/consoleFull)** for PR 13499 at commit [`3a4c04a`](https://github.com/apache/spark/commit/3a4c04a6c57851512e836d247b0c97e647aa86cd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: SPARK-15754 Not letting the credentials containing hdfs ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13499
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: SPARK-15754 Not letting the credentials containing hdfs ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13499
**[Test build #59938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59938/consoleFull)** for PR 13499 at commit [`16ce28c`](https://github.com/apache/spark/commit/16ce28c08e463567fcc5a66c80967923bd19193e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13499
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by subrotosanyal <gi...@git.apache.org>.
Github user subrotosanyal commented on the issue:
https://github.com/apache/spark/pull/13499
hi @vanzin
Thanks for your comments.
I have removed the code change for YarnClientSchedulerBackend.
I will raise a separate ticket after further more analysis for the problem I am assuming to happen.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13499
**[Test build #59955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59955/consoleFull)** for PR 13499 at commit [`d53ad55`](https://github.com/apache/spark/commit/d53ad55cf558c7222c7f2d501606f62513d4512f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: SPARK-15754 Not letting the credentials containing hdfs ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13499
@subrotosanyal could you update the title to follow the convention? `[SPARK-xxxx][yarn] Summary.` Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13499
Ran some kerberos tests here and all looks good, so I'm merging this (will also try 1.6.x if there are no conflicts). Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13499: [SPARK-15754] [yarn] Not letting the credentials ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13499#discussion_r65752573
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---
@@ -64,7 +68,7 @@ private[spark] class YarnClientSchedulerBackend(
// SPARK-8851: In yarn-client mode, the AM still does the credentials refresh. The driver
// reads the credentials from HDFS, just like the executors and updates its own credentials
// cache.
- if (conf.contains("spark.yarn.credentials.file")) {
+ if (!conf.contains(PRINCIPAL.key) && conf.contains("spark.yarn.credentials.file")) {
--- End diff --
Since you're touching this, you could also use `CREDENTIALS_FILE_PATH`.
BTW I'm not entirely sure why this change is needed, or related to the issue you reported. You want to start the token renewer code exactly when `PRINCIPAL` is defined. You're pretty much disabling this code now, because `CREDENTIALS_FILE_PATH` is never set when `PRINCIPAL` is not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13499
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59955/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: SPARK-15754 Not letting the credentials containing hdfs ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13499
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13499
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59958/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13499: [SPARK-15754] [yarn] Not letting the credentials ...
Posted by subrotosanyal <gi...@git.apache.org>.
Github user subrotosanyal commented on a diff in the pull request:
https://github.com/apache/spark/pull/13499#discussion_r65762991
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---
@@ -64,7 +68,7 @@ private[spark] class YarnClientSchedulerBackend(
// SPARK-8851: In yarn-client mode, the AM still does the credentials refresh. The driver
// reads the credentials from HDFS, just like the executors and updates its own credentials
// cache.
- if (conf.contains("spark.yarn.credentials.file")) {
+ if (!conf.contains(PRINCIPAL.key) && conf.contains("spark.yarn.credentials.file")) {
--- End diff --
hi @vanzin
the method: org.apache.spark.deploy.yarn.ExecutorDelegationTokenUpdater.updateCredentialsIfRequired() invokes the same operation of updating the current user credential and correct me if I am wrong; if user has logged in through a keytab file already then ideally there shouldn't be any need to update the credentials at regular intervals. This thread of updating the credential might be useful for executors where there is only CREDENTAILS_FILE or delegation token.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13499
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13499: [SPARK-15754] [yarn] Not letting the credentials ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13499#discussion_r65752132
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---
@@ -21,9 +21,13 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.hadoop.yarn.api.records.YarnApplicationState
-import org.apache.spark.{SparkContext, SparkException}
-import org.apache.spark.deploy.yarn.{Client, ClientArguments, YarnSparkHadoopUtil}
+import org.apache.spark.SparkContext
--- End diff --
These changes are unnecessary, and in fact we prefer the previous style.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by subrotosanyal <gi...@git.apache.org>.
Github user subrotosanyal commented on the issue:
https://github.com/apache/spark/pull/13499
@vanzin
Updated the title.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: SPARK-15754 Not letting the credentials containing hdfs ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13499
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59938/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13499
**[Test build #59955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59955/consoleFull)** for PR 13499 at commit [`d53ad55`](https://github.com/apache/spark/commit/d53ad55cf558c7222c7f2d501606f62513d4512f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: SPARK-15754 Not letting the credentials containing hdfs ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13499
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13499: [SPARK-15754] [yarn] Not letting the credentials contain...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13499
BTW the change looks fine, I just want to take a look at the rest of the code (the parts you're not touching) and also run some tests internally with kerberos enabled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org