You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2016/08/24 13:56:46 UTC
[GitHub] spark pull request #14789: [SPARK-17209][YARN] Add the ability to manually u...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/14789
[SPARK-17209][YARN] Add the ability to manually update credentials for Spark running on YARN
## What changes were proposed in this pull request?
This PR propose to add a new API in `SparkHadoopUtil` to trigger manual credential updating in the run-time.
This is mainly used in long running spark applications which needs to access different secured system in the run-time. For example, when Zeppelin / Spark Shell needs to access different secured HBase cluster or Hive metastore service in the run-time, it requires tokens from new services and updates to executors immediately. Previously either we need to relaunch the application to get new tokens, or we need to wait until the old tokens to expire to get new ones.
With this new API, user could manually trigger credential updating in the run-time when required. Credentials will be renewed in AM and updated in executor and driver side.
## How was this patch tested?
Manually verified in the secured cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-17209
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14789.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14789
----
commit 841bec0dfae79ecfce5dc535949879f397313736
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-08-24T07:12:22Z
Support manual credential update
commit 85370e787b719136cc97cfc358f0e50c55582bb8
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-08-24T13:23:36Z
Change the comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/14789
@vanzin You are right, this does look orthogonal to what you and Tom described.
'Where' the renewal is happening is the difference - from within spark or from outside of spark (oozie/from gateway node/etc).
I am wondering if we can model the change such that both usecases can be supported with minimal impedence.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #73461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73461/consoleFull)** for PR 14789 at commit [`b1cac5c`](https://github.com/apache/spark/commit/b1cac5c95f3c7f77a8253a88aff48d2a1934072c).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #68519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68519/consoleFull)** for PR 14789 at commit [`b1cac5c`](https://github.com/apache/spark/commit/b1cac5c95f3c7f77a8253a88aff48d2a1934072c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73461/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/14789
Interesting, thanks for clarifying.
The way I was looking at the change was, given we have ability for custom credential providers now, we would need to support for out-of-band (to the current expiry time based refresh/renew) updates to credentials.
If the same can be leveraged for other usecases (or evolved to do so), that would be even better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/14789
Hi @tgravescs,
This is specifically for the case where new input source/output destinations are getting added after the driver has already started. Notebooks, long running streaming applications applications, etc - where the configuration could be passed in dynamically.
I do agree that SparkHadoopUtil is a DeveloperApi which we are leveraging here, is there any other api in yarn module which will be a better fit ? I did not find, but could have overlooked.
I have other comments on the pr which I will add, but wanted to ensure there are no high level issues with the approach. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14789
bq. but I think the api makes sense to be very similar, or at least in the same sort of class
I think it will be hard to have the same API serve both use cases. You could have one API to generate new credentials and a second one to distribute a set of credentials, and then you could use the second one for the "update from outside of Spark" case. But they'd still be two different new user-facing methods.
I agree that `SparkHadoopUtil` is a weird place for this. I don't have a good suggestion for where to add it though - I've been planning to play with removing the credentials stuff from `SparkHadoopUtil`, since we don't support hadoop 1.x anymore and can clean up that code, but haven't had the time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14789
I guess I forgot about this pr. I don't think we have any yarn specific public apis. Also theoretically this isn't yarn specific if others decided to support it as well. mesos/standalone could also use credentials and updates its just no one has implemented it.
We do have api's that don't support all deploy modes though, like the SparkLauncher.
This feels like we should have public api in common place and just have backends handle if they support.
A little bit of a side note here, personally I would like to move away from keytabs because its not as secure and at least my company doesn't allow it. To get around keytabs, I was going to add an interface to push new credentials from the gateway box. Have a spark-submit argument that talked to the driver to push new credentials securely over rpc. The reason I bring that up is that if we are adding an api to updateCredentials it would be nice to make it flexible enough to handle that use case as well.
perhaps we should add an interface to get a Credentials like object that then can have the updateCredentials routine and others as needed, that way everything isn't directly in the SparkContext.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68519/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14789
@tgravescs , with [SPARK-14743](https://issues.apache.org/jira/browse/SPARK-14743), credentials/tokens can be managed out of Spark with their own credential provider. In that case user could update the credentials based on configuration changes or something else. For example: we could write a dynamic HBase credential provider which loads configuration from HDFS, once user what to change to another HBase cluster, they could update configuration files in HDFS and trigger credential updating manually using the API provided in this PR. It is mainly based on how user implement their own credential provider. What here provided is just a manual credential update mechanism, when and how to trigger this is relying on user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14789
So as far as the use case you are saying a user has a shell/process running, they are possibly accessing one hive or hbase cluster (or maybe none) and then they want to access another one. So how is it a user adds/changes the confs to the other hbase or hive clusters? which the trigger would then know how to go fetch.
What are all the steps required?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14789: [SPARK-17209][YARN] Add the ability to manually u...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14789#discussion_r87119403
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -138,6 +138,14 @@ class SparkHadoopUtil extends Logging {
}
/**
+ * Update credentials manually. This will trigger AM to renew the credentials immediately,
+ * also executors and driver (in client mode) will update the credentials accordingly.
+ *
+ * Note this will only be worked under YARN cluster manager.
+ */
+ def updateCredentials(sc: SparkContext): Unit = { }
--- End diff --
Agreed, `Future` might be better, I will change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #64352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64352/consoleFull)** for PR 14789 at commit [`85370e7`](https://github.com/apache/spark/commit/85370e787b719136cc97cfc358f0e50c55582bb8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14789: [SPARK-17209][YARN] Add the ability to manually u...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao closed the pull request at:
https://github.com/apache/spark/pull/14789
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #64352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64352/consoleFull)** for PR 14789 at commit [`85370e7`](https://github.com/apache/spark/commit/85370e787b719136cc97cfc358f0e50c55582bb8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14789: [SPARK-17209][YARN] Add the ability to manually u...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14789#discussion_r87072908
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -64,7 +64,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
// must be protected by `CoarseGrainedSchedulerBackend.this`. Besides, `executorDataMap` should
// only be modified in `DriverEndpoint.receive/receiveAndReply` with protection by
// `CoarseGrainedSchedulerBackend.this`.
- private val executorDataMap = new HashMap[String, ExecutorData]
+ protected val executorDataMap = new HashMap[String, ExecutorData]
--- End diff --
@tgravescs @vanzin Any thoughts on keeping this private and executing a closure on values of map for usecases like this ? (instead of exposing it or creating a copy via toMap). This can also ensure that outside uses will maintain the MT-invariance expected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #64390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64390/consoleFull)** for PR 14789 at commit [`a25a347`](https://github.com/apache/spark/commit/a25a3474a86f773ac0734c6f61063b842f3fd056).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14789: [SPARK-17209][YARN] Add the ability to manually u...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14789#discussion_r87116199
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala ---
@@ -189,6 +190,39 @@ private[spark] abstract class YarnSchedulerBackend(
}
/**
+ * Renew and update the credentials. This method will trigger credentials renewal in the AM
+ * side, once successfully renewed, it will trigger credentials updating in executor and
+ * driver side.
+ */
+ def updateCredentials(): Unit = {
+ yarnSchedulerEndpoint.amEndpoint match {
+ case Some(am) =>
+ val future = am.ask[Boolean](UpdateCredentials)
+
+ future.onSuccess {
+ case true =>
+ // Update credentials in the executor side only when AM is successfully renewed the
+ // Credentials.
+ synchronized {
+ executorDataMap.values.foreach(_.executorEndpoint.send(UpdateCredentials))
+ }
+ // This will trigger credential updating in the driver side.
+ SparkHadoopUtil.get.triggerCredentialUpdater()
--- End diff --
Note that when we modify the above to return future, this will also need to be in the future (or can always be done inline).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14789
so I agree these are separate cases but I think the api makes sense to be very similar, or at least in the same sort of class. I don't think we want a public end user api in SparkHadoopUtil.updateCredentials. I would rather us create something in core like a Credentials class and put it in there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14789: [SPARK-17209][YARN] Add the ability to manually u...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14789#discussion_r87075067
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -138,6 +138,14 @@ class SparkHadoopUtil extends Logging {
}
/**
+ * Update credentials manually. This will trigger AM to renew the credentials immediately,
+ * also executors and driver (in client mode) will update the credentials accordingly.
+ *
+ * Note this will only be worked under YARN cluster manager.
+ */
+ def updateCredentials(sc: SparkContext): Unit = { }
--- End diff --
You might want to return a Future for this, which can indicate when the credentials have been updated : as of now, developers invoking this api have no way to know when (if at all) the credentials have been propagated through the driver/executors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64352/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64390/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14789
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14789
I think there are two different issues, one that is being addressed by this change and one that is the one Tom and I are talking about:
1. triggering Spark's credential updater mechanism outside its period, because you want to connect to a new NN or something like that.
1. triggering the credential fetch mechanism from outside the internal credential updater, which would allow an external entity to update an existing Spark app's credentials.
The latter is a more complicated change since credential update and fetch are kinda intertwined right now; for example, for the latter to work, the AM would also need to fetch the new credentials as if it were an executor, and right now all the AM do is run the update thread.
So I think this is fine for covering case 1 (caveat: I haven't looked at the code yet), but it would be nice to have a solution for case 2 eventually.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14789
what or who exactly is calling the updateCredentials here? The user themselves are calling a developer api in SparkHadoopUtil? I don't know if we ever finished the discussion on DeveloperAPi, but that doesn't seem like a very stable interface for users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #64390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64390/consoleFull)** for PR 14789 at commit [`a25a347`](https://github.com/apache/spark/commit/a25a3474a86f773ac0734c6f61063b842f3fd056).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #73461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73461/consoleFull)** for PR 14789 at commit [`b1cac5c`](https://github.com/apache/spark/commit/b1cac5c95f3c7f77a8253a88aff48d2a1934072c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/14789
@tgravescs Interesting, so it means that there wont be a --principal/--keytab, and what is being done in AMCredentialRenewer will be done in the launcher itself ? So the driver behaves just like what executors currently do ?
The reason I want to understand more is that current spark design does allow for what is described above, but as soon as this change comes in, it will be difficult to support it : since driver knows about dynamically added clusters/sources, while launcher will not.
Unless we propagate from driver back to launcher, and then have the launcher update/push updates to driver and executors.
If yes, you are right, we will need to be careful while designing this to ensure there is minimal change when we evolve to allow for that scenario also.
Not sure if I got the problem statement totally confused though !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14789
The user themselves will call this API. Another option in [SPARK-14743](https://issues.apache.org/jira/browse/SPARK-14743) is to add this API in `SparkContext`. But from my understanding it is quite Hadoop related, so adding here might be more proper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14789: [SPARK-17209][YARN] Add the ability to manually u...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14789#discussion_r87078341
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala ---
@@ -189,6 +190,39 @@ private[spark] abstract class YarnSchedulerBackend(
}
/**
+ * Renew and update the credentials. This method will trigger credentials renewal in the AM
+ * side, once successfully renewed, it will trigger credentials updating in executor and
+ * driver side.
+ */
+ def updateCredentials(): Unit = {
+ yarnSchedulerEndpoint.amEndpoint match {
+ case Some(am) =>
+ val future = am.ask[Boolean](UpdateCredentials)
+
+ future.onSuccess {
+ case true =>
+ // Update credentials in the executor side only when AM is successfully renewed the
+ // Credentials.
+ synchronized {
+ executorDataMap.values.foreach(_.executorEndpoint.send(UpdateCredentials))
--- End diff --
Use ask instead of send.
This allows for consumers of the api to have reasonable confidence (based on Future's timeout) whether the update was propagated.
send could result in a task schedule before the update is processed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14789
**[Test build #68519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68519/consoleFull)** for PR 14789 at commit [`b1cac5c`](https://github.com/apache/spark/commit/b1cac5c95f3c7f77a8253a88aff48d2a1934072c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14789
Hey, I haven't had the chance to look at this closely, but this seems to just be an API to trigger the existing credential updater code in Spark, right?
When I filed SPARK-14743 I had a different use case in mind. When I talked about Oozie and HoS, Spark's credential updater would not be enabled. Those systems generally do not use Spark's credential updater, since they do not have the user's keytab. They have their own keytab, which they use to login to the KDC and generate the tokens, and the run the child applications using a proxy user. They need a way to give Spark those tokens after Spark has been started, which is different from having a way to trigger Spark's token updater mechanism.
I'm not sure I understand how something like Oozie or HoS would use the particular feature added with this change. They can't give their own keytab to the user's application, because the user code should not have access to that.
Tom:
> To get around keytabs, I was going to add an interface to push new credentials from the gateway box.
Yes basically that is more in line with what I had in mind.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14789
So I was intending to leave the keytab/principal stuff in there and just add the additional mechanism to push credentials. As long as you are ok with keytab from security point of view its easier for somethings.
I think both mechanisms could go through a similar type interface though which is why I brought it up here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org