You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by SaintBacchus <gi...@git.apache.org> on 2015/09/22 13:04:55 UTC
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
GitHub user SaintBacchus opened a pull request:
https://github.com/apache/spark/pull/8867
[SPARK-10755][YARN]Set driver also update the token for long-running application
In the yarn-client mode, driver will write the event logs into hdfs and get the partition information from hdfs, so it's nessary to update the token from the `AMDelegationTokenRenewer`.
In the yarn-cluster mode, driver is company with AM and token will update by AM. But it's still better to update the token for client process since the client wants to delete the staging dir with a expired token.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/SaintBacchus/spark SPARK-10755
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8867.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8867
----
commit 00fd0bc4cd2d6b31ba197629fbe1e9e07a2497bc
Author: huangzhaowei <hu...@huawei.com>
Date: 2015-09-22T11:00:47Z
[SPARK_10755][YARN]Set driver also update the token for long-running application.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40705866
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala ---
@@ -76,7 +76,10 @@ private[spark] class ExecutorDelegationTokenUpdater(
SparkHadoopUtil.get.getTimeFromNowToRenewal(
sparkConf, 0.8, UserGroupInformation.getCurrentUser.getCredentials)
if (timeFromNowToRenewal <= 0) {
- executorUpdaterRunnable.run()
+ // If the `timeFromNowToRenewal` equals 0, it's better to wait 1 minutes to schedule for
+ // avoid cycle calling and cause StackOverflow Exception.
--- End diff --
We end up scheduling the next run of the `executorUpdaterRunnable` from within this class. If the executor renewal interval is set to 0, we end up calling it again from within the same thread (indefinitely). But I am not sure why it would be 0 here though (since this is started only if the login is from keytab).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SaintBacchus <gi...@git.apache.org>.
Github user SaintBacchus commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-144262717
@tgravescs @harishreedharan this fix will still loss the event log, maybe it's not a better approach.
so we had raise a new [approach](https://github.com/apache/spark/pull/8942) to resolve this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-143012421
@tgravescs So seems like this would fix that issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142252552
[Test build #42829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42829/consoleFull) for PR 8867 at commit [`00fd0bc`](https://github.com/apache/spark/commit/00fd0bc4cd2d6b31ba197629fbe1e9e07a2497bc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SaintBacchus <gi...@git.apache.org>.
Github user SaintBacchus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40750990
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -544,6 +545,7 @@ private[spark] class Client(
logInfo(s"Credentials file set to: $credentialsFile")
val renewalInterval = getTokenRenewalInterval(stagingDirPath)
sparkConf.set("spark.yarn.token.renewal.interval", renewalInterval.toString)
+ SparkHadoopUtil.get.startExecutorDelegationTokenRenewer(sparkConf)
--- End diff --
This code had change the configuration `spark.yarn.credentials.file`, and this configuration would be used in `DelegationTokenUpdate`, so it had to put the `start` after this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40201021
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala ---
@@ -76,7 +76,10 @@ private[spark] class ExecutorDelegationTokenUpdater(
SparkHadoopUtil.get.getTimeFromNowToRenewal(
sparkConf, 0.8, UserGroupInformation.getCurrentUser.getCredentials)
if (timeFromNowToRenewal <= 0) {
- executorUpdaterRunnable.run()
+ // If the `timeFromNowToRenewal` equals 0, it's better to wait 1 minutes to schedule for
+ // avoid cycle calling and cause StackOverflow Exception.
--- End diff --
can you explain this, why do we get exception if it runs immediately?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-149372740
OK, I think I know the issue - the reason is probably that the credentials are cached in the `FileSystem` instance using which the write happens. Since we are replacing the credentials but not the `FileSystem` instance itself this might not work, which is why #8942 works. We can do with either that approach or we can replace the `FileSystem` instance which would require a close and reopen of the file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142251046
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SaintBacchus <gi...@git.apache.org>.
Github user SaintBacchus commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142516215
@harishreedharan I set `fs.hdfs.impl.disable.cache` to avoid cache mechanism in the hadoop.
I tested in the yarn-client mode, if I apply this pr the application will be OK and I remove it the application goes down.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by dengzhhu653 <gi...@git.apache.org>.
Github user dengzhhu653 commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-163601969
Agree with @harishreedharan. The client's rpc connection lacks awareness of token op after setting up, it needs to rebuild the connection to make the new token effective.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SaintBacchus <gi...@git.apache.org>.
Github user SaintBacchus commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142970395
@tgravescs I wrote a simple `DFSCliet` application to continuously write string into hdfs and this can work over the configuration `dfs.namenode.delegation.token.max-lifetime`.
So I open the DEBUG logging and find some regularities:
If using the **KERBEROS** to gain the authority of the `NameNode`, it can work over the it.
> 15/09/24 19:53:38 DEBUG SaslRpcClient: Use **KERBEROS** authentication for protocol ClientNamenodeProtocolPB
But if using **TOKEN**, the application may existed with *token expired exception*.
> 15/09/24 19:53:58 DEBUG SaslRpcClient: Use **TOKEN** authentication for protocol ClientNamenodeProtocolPB
Spark was using the *Token*.
One way to resolve this issue is that login with keytab again then the mode of the `SaslRpcClient` will be changed into *KERBEROS*.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40679882
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala ---
@@ -76,7 +76,10 @@ private[spark] class ExecutorDelegationTokenUpdater(
SparkHadoopUtil.get.getTimeFromNowToRenewal(
sparkConf, 0.8, UserGroupInformation.getCurrentUser.getCredentials)
if (timeFromNowToRenewal <= 0) {
- executorUpdaterRunnable.run()
+ // If the `timeFromNowToRenewal` equals 0, it's better to wait 1 minutes to schedule for
+ // avoid cycle calling and cause StackOverflow Exception.
--- End diff --
can you explain?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142976451
thanks for looking into that @SaintBacchus, that makes sense. I thought we were logging in from the keytab again which would then switch it to be logged in from keytab - see SparkSubmit:
// Let YARN know it's a pyspark app, so it distributes needed libraries.
if (clusterManager == YARN) {
if (args.isPython) {
sysProps.put("spark.yarn.isPython", "true")
}
if (args.principal != null) {
require(args.keytab != null, "Keytab must be specified when the keytab is specified")
UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)
}
}
perhaps that is somehow not changing the login type in the UserGroupInformation though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142596642
relogin from keytab should work past 7 days. as its a keytab not just renewing tokens.
@SaintBacchus if you change dfs.namenode.delegation.token.renew-interval to be smaller then max-life does it error out there or does it continue and only at max-life fail?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142495294
Hmm, this might actually be due to a hadoop bug. Even with a login we add tokens at the driver (in `Client.scala`) and I have hit this before -- when there are tokens present in the credentials cache, the re-login does not seem to help. The tokens still take precedence and an expired tokens exception is thrown. I think this is the issue here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142425148
I'll have to look at in more detail but in yarn cluster mode the client can go away and things run just fine so you can't rely on it being there to update tokens. If this is just updating from the one AM already gets its fine. Also note that most of the time the AM cleans up staging dir. There are exception cases it doesn't.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-143322902
@SaintBacchus Right. That is what I was talking about above. I think your fix takes care of this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142425525
I think the concern is client mode. Since the driver must still get new tokens so it can write event logs and such out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SaintBacchus <gi...@git.apache.org>.
Github user SaintBacchus closed the pull request at:
https://github.com/apache/spark/pull/8867
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142428624
ok, I'll have a look.. If the user logged in with keytab before launching then hadoop should automatically re login you in if you are using hadoop rpc and I thought you automatically logged in the user with keytab.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40708505
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -544,6 +545,7 @@ private[spark] class Client(
logInfo(s"Credentials file set to: $credentialsFile")
val renewalInterval = getTokenRenewalInterval(stagingDirPath)
sparkConf.set("spark.yarn.token.renewal.interval", renewalInterval.toString)
+ SparkHadoopUtil.get.startExecutorDelegationTokenRenewer(sparkConf)
--- End diff --
This depends on the lines immediately above this though. `setupCredentials` gets called way before this I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40710630
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala ---
@@ -76,7 +76,10 @@ private[spark] class ExecutorDelegationTokenUpdater(
SparkHadoopUtil.get.getTimeFromNowToRenewal(
sparkConf, 0.8, UserGroupInformation.getCurrentUser.getCredentials)
if (timeFromNowToRenewal <= 0) {
- executorUpdaterRunnable.run()
+ // If the `timeFromNowToRenewal` equals 0, it's better to wait 1 minutes to schedule for
+ // avoid cycle calling and cause StackOverflow Exception.
--- End diff --
Oh I see, I assume someone manually changed the renew time for testing purposes? I think perhaps the comment should be clarified saying if the renewal time is very small wait 1 minute to avoid calling over and over very quickly. Or alternatively where we set spark.yarn.token.renewal.interval we could just do a min of 1 minute.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142411569
This LGTM.
@tgravescs - Please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142296247
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42829/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SaintBacchus <gi...@git.apache.org>.
Github user SaintBacchus commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142468611
@harishreedharan @tgravescs Hadoop RPC actually will do the re-login with the keytab but the token only can persist 7 days by default. So it must be updated.
The test step is below:
1. shorter the configutation `dfs.namenode.delegation.token.max-lifetime` and `dfs.namenode.delegation.token.renew-interval`, maybe 10min
2.start a `spark-shell` or `spark-sql`
3.After 15min, execuse a job
Then the application will fail with token expired exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142429036
Oh, if the hadoop RPC did the re-login we should be ok. In that case the tokens should not matter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142296114
[Test build #42829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42829/console) for PR 8867 at commit [`00fd0bc`](https://github.com/apache/spark/commit/00fd0bc4cd2d6b31ba197629fbe1e9e07a2497bc).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142296244
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142251019
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-142599144
@SaintBacchus when you launch your job do you kinit first? did you use keytab to kinit? If you turn on debug logging for hadoop do you see anything like "Saving TGT from keytab to ticket cache".
If its actually relogging in from keytab then you should also see something like "Initiating logout for".. and then "Initiating re-login for" with DEBUG logging on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40680554
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -544,6 +545,7 @@ private[spark] class Client(
logInfo(s"Credentials file set to: $credentialsFile")
val renewalInterval = getTokenRenewalInterval(stagingDirPath)
sparkConf.set("spark.yarn.token.renewal.interval", renewalInterval.toString)
+ SparkHadoopUtil.get.startExecutorDelegationTokenRenewer(sparkConf)
--- End diff --
I would rather move this somewhere else. It doesn't really have anything to do with setting up the launch env. Perhaps move it to setupCredentials()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by SaintBacchus <gi...@git.apache.org>.
Github user SaintBacchus commented on the pull request:
https://github.com/apache/spark/pull/8867#issuecomment-143173483
@tgravescs I had noticed the code of `UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)`
After this login motivation, `yarn.Client` will change *KERBEROS* into *TOKEN* for the purpose of setting the token for the AM.
```scala
/** Set up security tokens for launching our ApplicationMaster container. */
private def setupSecurityToken(amContainer: ContainerLaunchContext): Unit = {
val dob = new DataOutputBuffer
credentials.writeTokenStorageToStream(dob)
amContainer.setTokens(ByteBuffer.wrap(dob.getData))
}
```
After this, the `Client` will use *TOKEN* in the RPC connection.
If I login again with keytab after this, the SAALClient will use *KERBEROS* again, and this can avoid token expired exception.
I had tested the recent spark and it still will throw this exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10755][YARN]Set driver also update the ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/8867#discussion_r40710647
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -544,6 +545,7 @@ private[spark] class Client(
logInfo(s"Credentials file set to: $credentialsFile")
val renewalInterval = getTokenRenewalInterval(stagingDirPath)
sparkConf.set("spark.yarn.token.renewal.interval", renewalInterval.toString)
+ SparkHadoopUtil.get.startExecutorDelegationTokenRenewer(sparkConf)
--- End diff --
ok, this is fine then
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org