You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2017/06/19 17:24:42 UTC
spark git commit: [SPARK-19688][STREAMING] Not to read
`spark.yarn.credentials.file` from checkpoint.
Repository: spark
Updated Branches:
refs/heads/master 9a145fd79 -> e92ffe6f1
[SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint.
## What changes were proposed in this pull request?
Reload the `spark.yarn.credentials.file` property when restarting a streaming application from checkpoint.
## How was this patch tested?
Manual tested with 1.6.3 and 2.1.1.
I didn't test this with master because of some compile problems, but I think it will be the same result.
## Notice
This should be merged into maintenance branches too.
jira: [SPARK-21008](https://issues.apache.org/jira/browse/SPARK-21008)
Author: saturday_s <sh...@gmail.com>
Closes #18230 from saturday-shi/SPARK-21008.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e92ffe6f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e92ffe6f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e92ffe6f
Branch: refs/heads/master
Commit: e92ffe6f1771e3fe9ea2e62ba552c1b5cf255368
Parents: 9a145fd
Author: saturday_s <sh...@gmail.com>
Authored: Mon Jun 19 10:24:29 2017 -0700
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Mon Jun 19 10:24:29 2017 -0700
----------------------------------------------------------------------
.../src/main/scala/org/apache/spark/streaming/Checkpoint.scala | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/e92ffe6f/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
----------------------------------------------------------------------
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala b/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
index 5cbad8b..b8c780d 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
@@ -55,6 +55,9 @@ class Checkpoint(ssc: StreamingContext, val checkpointTime: Time)
"spark.master",
"spark.yarn.keytab",
"spark.yarn.principal",
+ "spark.yarn.credentials.file",
+ "spark.yarn.credentials.renewalTime",
+ "spark.yarn.credentials.updateTime",
"spark.ui.filters")
val newSparkConf = new SparkConf(loadDefaults = false).setAll(sparkConfPairs)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org