You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by zs...@apache.org on 2017/11/10 18:58:02 UTC
spark git commit: [SPARK-22294][DEPLOY] Reset
spark.driver.bindAddress when starting a Checkpoint
Repository: spark
Updated Branches:
refs/heads/master b70aa9e08 -> 5ebdcd185
[SPARK-22294][DEPLOY] Reset spark.driver.bindAddress when starting a Checkpoint
## What changes were proposed in this pull request?
It seems that recovering from a checkpoint can replace the old
driver and executor IP addresses, as the workload can now be taking
place in a different cluster configuration. It follows that the
bindAddress for the master may also have changed. Thus we should not be
keeping the old one, and instead be added to the list of properties to
reset and recreate from the new environment.
## How was this patch tested?
This patch was tested via manual testing on AWS, using the experimental (not yet merged) Kubernetes scheduler, which uses bindAddress to bind to a Kubernetes service (and thus was how I first encountered the bug too), but it is not a code-path related to the scheduler and this may have slipped through when merging SPARK-4563.
Author: Santiago Saavedra <ss...@openshine.com>
Closes #19427 from ssaavedra/fix-checkpointing-master.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5ebdcd18
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5ebdcd18
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5ebdcd18
Branch: refs/heads/master
Commit: 5ebdcd185f2108a90e37a1aa4214c3b6c69a97a4
Parents: b70aa9e
Author: Santiago Saavedra <ss...@openshine.com>
Authored: Fri Nov 10 10:57:58 2017 -0800
Committer: Shixiong Zhu <zs...@gmail.com>
Committed: Fri Nov 10 10:57:58 2017 -0800
----------------------------------------------------------------------
.../src/main/scala/org/apache/spark/streaming/Checkpoint.scala | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/5ebdcd18/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
----------------------------------------------------------------------
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala b/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
index 3cfbced..aed67a5 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
@@ -51,6 +51,7 @@ class Checkpoint(ssc: StreamingContext, val checkpointTime: Time)
"spark.yarn.app.id",
"spark.yarn.app.attemptId",
"spark.driver.host",
+ "spark.driver.bindAddress",
"spark.driver.port",
"spark.master",
"spark.yarn.jars",
@@ -64,6 +65,7 @@ class Checkpoint(ssc: StreamingContext, val checkpointTime: Time)
val newSparkConf = new SparkConf(loadDefaults = false).setAll(sparkConfPairs)
.remove("spark.driver.host")
+ .remove("spark.driver.bindAddress")
.remove("spark.driver.port")
val newReloadConf = new SparkConf(loadDefaults = true)
propertiesToReload.foreach { prop =>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org