You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2019/04/02 16:23:28 UTC
[spark] branch branch-2.4 updated: [SPARK-26998][CORE] Remove SSL
configuration from executors
This is an automated email from the ASF dual-hosted git repository.
vanzin pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 55e6f7a [SPARK-26998][CORE] Remove SSL configuration from executors
55e6f7a is described below
commit 55e6f7ab2cc5b9e1a816929504471390fd0872c2
Author: Gabor Somogyi <ga...@gmail.com>
AuthorDate: Tue Apr 2 09:18:43 2019 -0700
[SPARK-26998][CORE] Remove SSL configuration from executors
## What changes were proposed in this pull request?
Different SSL passwords shown up as command line argument on executor side in standalone mode:
* keyStorePassword
* keyPassword
* trustStorePassword
In this PR I've removed SSL configurations from executors.
## How was this patch tested?
Existing + additional unit tests.
Additionally tested with standalone mode and checked the command line arguments:
```
[gaborsomogyi:~/spark] SPARK-26998(+4/-0,3)+ ± jps
94803 CoarseGrainedExecutorBackend
94818 Jps
90149 RemoteMavenServer
91925 Nailgun
94793 SparkSubmit
94680 Worker
94556 Master
398
[gaborsomogyi:~/spark] SPARK-26998(+4/-1,3)+ ± ps -ef | egrep "94556|94680|94793|94803"
502 94556 1 0 2:02PM ttys007 0:07.39 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host gsomogyi-MBP.local --port 7077 --webui-port 8080 --properties-file conf/spark-defaults.conf
502 94680 1 0 2:02PM ttys007 0:07.27 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 --properties-file conf/spark-defaults.conf spark://gsomogyi-MBP.local:7077
502 94793 94782 0 2:02PM ttys007 0:35.52 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Dscala.usejavacp=true -Xmx1g org.apache.spark.deploy.SparkSubmit --master spark://gsomogyi-MBP.local:7077 --class org.apache.spark.repl.Main --name Spark shell spark-shell
502 94803 94680 0 2:03PM ttys007 0:05.20 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Xmx1024M -Dspark.ssl.ui.port=0 -Dspark.driver.port=60902 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler172.30.65.186:60902 --executor-id 0 --hostname 172.30.65.186 --cores 8 --app-id app-20190326140311-0000 --worker-u [...]
502 94910 57352 0 2:05PM ttys008 0:00.00 egrep 94556|94680|94793|94803
```
Closes #24170 from gaborgsomogyi/SPARK-26998.
Authored-by: Gabor Somogyi <ga...@gmail.com>
Signed-off-by: Marcelo Vanzin <va...@cloudera.com>
(cherry picked from commit 57aff93886ac7d02b88294672ce0d2495b0942b8)
Signed-off-by: Marcelo Vanzin <va...@cloudera.com>
---
core/src/main/scala/org/apache/spark/SparkConf.scala | 1 -
core/src/test/scala/org/apache/spark/SparkConfSuite.scala | 11 +++++++++++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkConf.scala b/core/src/main/scala/org/apache/spark/SparkConf.scala
index ccafe16..0b24fe2 100644
--- a/core/src/main/scala/org/apache/spark/SparkConf.scala
+++ b/core/src/main/scala/org/apache/spark/SparkConf.scala
@@ -751,7 +751,6 @@ private[spark] object SparkConf extends Logging {
*/
def isExecutorStartupConf(name: String): Boolean = {
(name.startsWith("spark.auth") && name != SecurityManager.SPARK_AUTH_SECRET_CONF) ||
- name.startsWith("spark.ssl") ||
name.startsWith("spark.rpc") ||
name.startsWith("spark.network") ||
isSparkPortConf(name)
diff --git a/core/src/test/scala/org/apache/spark/SparkConfSuite.scala b/core/src/test/scala/org/apache/spark/SparkConfSuite.scala
index dcbfc93..66462de 100644
--- a/core/src/test/scala/org/apache/spark/SparkConfSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkConfSuite.scala
@@ -339,6 +339,17 @@ class SparkConfSuite extends SparkFunSuite with LocalSparkContext with ResetSyst
}
}
+ test("SPARK-26998: SSL configuration not needed on executors") {
+ val conf = new SparkConf(false)
+ conf.set("spark.ssl.enabled", "true")
+ conf.set("spark.ssl.keyPassword", "password")
+ conf.set("spark.ssl.keyStorePassword", "password")
+ conf.set("spark.ssl.trustStorePassword", "password")
+
+ val filtered = conf.getAll.filter { case (k, _) => SparkConf.isExecutorStartupConf(k) }
+ assert(filtered.isEmpty)
+ }
+
test("SPARK-27244 toDebugString redacts sensitive information") {
val conf = new SparkConf(loadDefaults = false)
.set("dummy.password", "dummy-password")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org