spark git commit: [SPARK-22554][PYTHON] Add a config to control if PySpark should use daemon or not for workers

[gu...@apache.org]

Repository: spark
Updated Branches:
  refs/heads/master b10837ab1 -> 57c5514de


[SPARK-22554][PYTHON] Add a config to control if PySpark should use daemon or not for workers

## What changes were proposed in this pull request?

This PR proposes to add a flag to control if PySpark should use daemon or not.

Actually, SparkR already has a flag for useDaemon:
https://github.com/apache/spark/blob/478fbc866fbfdb4439788583281863ecea14e8af/core/src/main/scala/org/apache/spark/api/r/RRunner.scala#L362

It'd be great if we have this flag too. It makes easier to debug Windows specific issue.

## How was this patch tested?

Manually tested.

Author: hyukjinkwon <gu...@gmail.com>

Closes #19782 from HyukjinKwon/use-daemon-flag.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/57c5514d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/57c5514d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/57c5514d

Branch: refs/heads/master
Commit: 57c5514de9dba1c14e296f85fb13fef23ce8c73f
Parents: b10837a
Author: hyukjinkwon <gu...@gmail.com>
Authored: Mon Nov 20 13:34:06 2017 +0900
Committer: hyukjinkwon <gu...@gmail.com>
Committed: Mon Nov 20 13:34:06 2017 +0900

----------------------------------------------------------------------
 .../org/apache/spark/api/python/PythonWorkerFactory.scala     | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/57c5514d/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala b/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala
index fc595ae..f53c617 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala
@@ -38,7 +38,12 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
   // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
   // only works on UNIX-based systems now because it uses signals for child management, so we can
   // also fall back to launching workers (pyspark/worker.py) directly.
-  val useDaemon = !System.getProperty("os.name").startsWith("Windows")
+  val useDaemon = {
+    val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
+
+    // This flag is ignored on Windows as it's unable to fork.
+    !System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
+  }
 
   var daemon: Process = null
   val daemonHost = InetAddress.getByAddress(Array(127, 0, 0, 1))


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org