You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2018/01/04 12:49:47 UTC
[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/20151
[SPARK-22959][PYTHON] Configuration to select the modules for daemon and worker in PySpark
## What changes were proposed in this pull request?
We are now forced to use `pyspark/daemon.py` and `pyspark/worker.py` in PySpark.
This doesn't allow a custom modification for it. For example, it's sometimes hard to debug what happens inside Python worker processes.
This is actually related with SPARK-7721 too as somehow Coverage is unable to detect the coverage from `os.fork`. If we have some custom fixes to force the coverage, it works fine.
This is also related with SPARK-20368. This JIRA describes Sentry support which (roughly) needs some changes within worker side.
With this configuration advanced users will be able to do a lot of pluggable workarounds and we can meet such potential needs in the future.
As an example, let's say if I configure the module `coverage_daemon` and had `coverage_daemon.py` in the python path:
```python
import os
from pyspark import daemon
if "COVERAGE_PROCESS_START" in os.environ:
from pyspark.worker import main
def _cov_wrapped(*args, **kwargs):
import coverage
cov = coverage.coverage(
config_file=os.environ["COVERAGE_PROCESS_START"])
cov.start()
try:
main(*args, **kwargs)
finally:
cov.stop()
cov.save()
daemon.worker_main = _cov_wrapped
if __name__ == '__main__':
daemon.manager()
```
More importantly, we can leave the main code intact but allow some workarounds.
## How was this patch tested?
Manually tested.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark configuration-daemon-worker
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20151.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20151
----
commit f74df4b566594152fa1efe1e3fb6033cbcf3993b
Author: hyukjinkwon <gu...@...>
Date: 2018-01-04T12:39:56Z
Configuration to select the modules for daemon and worker in PySpark
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #86075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86075/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/20151
Looks good. Let's wait for @rxin's response.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85917/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #85677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85677/testReport)** for PR 20151 at commit [`f74df4b`](https://github.com/apache/spark/commit/f74df4b566594152fa1efe1e3fb6033cbcf3993b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #85917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85917/testReport)** for PR 20151 at commit [`ea5b987`](https://github.com/apache/spark/commit/ea5b987d59f415045a2a890d9e4cf30198d82717).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86075/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20151#discussion_r160021803
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -34,17 +34,25 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
import PythonWorkerFactory._
- // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
- // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
- // only works on UNIX-based systems now because it uses signals for child management, so we can
- // also fall back to launching workers (pyspark/worker.py) directly.
+ // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
+ // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
+ // currently only works on UNIX-based systems now because it uses signals for child management,
+ // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
val useDaemon = {
val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
// This flag is ignored on Windows as it's unable to fork.
!System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
}
+ // This configuration indicates the module to run the daemon to execute its Python workers.
+ val daemonModule = SparkEnv.get.conf.get("spark.python.daemon.module", "pyspark.daemon")
--- End diff --
Ah, yup that's true in general. But please let me stick to "module" here as that's what we execute (`python -m`) describes:
```
python --help
...
-m mod : run library module as a script (terminates option list)
...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
Yup, thanks for all review @felixcheung and @ueshin BTW
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
Hey @rxin, I think I need your sign-off too as it's related with SPARK-7721.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
Yup, will write up some more warnings that says like it's expert only configuration, experimental and rather an internal configuration. Also, I will note that we should be super careful. Will update tonight (KST) :).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
I manually tested after setting `spark.python.daemon.module` to `nonexistantmodule`. It shows the error message like this:
```python
>>> spark.range(1).rdd.map(lambda x: x).collect()
```
```
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/rdd.py", line 824, in collect
port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
File "/.../spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__
File "/.../spark/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/.../spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost, executor driver): org.apache.spark.SparkException:
Error from python worker:
/usr/bin/python: No module named nonexistantmodule
PYTHONPATH was:
/.../spark/python/lib/pyspark.zip:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.3.0-SNAPSHOT.jar:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/python/:
java.io.EOFException
...
Driver stacktrace:
...
Caused by: org.apache.spark.SparkException:
Error from python worker:
/usr/bin/python: No module named nonexistantmodule
PYTHONPATH was:
/.../spark/python/lib/pyspark.zip:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.3.0-SNAPSHOT.jar:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/python/:
java.io.EOFException
...
... 1 more
18/01/08 15:54:06 WARN TaskSetManager: Lost task 6.0 in stage 0.0 (TID 6, localhost, executor driver): TaskKilled (Stage cancelled)
...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #85918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85918/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #85917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85917/testReport)** for PR 20151 at commit [`ea5b987`](https://github.com/apache/spark/commit/ea5b987d59f415045a2a890d9e4cf30198d82717).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20151#discussion_r160817581
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -34,17 +34,39 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
import PythonWorkerFactory._
- // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
- // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
- // only works on UNIX-based systems now because it uses signals for child management, so we can
- // also fall back to launching workers (pyspark/worker.py) directly.
+ // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
+ // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
+ // currently only works on UNIX-based systems now because it uses signals for child management,
+ // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
val useDaemon = {
val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
// This flag is ignored on Windows as it's unable to fork.
!System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
}
+ // WARN: Both configurations, 'spark.python.daemon.module' and 'spark.python.worker.module' are
+ // for very advanced users and they are experimental. This should be considered
+ // as expert-only option, and shouldn't be used before knowing what it means exactly.
+
+ // This configuration indicates the module to run the daemon to execute its Python workers.
+ val daemonModule = SparkEnv.get.conf.getOption("spark.python.daemon.module").map { value =>
--- End diff --
Do we need to restrict the module's package to only allow something like `pyspark.*`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20151
+1... this is "undocumented" conf, sooo it's an expert one :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #86075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86075/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20151
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20151#discussion_r160864049
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -34,17 +34,39 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
import PythonWorkerFactory._
- // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
- // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
- // only works on UNIX-based systems now because it uses signals for child management, so we can
- // also fall back to launching workers (pyspark/worker.py) directly.
+ // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
+ // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
+ // currently only works on UNIX-based systems now because it uses signals for child management,
+ // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
val useDaemon = {
val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
// This flag is ignored on Windows as it's unable to fork.
!System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
}
+ // WARN: Both configurations, 'spark.python.daemon.module' and 'spark.python.worker.module' are
+ // for very advanced users and they are experimental. This should be considered
+ // as expert-only option, and shouldn't be used before knowing what it means exactly.
+
+ // This configuration indicates the module to run the daemon to execute its Python workers.
+ val daemonModule = SparkEnv.get.conf.getOption("spark.python.daemon.module").map { value =>
--- End diff --
Hm, actually we could check like .. if it's empty string too. I wrote "shouldn't be used before knowing what it means exactly." above. So, I think it's fine.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
@rxin or @joshrosen could you guys take a quick look and see if it makes sense?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #85918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85918/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20151
**[Test build #85677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85677/testReport)** for PR 20151 at commit [`f74df4b`](https://github.com/apache/spark/commit/f74df4b566594152fa1efe1e3fb6033cbcf3993b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85677/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/20151#discussion_r159819670
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -34,17 +34,25 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
import PythonWorkerFactory._
- // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
- // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
- // only works on UNIX-based systems now because it uses signals for child management, so we can
- // also fall back to launching workers (pyspark/worker.py) directly.
+ // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
+ // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
+ // currently only works on UNIX-based systems now because it uses signals for child management,
+ // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
val useDaemon = {
val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
// This flag is ignored on Windows as it's unable to fork.
!System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
}
+ // This configuration indicates the module to run the daemon to execute its Python workers.
+ val daemonModule = SparkEnv.get.conf.get("spark.python.daemon.module", "pyspark.daemon")
--- End diff --
generally, I thought we use the name "command" as what we call the thing to execute
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
@holdenk, @rxin, @joshrosen and @ueshin, as you all might already know, I am working on Python coverage. Based on the top of this PR, I think we can leave the main codes intact while we properly track the coverage within worker processes. I believe this also partly covers SPARK-20368 too.
What do you guys think about this configuration?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
Will merge this one if there isn't any objection. I believe this doesn't affect the existing code path anyway ..
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/20151
The changes LGTM.
Btw, what if we miss the module in python path? Can we see that the error is caused by the missing module from the exception message?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20151
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/20151
So I think this could be the basis for solving a lot of related problems and I like the minimally invasive approach to it. I think the error message for setting it to a bad module rather than a nonexistent module is probably going to be very confusing. I think it would be good to make it clear that this is advanced setting we don't expect most users to modify directly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20151
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85918/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20151#discussion_r160661749
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -34,17 +34,39 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
import PythonWorkerFactory._
- // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
- // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
- // only works on UNIX-based systems now because it uses signals for child management, so we can
- // also fall back to launching workers (pyspark/worker.py) directly.
+ // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
+ // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
+ // currently only works on UNIX-based systems now because it uses signals for child management,
+ // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
val useDaemon = {
val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
// This flag is ignored on Windows as it's unable to fork.
!System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
}
+ // WARN: Both configurations, 'spark.python.daemon.module' and 'spark.python.worker.module' are
+ // for very advanced users and they are experimental. This should be considered
+ // as expert-only option, and shouldn't be used before knowing what it means exactly.
+
+ // This configuration indicates the module to run the daemon to execute its Python workers.
+ val daemonModule = SparkEnv.get.conf.getOption("spark.python.daemon.module").map { value =>
+ logInfo(
+ s"Python daemon module in PySpark is set to [$value] in 'spark.python.daemon.module', " +
+ "using this to start the daemon up. Note that this configuration only has an effect when " +
+ "'spark.python.use.daemon' is enabled and the platform is not Windows.")
--- End diff --
Just double checked it shows the log only when the configuration is explicitly set:
```
18/01/10 21:23:24 INFO PythonWorkerFactory: Python daemon module in PySpark is set to [pyspark.daemon] in
'spark.python.daemon.module', using this to start the daemon up. Note that this configuration only has an
effect when 'spark.python.use.daemon' is enabled and the platform is not Windows.
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org