You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2018/01/04 12:49:47 UTC

[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/20151

    [SPARK-22959][PYTHON] Configuration to select the modules for daemon and worker in PySpark

    ## What changes were proposed in this pull request?
    
    We are now forced to use `pyspark/daemon.py` and `pyspark/worker.py` in PySpark.
    
    This doesn't allow a custom modification for it. For example, it's sometimes hard to debug what happens inside Python worker processes.
    
    This is actually related with SPARK-7721 too as somehow Coverage is unable to detect the coverage from `os.fork`. If we have some custom fixes to force the coverage, it works fine.
    
    This is also related with SPARK-20368. This JIRA describes Sentry support which (roughly) needs some changes within worker side. 
    
    With this configuration advanced users will be able to do a lot of pluggable workarounds and we can meet such potential needs in the future.
    
    As an example, let's say if I configure the module `coverage_daemon` and had `coverage_daemon.py` in the python path:
    
    ```python
    import os
    
    from pyspark import daemon
    
    
    if "COVERAGE_PROCESS_START" in os.environ:
        from pyspark.worker import main
    
        def _cov_wrapped(*args, **kwargs):
            import coverage
            cov = coverage.coverage(
                config_file=os.environ["COVERAGE_PROCESS_START"])
            cov.start()
            try:
                main(*args, **kwargs)
            finally:
                cov.stop()
                cov.save()
        daemon.worker_main = _cov_wrapped
    
    
    if __name__ == '__main__':
        daemon.manager()
    ```
    
    More importantly, we can leave the main code intact but allow some workarounds.
    
    ## How was this patch tested?
    
    Manually tested.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark configuration-daemon-worker

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20151.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20151
    
----
commit f74df4b566594152fa1efe1e3fb6033cbcf3993b
Author: hyukjinkwon <gu...@...>
Date:   2018-01-04T12:39:56Z

    Configuration to select the modules for daemon and worker in PySpark

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #86075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86075/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Looks good. Let's wait for @rxin's response.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85917/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #85677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85677/testReport)** for PR 20151 at commit [`f74df4b`](https://github.com/apache/spark/commit/f74df4b566594152fa1efe1e3fb6033cbcf3993b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #85917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85917/testReport)** for PR 20151 at commit [`ea5b987`](https://github.com/apache/spark/commit/ea5b987d59f415045a2a890d9e4cf30198d82717).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86075/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20151#discussion_r160021803
  
    --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
    @@ -34,17 +34,25 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
     
       import PythonWorkerFactory._
     
    -  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
    -  // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
    -  // only works on UNIX-based systems now because it uses signals for child management, so we can
    -  // also fall back to launching workers (pyspark/worker.py) directly.
    +  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
    +  // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
    +  // currently only works on UNIX-based systems now because it uses signals for child management,
    +  // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
       val useDaemon = {
         val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
     
         // This flag is ignored on Windows as it's unable to fork.
         !System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
       }
     
    +  // This configuration indicates the module to run the daemon to execute its Python workers.
    +  val daemonModule = SparkEnv.get.conf.get("spark.python.daemon.module", "pyspark.daemon")
    --- End diff --
    
    Ah, yup that's true in general. But please let me stick to "module" here as that's what we execute (`python -m`) describes:
    
    ```
    python --help
    ...
    -m mod : run library module as a script (terminates option list)
    ...
    ```



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Yup, thanks for all review @felixcheung and @ueshin BTW


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Hey @rxin, I think I need your sign-off too as it's related with SPARK-7721.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Yup, will write up some more warnings that says like it's expert only configuration, experimental and rather an internal configuration. Also, I will note that we should be super careful. Will update tonight (KST) :).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    I manually tested after setting `spark.python.daemon.module` to `nonexistantmodule`. It shows the error message like this:
    
    ```python
    >>> spark.range(1).rdd.map(lambda x: x).collect()
    ```
    
    ```
    ...
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/.../spark/python/pyspark/rdd.py", line 824, in collect
        port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
      File "/.../spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__
      File "/.../spark/python/pyspark/sql/utils.py", line 63, in deco
        return f(*a, **kw)
      File "/.../spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value
    py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
    : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost, executor driver): org.apache.spark.SparkException:
    Error from python worker:
      /usr/bin/python: No module named nonexistantmodule
    PYTHONPATH was:
      /.../spark/python/lib/pyspark.zip:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.3.0-SNAPSHOT.jar:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/python/:
    java.io.EOFException
        ...
    
    Driver stacktrace:
        ...
    Caused by: org.apache.spark.SparkException:
    Error from python worker:
      /usr/bin/python: No module named nonexistantmodule
    PYTHONPATH was:
      /.../spark/python/lib/pyspark.zip:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.3.0-SNAPSHOT.jar:/.../spark/python/lib/py4j-0.10.6-src.zip:/.../spark/python/:
    java.io.EOFException
        ...
        ... 1 more
    
    18/01/08 15:54:06 WARN TaskSetManager: Lost task 6.0 in stage 0.0 (TID 6, localhost, executor driver): TaskKilled (Stage cancelled)
    ...
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #85918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85918/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #85917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85917/testReport)** for PR 20151 at commit [`ea5b987`](https://github.com/apache/spark/commit/ea5b987d59f415045a2a890d9e4cf30198d82717).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20151#discussion_r160817581
  
    --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
    @@ -34,17 +34,39 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
     
       import PythonWorkerFactory._
     
    -  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
    -  // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
    -  // only works on UNIX-based systems now because it uses signals for child management, so we can
    -  // also fall back to launching workers (pyspark/worker.py) directly.
    +  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
    +  // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
    +  // currently only works on UNIX-based systems now because it uses signals for child management,
    +  // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
       val useDaemon = {
         val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
     
         // This flag is ignored on Windows as it's unable to fork.
         !System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
       }
     
    +  // WARN: Both configurations, 'spark.python.daemon.module' and 'spark.python.worker.module' are
    +  // for very advanced users and they are experimental. This should be considered
    +  // as expert-only option, and shouldn't be used before knowing what it means exactly.
    +
    +  // This configuration indicates the module to run the daemon to execute its Python workers.
    +  val daemonModule = SparkEnv.get.conf.getOption("spark.python.daemon.module").map { value =>
    --- End diff --
    
    Do we need to restrict the module's package to only allow something like `pyspark.*`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    +1... this is "undocumented" conf, sooo it's an expert one :)



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #86075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86075/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20151


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20151#discussion_r160864049
  
    --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
    @@ -34,17 +34,39 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
     
       import PythonWorkerFactory._
     
    -  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
    -  // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
    -  // only works on UNIX-based systems now because it uses signals for child management, so we can
    -  // also fall back to launching workers (pyspark/worker.py) directly.
    +  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
    +  // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
    +  // currently only works on UNIX-based systems now because it uses signals for child management,
    +  // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
       val useDaemon = {
         val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
     
         // This flag is ignored on Windows as it's unable to fork.
         !System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
       }
     
    +  // WARN: Both configurations, 'spark.python.daemon.module' and 'spark.python.worker.module' are
    +  // for very advanced users and they are experimental. This should be considered
    +  // as expert-only option, and shouldn't be used before knowing what it means exactly.
    +
    +  // This configuration indicates the module to run the daemon to execute its Python workers.
    +  val daemonModule = SparkEnv.get.conf.getOption("spark.python.daemon.module").map { value =>
    --- End diff --
    
    Hm, actually we could check like .. if it's empty string too. I wrote "shouldn't be used before knowing what it means exactly." above. So, I think it's fine.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    @rxin or @joshrosen could you guys take a quick look and see if it makes sense?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #85918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85918/testReport)** for PR 20151 at commit [`fc65803`](https://github.com/apache/spark/commit/fc658034639c1aa56ff5b9a44624cad05377fe51).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    **[Test build #85677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85677/testReport)** for PR 20151 at commit [`f74df4b`](https://github.com/apache/spark/commit/f74df4b566594152fa1efe1e3fb6033cbcf3993b).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85677/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20151#discussion_r159819670
  
    --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
    @@ -34,17 +34,25 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
     
       import PythonWorkerFactory._
     
    -  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
    -  // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
    -  // only works on UNIX-based systems now because it uses signals for child management, so we can
    -  // also fall back to launching workers (pyspark/worker.py) directly.
    +  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
    +  // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
    +  // currently only works on UNIX-based systems now because it uses signals for child management,
    +  // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
       val useDaemon = {
         val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
     
         // This flag is ignored on Windows as it's unable to fork.
         !System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
       }
     
    +  // This configuration indicates the module to run the daemon to execute its Python workers.
    +  val daemonModule = SparkEnv.get.conf.get("spark.python.daemon.module", "pyspark.daemon")
    --- End diff --
    
    generally, I thought we use the name "command" as what we call the thing to execute 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    @holdenk, @rxin, @joshrosen and @ueshin,  as you all might already know, I am working on Python coverage. Based on the top of this PR, I think we can leave the main codes intact while we properly track the coverage within worker processes. I believe this also partly covers SPARK-20368 too.
    
    What do you guys think about this configuration?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Will merge this one if there isn't any objection. I believe this doesn't affect the existing code path anyway .. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    The changes LGTM.
    Btw, what if we miss the module in python path? Can we see that the error is caused by the missing module from the exception message?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    So I think this could be the basis for solving a lot of related problems and I like the minimally invasive approach to it. I think the error message for setting it to a bad module rather than a nonexistent module is probably going to be very confusing. I think it would be good to make it clear that this is advanced setting we don't expect most users to modify directly.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20151
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85918/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20151: [SPARK-22959][PYTHON] Configuration to select the...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20151#discussion_r160661749
  
    --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
    @@ -34,17 +34,39 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
     
       import PythonWorkerFactory._
     
    -  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon
    -  // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
    -  // only works on UNIX-based systems now because it uses signals for child management, so we can
    -  // also fall back to launching workers (pyspark/worker.py) directly.
    +  // Because forking processes from Java is expensive, we prefer to launch a single Python daemon,
    +  // pyspark/daemon.py (by default) and tell it to fork new workers for our tasks. This daemon
    +  // currently only works on UNIX-based systems now because it uses signals for child management,
    +  // so we can also fall back to launching workers, pyspark/worker.py (by default) directly.
       val useDaemon = {
         val useDaemonEnabled = SparkEnv.get.conf.getBoolean("spark.python.use.daemon", true)
     
         // This flag is ignored on Windows as it's unable to fork.
         !System.getProperty("os.name").startsWith("Windows") && useDaemonEnabled
       }
     
    +  // WARN: Both configurations, 'spark.python.daemon.module' and 'spark.python.worker.module' are
    +  // for very advanced users and they are experimental. This should be considered
    +  // as expert-only option, and shouldn't be used before knowing what it means exactly.
    +
    +  // This configuration indicates the module to run the daemon to execute its Python workers.
    +  val daemonModule = SparkEnv.get.conf.getOption("spark.python.daemon.module").map { value =>
    +    logInfo(
    +      s"Python daemon module in PySpark is set to [$value] in 'spark.python.daemon.module', " +
    +      "using this to start the daemon up. Note that this configuration only has an effect when " +
    +      "'spark.python.use.daemon' is enabled and the platform is not Windows.")
    --- End diff --
    
    Just double checked it shows the log only when the configuration is explicitly set:
    
    ```
    18/01/10 21:23:24 INFO PythonWorkerFactory: Python daemon module in PySpark is set to [pyspark.daemon] in
     'spark.python.daemon.module', using this to start the daemon up. Note that this configuration only has an 
    effect when 'spark.python.use.daemon' is enabled and the platform is not Windows.
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org