You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by zjffdu <gi...@git.apache.org> on 2016/05/17 09:48:27 UTC

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

GitHub user zjffdu opened a pull request:

    https://github.com/apache/spark/pull/13146

    [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonExec of driver and executor through conf…

    ## What changes were proposed in this pull request?
    
    (Please fill in changes proposed in this fix)
    
    Before this PR, user have to export environment variable to specify the python of driver & executor which is not so convenient for users. This PR is trying to allow user to specify python through configuration "--pyspark-driver-python" & "--pyspark-executor-python"
    
    ## How was this patch tested?
    
    Manually test in local & yarn mode for pyspark-shell and pyspark batch mode. 
    
    (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
    
    
    (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
    
    
    …and executor through configuration

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zjffdu/spark SPARK-13081

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13146.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13146
    
----
commit 62db33040a2f3ab9bea942d88c820beabf39977f
Author: Jeff Zhang <zj...@apache.org>
Date:   2016-05-17T08:07:06Z

    [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonExec of driver and executor through configuration

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonEx...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13146
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set p...

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13146#discussion_r73975182
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java ---
    @@ -294,11 +294,23 @@
         appResource = PYSPARK_SHELL_RESOURCE;
         constructEnvVarArgs(env, "PYSPARK_SUBMIT_ARGS");
     
    -    // The executable is the PYSPARK_DRIVER_PYTHON env variable set by the pyspark script,
    -    // followed by PYSPARK_DRIVER_PYTHON_OPTS.
    +    // Will pick up the binary executable in the following order
    +    // 1. conf spark.pyspark.driver.python
    +    // 2. conf spark.pyspark.python
    +    // 3. environment variable PYSPARK_DRIVER_PYTHON
    +    // 4. environment variable PYSPARK_PYTHON
    +    // 5. python
         List<String> pyargs = new ArrayList<>();
    -    pyargs.add(firstNonEmpty(System.getenv("PYSPARK_DRIVER_PYTHON"), "python"));
    +    pyargs.add(firstNonEmpty(conf.get(SparkLauncher.PYSPARK_DRIVER_PYTHON),
    +            conf.get(SparkLauncher.PYSPARK_PYTHON),
    +            System.getenv("PYSPARK_DRIVER_PYTHON"),
    +            System.getenv("PYSPARK_PYTHON"),
    +            "python"));
         String pyOpts = System.getenv("PYSPARK_DRIVER_PYTHON_OPTS");
    +    if (conf.containsKey(SparkLauncher.PYSPARK_PYTHON)) {
    --- End diff --
    
    Nevermind. This seems to be the used by pyspark to define the executor's python interpreter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonEx...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13146
  
    **[Test build #63403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63403/consoleFull)** for PR 13146 at commit [`1e9053a`](https://github.com/apache/spark/commit/1e9053a9036d8405abf5c06c772eef80f4f50ca4).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonEx...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13146
  
    **[Test build #63566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63566/consoleFull)** for PR 13146 at commit [`4fa7bda`](https://github.com/apache/spark/commit/4fa7bdaa2646acf487f96770f9dc22833e5d1c7e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set p...

Posted by zjffdu <gi...@git.apache.org>.

Github user zjffdu closed the pull request at:

    https://github.com/apache/spark/pull/13146


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonEx...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13146
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonEx...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13146
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13146: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonEx...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13146
  
    **[Test build #63566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63566/consoleFull)** for PR 13146 at commit [`4fa7bda`](https://github.com/apache/spark/commit/4fa7bdaa2646acf487f96770f9dc22833e5d1c7e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org