You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by KevinGrealish <gi...@git.apache.org> on 2016/06/21 21:55:29 UTC

[GitHub] spark pull request #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python ...

GitHub user KevinGrealish opened a pull request:

    https://github.com/apache/spark/pull/13824

    [SPARK-16110][YARN][PYSPARK] Fix allowing python version to be speicifed per submit for cluster mode.

    ## What changes were proposed in this pull request?
    
    This fix allows submit of pyspark jobs to specify python 2 or 3.
    
    Change ordering in setup for application master environment so env vars PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON can be overridden by spark.yarn.appMasterEnv.* conf settings. This applies to YARN in cluster mode. This allows them to be set per submission without needing the unset the env vars (which is not always possible - e.g. batch submit with LIVY only exposes the arguments to spark-submit)
    
    ## How was this patch tested?
    Manual and existing unit tests.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/KevinGrealish/spark SPARK-16110

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13824
    
----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62899/consoleFull)** for PR 13824 at commit [`6791edc`](https://github.com/apache/spark/commit/6791edc27a8b33bdf8c516a8cde5f431f6b0b07e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62935/consoleFull)** for PR 13824 at commit [`7ad3d77`](https://github.com/apache/spark/commit/7ad3d77968338b0747a134752f4af658b47b0cfe).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    @Vanzin I looked at refactoring the client to be more amenable to more granular unit testing, including using the mockable env var on conf you mentioned, but concluded it would be too disruptive to be part of this simple bug fix. I have included a test (that found an issue) and believe this should be good to go now. I'm not sure why the spark.yarn.appMasterEnv was do java path style appending, but that appears to be simply incorrect and was changed to a plain set.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    @Vanzin, I think we are good to good now. Agree?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    I think there are two ways to solve the problem that might be a little better...
    
    The first is to try to keep the current behavior. If, in L734 (where you're removing the current code that checks the `appMasterEnv` conf), you make a copy of the keys that were added to the `env` variable, you can then apply `addPathToEnvironment` just to the keys that are there. That means that the code will merge any user configuration with env variables created by Spark itself; otherwise it will use the user's override.
    
    The second is to read certain env variables using a special method that first looks at `spark.yarn.appMasterEnv.FOO` and if it doesn't exist, `sys.env("FOO")`. Then you could modify the code that currently reads `PYSPARK_DRIVER_PYTHON` and friends using that new method, instead of directly peeking at `sys.env`. You could also apply that new method to the code that currently reads `PYTHONPATH`.
    
    I think the latter is a better solution than you currently have, since it avoids hardcoding these env variable names in more places.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    You could change that code to use `sparkConf.getenv`, since there's a SparkConf instance available.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62899/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #61407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61407/consoleFull)** for PR 13824 at commit [`4ad7904`](https://github.com/apache/spark/commit/4ad790415dc504303e64c1c704655dc72af75fb1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python ...

Posted by KevinGrealish <gi...@git.apache.org>.
GitHub user KevinGrealish reopened a pull request:

    https://github.com/apache/spark/pull/13824

    [SPARK-16110][YARN][PYSPARK] Fix allowing python version to be speicifed per submit for cluster mode.

    ## What changes were proposed in this pull request?
    
    This fix allows submit of pyspark jobs to specify python 2 or 3.
    
    Change ordering in setup for application master environment so env vars PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON can be overridden by spark.yarn.appMasterEnv.* conf settings. This applies to YARN in cluster mode. This allows them to be set per submission without needing the unset the env vars (which is not always possible - e.g. batch submit with LIVY only exposes the arguments to spark-submit)
    
    ## How was this patch tested?
    Manual and existing unit tests.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/KevinGrealish/spark SPARK-16110

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13824
    
----
commit 2eb3833d134009e25f8717688a3fe0962dfac0e3
Author: KevinGrealish <ke...@microsoft.com>
Date:   2016-06-21T21:28:40Z

    SPARK-16110 ensure that PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are
    propogated to YARN application master in cluster mode before applying
    spark.yarn.appMasterEnv.* conf settings so that they can be set
    per submission. This allows submit of pyspark jobs via livy to
    specify python 2 or 3.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62935/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62905/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Unit tests involving env vars can get ugly:
    [http://stackoverflow.com/questions/318239/how-do-i-set-environment-variables-from-java](http://stackoverflow.com/questions/318239/how-do-i-set-environment-variables-from-java). I'm assuming such hacks would not be acceptable in this codebase.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    That is fine too. I'd just do something like:
    
    ```
    Seq("PYSPARK_DRIVER_PYTHON", "PYSPARK_PYTHON").foreach { envname =>
      // code to set the value
    }
    ```
    
    To avoid the repetition. BTW I just noticed you have a typo in the PR title.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62855/consoleFull)** for PR 13824 at commit [`cb41e9e`](https://github.com/apache/spark/commit/cb41e9edb9ff2fc0ade7aa6e61b0981cb27ab873).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62855/consoleFull)** for PR 13824 at commit [`cb41e9e`](https://github.com/apache/spark/commit/cb41e9edb9ff2fc0ade7aa6e61b0981cb27ab873).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/13824


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Cool, LGTM. Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    ping @KevinGrealish?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    @KevinGrealish you can read env vars through `SparkConf.getenv` and that allows you to override them in tests somewhat easily (look for `SparkConfWithEnv`).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Is it possible to move "spark.yarn.appMasterEnv.*" related codes to the bottom of the function? Looks like other env variables may also have the same problem like you mentioned for `PYSPARK_DRIVER_PATH`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    @tgravescs do you have a preference for this bug fix to use a hard coded list of PATH, LD_LIBRARY_PATH, CLASSPATH, APP_CLASSPATH (an append list), or a list of PYSPARK_PYTHON, PYSPARK_DRIVER_PYTHON (a override list). The later would narrow this fix, and reduce risk of it breaking anyone, so I'll propose that.
    
    A distinct issue/JIRA should be added for making the append/override work more generally long term, likely aligning with the Hadoop one. That should be done but I don't think it should block fixing this bug.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    @vanzin the function setupLaunchEnv reads env vars directly using sys.env.get so I don't understand how SparkConf helps you setup values for this function to read. See my earlier link to setting environment variables from Java/Scala code. Without the ability to set the values for test cases, unit testing that environment vars are overridden by conf is difficult to unit test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62933/consoleFull)** for PR 13824 at commit [`ca4d5fa`](https://github.com/apache/spark/commit/ca4d5fab7751d91387189b869489532a30d19fe1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    That is possible. It was done as is to be a narrower fix and less likely to cause other behavior to change. It does not look like env hashmap is read by the rest of the function (only written or values appended to) so the appMasterEnv could be moved down to give it precedence. I'll update the pull request.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python ...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish closed the pull request at:

    https://github.com/apache/spark/pull/13824


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62899/consoleFull)** for PR 13824 at commit [`6791edc`](https://github.com/apache/spark/commit/6791edc27a8b33bdf8c516a8cde5f431f6b0b07e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #61407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61407/consoleFull)** for PR 13824 at commit [`4ad7904`](https://github.com/apache/spark/commit/4ad790415dc504303e64c1c704655dc72af75fb1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    I'm fine with the change to allow this overriding the existing PYSPARK_PYTHON and others, but on the change from addPathToEnvironment I'm against.   The problem is that some env variables don't take this but others do.  I think it was this way just incase the variable required a classpath type combining, for instance LD_LIBRARY_PATH, or I think any other env variables that have "PATH" in the name of them.  We don't have a current way to differentiate when they should be combined with existing vs just overwritten.
    
    If you want to change that I would say we need a way to differentiate.  So whether that is looking for "PATH" in the env variable name and appending or having a config or something that says for these variables use append else don't.  Probably should be a separate jira for this one though.
    
    Note here is jira very similar in MapReduce: https://issues.apache.org/jira/browse/MAPREDUCE-6491


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    I'm ok with the behavior change. Maybe other people have more background on that behavior though; @tgravescs?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    @KevinGrealish were you planning to add a unit test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    sorry missed that this wouldn't work without not appending, yeah I like @vanzin latter suggestion if this change is needed soon. then figure out long term way to support with SPARK-16744. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62905/consoleFull)** for PR 13824 at commit [`f2c2e4a`](https://github.com/apache/spark/commit/f2c2e4a82ed44d367db67f9382024a619b688104).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62905/consoleFull)** for PR 13824 at commit [`f2c2e4a`](https://github.com/apache/spark/commit/f2c2e4a82ed44d367db67f9382024a619b688104).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62933/consoleFull)** for PR 13824 at commit [`ca4d5fa`](https://github.com/apache/spark/commit/ca4d5fab7751d91387189b869489532a30d19fe1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62933/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    How about just this:
    `      // propagate PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON to driver in cluster mode`
    `      if (!env.contains("PYSPARK_DRIVER_PYTHON")) {`
    `        sys.env.get("PYSPARK_DRIVER_PYTHON").foreach(env("PYSPARK_DRIVER_PYTHON") = _)`
    `      }`
    `      if (!env.contains("PYSPARK_PYTHON")) {`
    `        sys.env.get("PYSPARK_PYTHON").foreach(env("PYSPARK_PYTHON") = _)`
    `      }`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Created https://issues.apache.org/jira/browse/SPARK-16744 for the override/append issue. linked to 16110. This fix remains just about being able to run Python 3.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Even when working with env vars values that will be interpreted as lists of paths, the user intent could still be append or override. There could be a spark.yarn.appMasterAppendEnv.* config setting for appending (with the existing assumption of the list delimiter). Tom, how are you proposing we fix the PYSPARK_PYTHON need without the change? That environment variable does not work like a java path. Different behavior based on var name contains "PATH" seems like a temp hack at best.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61407/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62855/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by KevinGrealish <gi...@git.apache.org>.
Github user KevinGrealish commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Was on vacation. Looking again today.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    Also would you please add a unit test about it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python ...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13824#discussion_r72509645
  
    --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala ---
    @@ -188,7 +202,9 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
         checkResult(finalState, executorResult, "ORIGINAL")
       }
     
    -  private def testPySpark(clientMode: Boolean): Unit = {
    +  private def testPySpark(clientMode: Boolean,
    +                          extraConf: Map[String, String] = Map(),
    --- End diff --
    
    sorry, one last nit. style for multi-line args is:
    
    ```
    def method(
        arg1: Foo,
        arg2: Bar): Unit = {
    
    }
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #13824: [SPARK-16110][YARN][PYSPARK] Fix allowing python version...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13824
  
    **[Test build #62935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62935/consoleFull)** for PR 13824 at commit [`7ad3d77`](https://github.com/apache/spark/commit/7ad3d77968338b0747a134752f4af658b47b0cfe).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org