You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shearerp <gi...@git.apache.org> on 2016/04/20 15:46:16 UTC
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
GitHub user shearerp opened a pull request:
https://github.com/apache/spark/pull/12528
[SPARK-13973][PYSPARK] Make pyspark fail noisily if IPYTHON or IPYTHON_OPTS are set
## What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-13973
Following discussion with @srowen the IPYTHON and IPYTHON_OPTS variables are removed. If they are set in the user's environment, pyspark will not execute and prints an error message. Failing noisily will force users to remove these options and learn the new configuration scheme, which is much more sustainable and less confusing.
## How was this patch tested?
Manual testing; set IPYTHON=1 and verified that the error message prints.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shearerp/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12528.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12528
----
commit a2f8277886e66282898c2f015565ed1a8950deb6
Author: pshearer <ps...@massmutual.com>
Date: 2016-04-20T13:43:52Z
Make pyspark fail noisily if IPYTHON or IPYTHON_OPTS are set
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12528
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215771753
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-212433256
That also seems OK to me. Eh, @JoshRosen do you have an opinion on just removing these options?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by shearerp <gi...@git.apache.org>.
Github user shearerp commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215699051
@srowen I tried twice to figure out how I change the Spark documentation and so far I've been unable to figure it out. Could I get a little help?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by shearerp <gi...@git.apache.org>.
Github user shearerp commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-212593947
The problem with IPYTHON and IPYTHON_OPTS is, it's not quite clear how to get it to work as expected for both users of IPython shell and IPython notebook. The latter has been deprecated in favor of the Jupyter notebook, and with this change, the user's choice has become a bit fragile and unclear.
The other option is to have
- IPYTHON=1 by itself means the IPython shell
- you get the notebook if IPYTHON_OPTS starts with "notebook". Specifically, the Jupyter notebook if it's installed, the IPython notebook if it's not.
This requires a bit more convoluted logic in the pyspark shell script, but maybe the backward compatibility is worth it?
See the JIRA discussion for more info: https://issues.apache.org/jira/browse/SPARK-13973
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12528#discussion_r60841893
--- Diff: bin/pyspark ---
@@ -33,8 +33,9 @@ export _SPARK_CMD_USAGE="Usage: ./bin/pyspark [options]"
# PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver
# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook'). This supports full customization of the IPython
# and executor Python executables.
-#
-# For backwards-compatibility, we retain the old IPYTHON and IPYTHON_OPTS variables.
+#
--- End diff --
The whole block of docs above this could be removed.
while you're at it, one last change would be helpful: mention in `programming-guide.md` that `jupyter` works too, where it shows the `ipython` example. Just a sentence noting that is OK.
LGTM otherwise.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by shearerp <gi...@git.apache.org>.
Github user shearerp commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215706060
@srowen done, sorry for the delay. I tried to do this a couple times before but couldn't figure out where the docs were. Github search is actively misleading; when you search for programming-guide.md it shows you everything except programming-guide.md, giving the impression that it doesn't exist in the repo. But looking at the wiki I realized it is, in fact, in docs/ so I browsed to it and found it at last :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215771760
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57337/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-212596539
Alright - I don't have a problem with removing it.
@JoshRosen should chime in too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215771120
**[Test build #57337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57337/consoleFull)** for PR 12528 at commit [`430dc76`](https://github.com/apache/spark/commit/430dc7602761c8c3c27692a7eda0673cd3247593).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215725642
**[Test build #57337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57337/consoleFull)** for PR 12528 at commit [`430dc76`](https://github.com/apache/spark/commit/430dc7602761c8c3c27692a7eda0673cd3247593).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215948496
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215365706
Thanks @shearerp but can you also very briefly mention how to invoke `jupyter` in the programming guide? just a sentence would suffice, so that people are clear this is how you can select it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-212432093
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-215725111
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-212572520
hm I always use this IPYTHON=1 variable. Any benefit of us removing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13973][PYSPARK] Make pyspark fail noisi...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12528#issuecomment-212575352
It's been hidden / deprecated since 1.2, it seems: https://github.com/apache/spark/blob/master/bin/pyspark#L30
I suppose the issue is it's redundant with the replacement env var, and causes this particular problem with notebook vs non notebook use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org