You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shaneknapp <gi...@git.apache.org> on 2018/08/09 17:36:35 UTC
[GitHub] spark pull request #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bu...
GitHub user shaneknapp opened a pull request:
https://github.com/apache/spark/pull/22061
[SPARK-25079][PYTHON] preparing for python 3.5 bump
## What changes were proposed in this pull request?
preparing to bump python 3.4 to 3.5
## How was this patch tested?
tested against a python3.5 installation on my ubuntu testing node, and everything passes:
https://issues.apache.org/jira/browse/SPARK-25079
this will need to be orchestrated carefully... details of steps needed to be taken are in the jira issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shaneknapp/spark prepare-for-python3.5
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22061.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22061
----
commit 80b8c9c23ff8e6613d1abc023cf6a30b673307db
Author: shane knapp <in...@...>
Date: 2018-08-09T17:31:00Z
preparing for python 3.5 bump
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bu...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/22061#discussion_r209017243
--- Diff: python/run-tests.py ---
@@ -162,7 +162,7 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
def get_default_python_executables():
- python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)]
+ python_execs = [x for x in ["python2.7", "python3.4", "python3.5", "pypy"] if which(x)]
--- End diff --
i will push another commit at some point in the future to remove the reference to python3.4.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22061
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94515/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22061
**[Test build #94515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94515/testReport)** for PR 22061 at commit [`80b8c9c`](https://github.com/apache/spark/commit/80b8c9c23ff8e6613d1abc023cf6a30b673307db).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22061
another test triggered. forgot to actually test against 3.5... :(
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22061
this probably won't be able to get merged for a while.
On Thu, Aug 9, 2018 at 7:01 PM, Hyukjin Kwon <no...@github.com>
wrote:
> *@HyukjinKwon* approved this pull request.
>
> Change itself LGTM. Will push this in when it's ready and if my hand is
> needed to get this in. Please let me know later @shaneknapp
> <https://github.com/shaneknapp>.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/22061#pullrequestreview-145083409>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABiDrITC9weSTWgqbdTiAdwBlee0L5hoks5uPOlvgaJpZM4V2HgI>
> .
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bu...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/22061#discussion_r209669049
--- Diff: python/run-tests.py ---
@@ -162,7 +162,7 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
def get_default_python_executables():
- python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)]
+ python_execs = [x for x in ["python2.7", "python3.4", "python3.5", "pypy"] if which(x)]
--- End diff --
Just wondering why not replace python3.4 now? From the steps in jira, there will be a symlink for 3.4 -> 3.5 so it would be running the same tests twice right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22061
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22061
@BryanCutler the reason i'm not doing the upgrade is that the path for the current py3.4 anaconda env is hard-coded in to the python test running code across all branches.
this means if i do the upgrade to 3.5 on the workers and create the symlink, all branches will be running 3.5 w/only a small amount of testing. also, i'll need to ensure that there are no package/dependency issues w/previously installed libs.
if i create a new anaconda 3.5 env, then i will need to immediately change the path in dev/run-tests.py for whatever branch(es) that we want 3.5... as well as hack on the conda package definitions so that i can ansibileize things.
it's a bit more work that i think you might expect... i'll be working on getting this sorted this week.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22061
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2012/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22061
i launched a PRB build against a freshly-installed python 3.5 environment:
https://rise.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/71/console
i'm sure we'll have some things to fix, etc when it finishes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22061
@shaneknapp should this go in now? replacing 3.4?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22061
**[Test build #94515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94515/testReport)** for PR 22061 at commit [`80b8c9c`](https://github.com/apache/spark/commit/80b8c9c23ff8e6613d1abc023cf6a30b673307db).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22061
no, not yet. i'll give a bigger update later today on the status of this,
but the TL;DR is that it's a bigger project than i expected. :\
On Mon, Sep 24, 2018 at 9:18 AM, Sean Owen <no...@github.com> wrote:
> @shaneknapp <https://github.com/shaneknapp> should this go in now?
> replacing 3.4?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/22061#issuecomment-424033759>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABiDrI0nMUTlFHKimrukHjoO-NM5kiAEks5ueQXXgaJpZM4V2HgI>
> .
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/22061
also, i'll be taking the ubuntu-testing worker offline today to wipe and do a fresh install of conda w/python3.5.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/22061
Ok, make sense. Thanks @shaneknapp !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22061: [SPARK-25079][PYTHON] preparing for python 3.5 bump
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22061
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org