You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2017/03/20 04:45:57 UTC
[GitHub] spark pull request #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda ...
GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/17355
[SPARK-19955][WIP][PySpark] Jenkins Python Conda based test.
## What changes were proposed in this pull request?
Allow Jenkins Python tests to use the installed conda to test Python 2.7 support & test pip installability.
## How was this patch tested?
Updated shell scripts, ran tests locally with installed conda. WIP: verify works in Jenkins environment.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark SPARK-19955-support-python-tests-with-conda
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17355.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17355
----
commit d4b7e372fd86ec06de772143df4653404e48ee48
Author: Holden Karau <ho...@us.ibm.com>
Date: 2017-03-17T22:47:22Z
I'm pretty sure we can just add anaconda to the path cause its got python2.7 right now
commit 050c0914a18e5d334633e345b1ba9f07f5777cba
Author: Holden Karau <ho...@us.ibm.com>
Date: 2017-03-17T22:47:39Z
Oh right and check for python2.7
commit 3f0e0ef716221812d9c38e5bf2e488604d9c6c8f
Author: Holden Karau <ho...@us.ibm.com>
Date: 2017-03-17T22:47:58Z
Start looking at using conda for the pip installability tests
commit 480846024c39dd4a8a03d1826e65bc4eeb48770a
Author: Holden Karau <ho...@us.ibm.com>
Date: 2017-03-20T00:05:18Z
Update run pip tests to try and work with conda enviroments as well
commit 267837cd741b9a1d50842e485c20033aa9b77f8f
Author: Holden Karau <ho...@us.ibm.com>
Date: 2017-03-20T04:43:40Z
Remove set -x
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
cc @joshrosen & @shaneknapp : this PR allows us to keep our existing Jenkins worker setup while still moving away from 2.6 to 2.7 & enables pip packaging tests in Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda ...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/17355#discussion_r107755248
--- Diff: dev/run-pip-tests ---
@@ -75,18 +81,24 @@ for python in "${PYTHON_EXECS[@]}"; do
echo "Using $VIRTUALENV_BASE for virtualenv"
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
rm -rf "$VIRTUALENV_PATH"
- mkdir -p "$VIRTUALENV_PATH"
- virtualenv --python=$python "$VIRTUALENV_PATH"
- source "$VIRTUALENV_PATH"/bin/activate
- # Upgrade pip & friends
- pip install --upgrade pip pypandoc wheel
- pip install numpy # Needed so we can verify mllib imports
+ if [ -n "$USE_CONDA" ]; then
+ conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
--- End diff --
Setting `python=$python` led to "python=3" which then tried to install python 3.6
```
+ conda create -y -p /tmp/tmp.OymEZOKFzo/3 python=3 numpy pandas pip setuptools
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /tmp/tmp.OymEZOKFzo/3:
The following NEW packages will be INSTALLED:
mkl: 2017.0.1-0
numpy: 1.12.1-py36_0
openssl: 1.0.2k-1
pandas: 0.19.2-np112py36_1
pip: 9.0.1-py36_1
python: 3.6.1-0
...
```
And that led to a conflict with pypandoc:
```
UnsatisfiableError: The following specifications were found to be in conflict:
- pypandoc -> python 3.5* -> sqlite 3.9.*
- pypandoc -> python 3.5* -> xz 5.0.*
- python 3.6*
```
manually setting "python=3.5" seemed to clear things up so it could complete the test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #74871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74871/testReport)** for PR 17355 at commit [`267837c`](https://github.com/apache/spark/commit/267837cd741b9a1d50842e485c20033aa9b77f8f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75033/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Great, yay 2.6 deprecation adventures :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75090/testReport)** for PR 17355 at commit [`16d2773`](https://github.com/apache/spark/commit/16d2773f4154a7b2324e9083c2f7d2b61da2ac35).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75333 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75333/testReport)** for PR 17355 at commit [`a7bf53f`](https://github.com/apache/spark/commit/a7bf53f1b0f3c7104d23a0c1153b15eddceb9169).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75274/testReport)** for PR 17355 at commit [`5db1bc7`](https://github.com/apache/spark/commit/5db1bc7a0fe1c511af1e2a9b26909c593bc01316).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75224/testReport)** for PR 17355 at commit [`f99f222`](https://github.com/apache/spark/commit/f99f222f2979a72f44520442e594d090200bf4ec).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #74853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74853/testReport)** for PR 17355 at commit [`267837c`](https://github.com/apache/spark/commit/267837cd741b9a1d50842e485c20033aa9b77f8f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75025/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74871/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74853/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75057/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75090/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75063/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
.@bryanxutler so I left out pypandoc because there isn't pandoc on the machines and it's optional (prints a warning to stderr - but should work fine). I get back from vacation next week so let's chat then :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75138/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75274/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75073/testReport)** for PR 17355 at commit [`6f33633`](https://github.com/apache/spark/commit/6f33633348b9bf735074f2596e6f130b5d8dba04).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/17355
lgtm++
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda ...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17355#discussion_r108255885
--- Diff: dev/run-pip-tests ---
@@ -75,18 +81,24 @@ for python in "${PYTHON_EXECS[@]}"; do
echo "Using $VIRTUALENV_BASE for virtualenv"
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
rm -rf "$VIRTUALENV_PATH"
- mkdir -p "$VIRTUALENV_PATH"
- virtualenv --python=$python "$VIRTUALENV_PATH"
- source "$VIRTUALENV_PATH"/bin/activate
- # Upgrade pip & friends
- pip install --upgrade pip pypandoc wheel
- pip install numpy # Needed so we can verify mllib imports
+ if [ -n "$USE_CONDA" ]; then
+ conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
--- End diff --
sounds reasonable, for packaging I've made it explicitly request Python 3.5 (at some point if PyPandoc doesn't make it into 3.6 on conda forge we should ping them but no rush).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75057 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75057/testReport)** for PR 17355 at commit [`a722140`](https://github.com/apache/spark/commit/a722140c895c0066502d0f4d6c7e120cb2092d1e).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][PySpark] Jenkins Python Conda based...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on a diff in the pull request:
https://github.com/apache/spark/pull/17355#discussion_r108534754
--- Diff: dev/run-pip-tests ---
@@ -35,32 +35,38 @@ function delete_virtualenv() {
}
trap delete_virtualenv EXIT
+PYTHON_EXECS=()
# Some systems don't have pip or virtualenv - in those cases our tests won't work.
-if ! hash virtualenv 2>/dev/null; then
- echo "Missing virtualenv skipping pip installability tests."
+if hash virtualenv 2>/dev/null && [ ! -n "$USE_CONDA" ]; then
+ echo "virtualenv installed - using. Note if this is a conda virtual env you may wish to set USE_CONDA"
+ # Figure out which Python execs we should test pip installation with
+ if hash python2 2>/dev/null; then
+ # We do this since we are testing with virtualenv and the default virtual env python
+ # is in /usr/bin/python
+ PYTHON_EXECS+=('python2')
+ elif hash python 2>/dev/null; then
+ # If python2 isn't installed fallback to python if available
+ PYTHON_EXECS+=('python')
+ fi
+ if hash python3 2>/dev/null; then
+ PYTHON_EXECS+=('python3')
+ fi
+elif hash conda 2>/dev/null; then
+ echo "Using conda virtual enviroments"
+ PYTHON_EXECS=('3.5')
+ USE_CONDA=1
+else
+ echo "Missing virtualenv & conda, skipping pip installability tests"
exit 0
fi
if ! hash pip 2>/dev/null; then
echo "Missing pip, skipping pip installability tests."
exit 0
fi
-# Figure out which Python execs we should test pip installation with
-PYTHON_EXECS=()
-if hash python2 2>/dev/null; then
- # We do this since we are testing with virtualenv and the default virtual env python
- # is in /usr/bin/python
- PYTHON_EXECS+=('python2')
-elif hash python 2>/dev/null; then
- # If python2 isn't installed fallback to python if available
- PYTHON_EXECS+=('python')
-fi
-if hash python3 2>/dev/null; then
- PYTHON_EXECS+=('python3')
-fi
-
+set -x
--- End diff --
is this just there for debugging? if so, pls remove before merging. otherwise, consider sticking it at the beginning of the script.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Oops @bryancutler damn phone keyboard.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75081/testReport)** for PR 17355 at commit [`16d2773`](https://github.com/apache/spark/commit/16d2773f4154a7b2324e9083c2f7d2b61da2ac35).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #74998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74998/testReport)** for PR 17355 at commit [`267837c`](https://github.com/apache/spark/commit/267837cd741b9a1d50842e485c20033aa9b77f8f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda ...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/17355#discussion_r107757313
--- Diff: python/run-tests.py ---
@@ -111,9 +111,9 @@ def run_individual_python_test(test_name, pyspark_python):
def get_default_python_executables():
- python_execs = [x for x in ["python2.6", "python3.4", "pypy"] if which(x)]
- if "python2.6" not in python_execs:
- LOGGER.warning("Not testing against `python2.6` because it could not be found; falling"
+ python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)]
--- End diff --
Does this mean we are not supporting 2.6 anymore!?!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75073/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75224 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75224/testReport)** for PR 17355 at commit [`f99f222`](https://github.com/apache/spark/commit/f99f222f2979a72f44520442e594d090200bf4ec).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75090/testReport)** for PR 17355 at commit [`16d2773`](https://github.com/apache/spark/commit/16d2773f4154a7b2324e9083c2f7d2b61da2ac35).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Merged to master. Please do not backport.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75033/testReport)** for PR 17355 at commit [`bc4f673`](https://github.com/apache/spark/commit/bc4f673a901ed218c6a5d1b7afac622a85a20eac).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Jenkins retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75333/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda ...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17355#discussion_r107816730
--- Diff: python/run-tests.py ---
@@ -111,9 +111,9 @@ def run_individual_python_test(test_name, pyspark_python):
def get_default_python_executables():
- python_execs = [x for x in ["python2.6", "python3.4", "pypy"] if which(x)]
- if "python2.6" not in python_execs:
- LOGGER.warning("Not testing against `python2.6` because it could not be found; falling"
+ python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)]
--- End diff --
Indeed we've been talking about removing it but it's been blocked on Jenkins work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75333/testReport)** for PR 17355 at commit [`a7bf53f`](https://github.com/apache/spark/commit/a7bf53f1b0f3c7104d23a0c1153b15eddceb9169).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17355
Hope you feel better soon @shaneknapp :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75025/testReport)** for PR 17355 at commit [`bc4f673`](https://github.com/apache/spark/commit/bc4f673a901ed218c6a5d1b7afac622a85a20eac).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75063 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75063/testReport)** for PR 17355 at commit [`57a1f6e`](https://github.com/apache/spark/commit/57a1f6e27132d66d2f5e7d1915d7c9e53eb86471).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda ...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/17355#discussion_r107755910
--- Diff: dev/run-pip-tests ---
@@ -75,18 +81,24 @@ for python in "${PYTHON_EXECS[@]}"; do
echo "Using $VIRTUALENV_BASE for virtualenv"
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
rm -rf "$VIRTUALENV_PATH"
- mkdir -p "$VIRTUALENV_PATH"
- virtualenv --python=$python "$VIRTUALENV_PATH"
- source "$VIRTUALENV_PATH"/bin/activate
- # Upgrade pip & friends
- pip install --upgrade pip pypandoc wheel
- pip install numpy # Needed so we can verify mllib imports
+ if [ -n "$USE_CONDA" ]; then
+ conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
+ source activate "$VIRTUALENV_PATH"
--- End diff --
I had to add this line after `source activate ..` to get pypandoc installed
```
conda install -y -c conda-forge pypandoc
```
Otherwise I got this error:
```
Could not import pypandoc - required to package PySpark
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75138/testReport)** for PR 17355 at commit [`8fe8ada`](https://github.com/apache/spark/commit/8fe8adab7abc09dc595c2ee724b615a12107e7a6).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/17355
changes look good. i'll give it a closer look tomorrow... i've been out
of town and down w/bronchitis for the past week and a half.
On Mon, Mar 27, 2017 at 12:19 PM, Holden Karau <no...@github.com>
wrote:
> cc @joshrosen <https://github.com/joshrosen> & @shaneknapp
> <https://github.com/shaneknapp> : this PR allows us to keep our existing
> Jenkins worker setup while still moving away from 2.6 to 2.7 & enables pip
> packaging tests in Jenkins.
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/17355#issuecomment-289556742>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABiDrIr3eoHyqOSUPRP0zti_zfWomTPBks5rqAutgaJpZM4MiBuI>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75025/testReport)** for PR 17355 at commit [`bc4f673`](https://github.com/apache/spark/commit/bc4f673a901ed218c6a5d1b7afac622a85a20eac).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda ...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17355#discussion_r108256078
--- Diff: dev/run-pip-tests ---
@@ -75,18 +81,24 @@ for python in "${PYTHON_EXECS[@]}"; do
echo "Using $VIRTUALENV_BASE for virtualenv"
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
rm -rf "$VIRTUALENV_PATH"
- mkdir -p "$VIRTUALENV_PATH"
- virtualenv --python=$python "$VIRTUALENV_PATH"
- source "$VIRTUALENV_PATH"/bin/activate
- # Upgrade pip & friends
- pip install --upgrade pip pypandoc wheel
- pip install numpy # Needed so we can verify mllib imports
+ if [ -n "$USE_CONDA" ]; then
+ conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
+ source activate "$VIRTUALENV_PATH"
--- End diff --
So its not a hard error, and since the workers don't have pandoc installed (a separate binary) leaving it out for now seems like the easist path. Once we're all dockerized and happy we can add pandoc & pypandoc to the docker image.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #74871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74871/testReport)** for PR 17355 at commit [`267837c`](https://github.com/apache/spark/commit/267837cd741b9a1d50842e485c20033aa9b77f8f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75224/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #74998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74998/testReport)** for PR 17355 at commit [`267837c`](https://github.com/apache/spark/commit/267837cd741b9a1d50842e485c20033aa9b77f8f).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17355: [SPARK-19955][PySpark] Jenkins Python Conda based...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17355
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74998/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75057/testReport)** for PR 17355 at commit [`a722140`](https://github.com/apache/spark/commit/a722140c895c0066502d0f4d6c7e120cb2092d1e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75063/testReport)** for PR 17355 at commit [`57a1f6e`](https://github.com/apache/spark/commit/57a1f6e27132d66d2f5e7d1915d7c9e53eb86471).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75073/testReport)** for PR 17355 at commit [`6f33633`](https://github.com/apache/spark/commit/6f33633348b9bf735074f2596e6f130b5d8dba04).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75274/testReport)** for PR 17355 at commit [`5db1bc7`](https://github.com/apache/spark/commit/5db1bc7a0fe1c511af1e2a9b26909c593bc01316).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17355: [SPARK-19955][WIP][PySpark] Jenkins Python Conda based t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17355
**[Test build #75138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75138/testReport)** for PR 17355 at commit [`8fe8ada`](https://github.com/apache/spark/commit/8fe8adab7abc09dc595c2ee724b615a12107e7a6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org