You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2017/07/31 11:26:00 UTC
[GitHub] spark pull request #18785: [SPARK-21573][BUILD] Tests failing with run-tests...
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/18785
[SPARK-21573][BUILD] Tests failing with run-tests.py SyntaxError occasionally in Jenkins
## What changes were proposed in this pull request?
Avoid dict comprehension to mae 2.6 compatibility a nonissue
## How was this patch tested?
Existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-21573
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18785.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18785
----
commit 3c2479c3cdea7855419f91ce8db23388eb5eb2e0
Author: Sean Owen <so...@cloudera.com>
Date: 2017-07-31T11:22:16Z
Avoid dict comprehension to mae 2.6 compatibility a nonissue
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18785
I googled a bit before. It looks one way I could find is, we should manually add `total_ordering` - `https://github.com/nvie/rq/commit/282f4be9316d608ebbacd6114aab1203591e8f95` however, this looked too much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18785
**[Test build #80073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80073/testReport)** for PR 18785 at commit [`3c2479c`](https://github.com/apache/spark/commit/3c2479c3cdea7855419f91ce8db23388eb5eb2e0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18785
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18785
@srowen, IIRC, I guess this should be correct when we run this as executable. If I understood correctly, we run `run-tests` - https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L115 which actually runs `run-tests.py` with `python` rather than as executable - https://github.com/apache/spark/blob/master/dev/run-tests#L23.
I was thinking a way such as,
```diff
--- a/dev/run-tests
+++ b/dev/run-tests
@@ -20,4 +20,8 @@
FWDIR="$(cd "`dirname $0`"/..; pwd)"
cd "$FWDIR"
-exec python -u ./dev/run-tests.py "$@"
+PYTHON_EXECUTABLE=python
+if hash python2.7 2>/dev/null; then
+ PYTHON_EXECUTABLE=python2.7
+fi
+exec $PYTHON_EXECUTABLE -u ./dev/run-tests.py "$@"
```
I know this bit is hard to test (existing tests indeed cover this but .. I guess it is rather randomly covered) but fortunately I already have an Ubuntu VM having `python` as Python 2.6.9. Do you mind if I take over this and CC you if this way makes sense to you? I guess it's easy to test for me now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18785
Will double check this one and merge it if we need no more changes. Or, will send a PR to your branch or open another PR. I will handle this in any event.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18785
That could just mean the machine doesn't have 2.7 installed then. This may really be an env issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18785
Ah OK, so it's a no-go. I wonder if writing python2.7 in the shebang will work on the Jenkins machines? @shaneknapp
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18785
**[Test build #80075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80075/testReport)** for PR 18785 at commit [`7e75266`](https://github.com/apache/spark/commit/7e75266dd956990387efe0bb432949da583edc05).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18785
Ugh.. this still fails:
```
========================================================================
Running Python style checks
========================================================================
PEP8 checks failed.
Compiling ./dev/run-tests.py ...
SyntaxError: ('invalid syntax', ('./dev/run-tests.py', 124, 65, ' {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)\n'))
[error] running /root/test/spark/dev/lint-python ; received return code 1
```
https://github.com/apache/spark/blob/864d94fe879a32de324da65a844e62a0260b222d/dev/run-tests.py#L212 -> https://github.com/apache/spark/blob/864d94fe879a32de324da65a844e62a0260b222d/dev/lint-python#L33
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18785
@HyukjinKwon this one just tries to explicitly select 2.7. If it fails, well, I suppose it would already fail the build anyway. If it successfully picks 2.7 then it should resolve this I guess. These are the two instances that explicitly select `python2`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18785
Just as a last note I checked, after fixing `lint-python`, it failed again somewhere:
```
========================================================================
Running PySpark tests
========================================================================
Traceback (most recent call last):
File "./python/run-tests.py", line 42, in <module>
from sparktestsupport.modules import all_modules # noqa
File "/.../spark/python/../dev/sparktestsupport/modules.py", line 18, in <module>
from functools import total_ordering
ImportError: cannot import name total_ordering
[error] running /.../spark/python/run-tests --parallelism=4 ; received return code 1
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18785
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80073/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18785: [SPARK-21573][BUILD] Tests failing with run-tests...
Posted by srowen <gi...@git.apache.org>.
Github user srowen closed the pull request at:
https://github.com/apache/spark/pull/18785
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18785
CC @holdenk @dongjoon-hyun
I know the point is to not support 2.6, but, maybe this is just as good as anything as a quick fix?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18785
**[Test build #80076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80076/testReport)** for PR 18785 at commit [`8516eb6`](https://github.com/apache/spark/commit/8516eb6360da76905af2a2fc47c1d82dc20e28a7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18785
Yeah go ahead. I pushed your change here anyway
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18785
@srowen, I actually tried this way ahead. I just ran `python2.6 run-tests.py` for sure against this PR:
```
Traceback (most recent call last):
File "run-tests.py", line 42, in <module>
from sparktestsupport.modules import all_modules # noqa
File "/.../spark/python/../dev/sparktestsupport/modules.py", line 18, in <module>
from functools import total_ordering
ImportError: cannot import name total_ordering
```
It looks we need several places to fix. For example, `total_ordering` is new in Python 2.7 -
https://docs.python.org/2/library/functools.html#functools.total_ordering.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18785: [SPARK-21573][BUILD] Tests failing with run-tests.py Syn...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18785
**[Test build #80073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80073/testReport)** for PR 18785 at commit [`3c2479c`](https://github.com/apache/spark/commit/3c2479c3cdea7855419f91ce8db23388eb5eb2e0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org