You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/07/15 01:45:19 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
HyukjinKwon opened a new pull request #29116:
URL: https://github.com/apache/spark/pull/29116
### What changes were proposed in this pull request?
This PR aims to test PySpark with Python 3.8 in Github Actions. In the script side, it is already ready:
https://github.com/apache/spark/blob/4ad9bfd53b84a6d2497668c73af6899bae14c187/python/run-tests.py#L161
This PR includes small related fixes together:
1. Install Python 3.8
2. Only install one Python implementation instead of installing many for SQL and Yarn test cases because they need one Python executable in their test cases that is higher than Python 2.
3. Do not install Python 2 which is not needed anymore
### Why are the changes needed?
Currently, only PyPy3 and Python 3.6 are being tested with PySpark in Github Actions. We should test the latest version of Python as well because some optimizations can be only enabled with Python 3.8+. See also https://github.com/apache/spark/pull/29114
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Was not tested. Github Actions build in this PR will test it out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658497629
**[Test build #125868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125868/testReport)** for PR 29116 at commit [`16a2b75`](https://github.com/apache/spark/commit/16a2b75233cf9b9086a8c29842ea995bfc3d1b3d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658510760
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29116:
URL: https://github.com/apache/spark/pull/29116
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658504192
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658533425
Thanks, @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658497933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658510741
**[Test build #125870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125870/testReport)** for PR 29116 at commit [`7b47107`](https://github.com/apache/spark/commit/7b47107fe616ca5882c5e817ac91b4c50593911c).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29116:
URL: https://github.com/apache/spark/pull/29116#discussion_r454754613
##########
File path: .github/workflows/master.yml
##########
@@ -117,38 +117,42 @@ jobs:
java-version: ${{ matrix.java }}
# PySpark
- name: Install PyPy3
- # SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
# Note that order of Python installations here matters because default python3 is
# overridden by pypy3.
uses: actions/setup-python@v2
- if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
+ if: contains(matrix.modules, 'pyspark')
with:
python-version: pypy3
architecture: x64
- - name: Install Python 2.7
+ - name: Install Python 3.6
uses: actions/setup-python@v2
- if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
+ if: contains(matrix.modules, 'pyspark')
with:
- python-version: 2.7
+ python-version: 3.6
architecture: x64
- - name: Install Python 3.6
+ - name: Install Python 3.8
uses: actions/setup-python@v2
- # Yarn has a Python specific test too, for example, YarnClusterSuite.
+ # We should install one Python that is higher then 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
if: contains(matrix.modules, 'yarn') || contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
with:
- python-version: 3.6
+ python-version: 3.8
architecture: x64
- - name: Install Python packages
- if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
+ - name: Install Python packages (Python 3.6 and PyPy3)
+ if: contains(matrix.modules, 'pyspark')
# PyArrow is not supported in PyPy yet, see ARROW-2651.
# TODO(SPARK-32247): scipy installation with PyPy fails for an unknown reason.
run: |
- python3 -m pip install numpy pyarrow pandas scipy
- python3 -m pip list
- python2 -m pip install numpy pyarrow pandas scipy
- python2 -m pip list
+ python3.6 -m pip install numpy pyarrow pandas scipy
+ python3.6 -m pip list
pypy3 -m pip install numpy pandas
pypy3 -m pip list
+ - name: Install Python packages (Python 3.8)
+ if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
Review comment:
Yarn test does not need other packages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658503968
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658503968
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658497933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658510771
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125870/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658497629
**[Test build #125868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125868/testReport)** for PR 29116 at commit [`16a2b75`](https://github.com/apache/spark/commit/16a2b75233cf9b9086a8c29842ea995bfc3d1b3d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658510760
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658503889
**[Test build #125868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125868/testReport)** for PR 29116 at commit [`16a2b75`](https://github.com/apache/spark/commit/16a2b75233cf9b9086a8c29842ea995bfc3d1b3d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658503843
**[Test build #125870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125870/testReport)** for PR 29116 at commit [`7b47107`](https://github.com/apache/spark/commit/7b47107fe616ca5882c5e817ac91b4c50593911c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658504192
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29116: [SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Actions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29116:
URL: https://github.com/apache/spark/pull/29116#issuecomment-658503843
**[Test build #125870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125870/testReport)** for PR 29116 at commit [`7b47107`](https://github.com/apache/spark/commit/7b47107fe616ca5882c5e817ac91b4c50593911c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org