You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2024/01/16 18:50:44 UTC
[PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas tests if not available [spark]
dongjoon-hyun opened a new pull request, #44759:
URL: https://github.com/apache/spark/pull/44759
### What changes were proposed in this pull request?
This PR aims to skip `Pandas`-related tests in `pyspark.sql.tests.test_group` if `Pandas` is not installed.
### Why are the changes needed?
Since `Pandas` is optional, we need to skip the tests instead of failures.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually with the Python installation without Pandas.
```
$ python/run-tests.py --testnames pyspark.sql.tests.test_group
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark-merge/python/unit-tests.log
Will test against the following Python executables: ['python3.9', 'pypy3']
Will test the following Python tests: ['pyspark.sql.tests.test_group']
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.18
pypy3 python_implementation is PyPy
pypy3 version is: Python 3.10.13 (f1607341da97ff5a1e93430b6e8c4af0ad1aa019, Sep 28 2023, 20:47:55)
[PyPy 7.3.13 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)]
Starting test(python3.9): pyspark.sql.tests.test_group (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/ac9269b6-f0df-4d06-88b8-e5e710202b60/python3.9__pyspark.sql.tests.test_group__9zjp5i4z.log)
Starting test(pypy3): pyspark.sql.tests.test_group (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/cab6ebed-e49f-4d86-80db-0dc3928079e3/pypy3__pyspark.sql.tests.test_group__thw6hily.log)
Finished test(pypy3): pyspark.sql.tests.test_group (6s) ... 3 tests were skipped
Finished test(python3.9): pyspark.sql.tests.test_group (7s) ... 3 tests were skipped
Tests passed in 7 seconds
Skipped tests in pyspark.sql.tests.test_group with pypy3:
test_agg_func (pyspark.sql.tests.test_group.GroupTests) ... skipped '[PACKAGE_NOT_INSTALLED] Pandas >= 1.4.4 must be installed; however, it was not found.'
test_group_by_ordinal (pyspark.sql.tests.test_group.GroupTests) ... skipped '[PACKAGE_NOT_INSTALLED] Pandas >= 1.4.4 must be installed; however, it was not found.'
test_order_by_ordinal (pyspark.sql.tests.test_group.GroupTests) ... skipped '[PACKAGE_NOT_INSTALLED] Pandas >= 1.4.4 must be installed; however, it was not found.'
Skipped tests in pyspark.sql.tests.test_group with python3.9:
test_agg_func (pyspark.sql.tests.test_group.GroupTests) ... SKIP (0.000s)
test_group_by_ordinal (pyspark.sql.tests.test_group.GroupTests) ... SKIP (0.000s)
test_order_by_ordinal (pyspark.sql.tests.test_group.GroupTests) ... SKIP (0.000s)
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44759:
URL: https://github.com/apache/spark/pull/44759#issuecomment-1894487512
Could you review this PR when you have some time, @xinrong-meng ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #44759:
URL: https://github.com/apache/spark/pull/44759#issuecomment-1894584582
Thank you @dongjoon-hyun for catching that and for the fix! LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas tests if not available [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44759:
URL: https://github.com/apache/spark/pull/44759#issuecomment-1894336202
cc @xinrong-meng, @zhengruifeng, @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44759:
URL: https://github.com/apache/spark/pull/44759#issuecomment-1894529741
Could you review this Python test PR, @viirya ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #44759:
URL: https://github.com/apache/spark/pull/44759#issuecomment-1894603816
I wanted to add that those tests do not necessarily rely on Pandas/PyArrow, but the "assertDataFrameEqual" utility used in the tests does. I'll file a follow-up PR to adjust that. CC @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #44759:
URL: https://github.com/apache/spark/pull/44759#issuecomment-1894721157
Late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #44759: [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available
URL: https://github.com/apache/spark/pull/44759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44759:
URL: https://github.com/apache/spark/pull/44759#issuecomment-1894584317
Thank you, @xinrong-meng and @viirya .
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org