You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/09 06:43:35 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request, #38991: [SPARK-41457][PYTHON][TESTS] Refactor pandas, pyarrow and grpc check in tests
HyukjinKwon opened a new pull request, #38991:
URL: https://github.com/apache/spark/pull/38991
### What changes were proposed in this pull request?
This PR proposes to:
- Print out the correct error message when dependencies are not installed for `pyspark.sql.connect`
- Ignore all errors from tests:
- Initial intention was to annotate types for public APIs only.
- Also removed unused ignores in other modules
- Print out the correct message if 1. dependencies are missing or 2. build is not there.
- Move `python/mypy.ini` to `dev/mypy.ini` to be consistentl with `dev/tox.ini`
Note that, for Spark Connect, we should follow both below to make sure they are properly skipped when dependences are not there, or jars do not exist.
- Always use `if should_test_connect` in `pyspark.sql.connect` imports in tests.
- Use `@unittest.skipIf(not should_test_connect, connect_requirement_message)`
### Why are the changes needed?
- To make the development easier and faster.
- To make sure our Jenkins build pass without Python dependencies installed
- See [Scaleaway](https://apache-spark.s3.fr-par.scw.cloud/index.html (), see also https://github.com/apache/spark-website/blob/asf-site/developer-tools.md#test-coverage)
### Does this PR introduce _any_ user-facing change?
No, dev and test-only.
### How was this patch tested?
Manually tested without pandas, pyarrow and grpc, and verified the error messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38991:
URL: https://github.com/apache/spark/pull/38991#issuecomment-1345145538
All python and linter tests passed. Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor pandas, pyarrow and grpc check in tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38991:
URL: https://github.com/apache/spark/pull/38991#issuecomment-1343919107
cc @grundprinzip @hvanhovell @dongjoon-hyun @amaliujia @zhengruifeng @xinrong-meng FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
URL: https://github.com/apache/spark/pull/38991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38991:
URL: https://github.com/apache/spark/pull/38991#issuecomment-1344297493
rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38991:
URL: https://github.com/apache/spark/pull/38991#issuecomment-1343920074
Thank you, @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38991:
URL: https://github.com/apache/spark/pull/38991#issuecomment-1344612390
Could you check the linter failure?
```
starting mypy data test...
annotations failed data checks:
============================= test session starts ==============================
platform linux -- Python 3.9.5, pytest-7.1.3, pluggy-1.0.0
rootdir: /__w/spark/spark/python, configfile: ../dev/pyproject.toml
plugins: mypy-plugins-1.9.3
collected 40 items
python/pyspark/ml/tests/typing/test_classification.yml FF [ 5%]
python/pyspark/ml/tests/typing/test_clustering.yaml F [ 7%]
python/pyspark/ml/tests/typing/test_evaluation.yml F [ 10%]
python/pyspark/ml/tests/typing/test_feature.yml FF [ 15%]
python/pyspark/ml/tests/typing/test_param.yml F [ 17%]
python/pyspark/ml/tests/typing/test_readable.yml F [ 20%]
python/pyspark/ml/tests/typing/test_regression.yml FFF [ 27%]
python/pyspark/sql/tests/typing/test_column.yml F [ 30%]
python/pyspark/sql/tests/typing/test_dataframe.yml FFFFFFF [ 47%]
python/pyspark/sql/tests/typing/test_functions.yml F [ 50%]
python/pyspark/sql/tests/typing/test_readwriter.yml FF [ 55%]
python/pyspark/sql/tests/typing/test_session.yml FFFFF [ 67%]
python/pyspark/sql/tests/typing/test_udf.yml FFFFFFF [ 85%]
python/pyspark/tests/typing/test_context.yml F [ 87%]
python/pyspark/tests/typing/test_core.yml F [ 90%]
python/pyspark/tests/typing/test_rdd.yml FFF [ 97%]
python/pyspark/tests/typing/test_resultiterable.yml F [100%]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38991:
URL: https://github.com/apache/spark/pull/38991#discussion_r1044955147
##########
dev/lint-python:
##########
@@ -104,7 +104,7 @@ function mypy_data_test {
-c dev/pyproject.toml \
--rootdir python \
--mypy-only-local-stub \
- --mypy-ini-file python/mypy.ini \
+ --mypy-ini-file dev/mypy.ini \
Review Comment:
Okay, MyPy requires the configuration file to locate in the root directory of the package. Let me move it back for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38991: [SPARK-41457][PYTHON][TESTS] Refactor type annotations and dependency checks in tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38991:
URL: https://github.com/apache/spark/pull/38991#issuecomment-1345238945
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org