You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2023/11/29 01:33:35 UTC
[PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
HyukjinKwon opened a new pull request, #44065:
URL: https://github.com/apache/spark/pull/44065
### What changes were proposed in this pull request?
This PR proposes to skip doctest, `pyspark.pandas.series.Seires.to_dict`, compatible with Python 3.12.
### Why are the changes needed?
- For the proper test for Python 3.12. It is failing, see SPARK-46149 and https://github.com/apache/spark/actions/runs/7020654429/job/19100964890
- Current dev torch version does not work very well with Python 3.12. It should be enabled together with SPARK-46078.
- The build with Python 3.12 halts so can't see if other tests pass fine.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Yes:
```
./python/run-tests --python-executables=python3 --testnames 'pyspark.ml.torch.tests.test_distributor TorchDistributorLocalUnitTests'
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617098
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
To @HyukjinKwon , do we need to reevaluate every instances like this? It looks a little counter-intuitive.
```
$ git grep 'sys.version_info >'
python/pyspark/pandas/__init__.py: if sys.version_info >= (3, 7):
python/pyspark/pandas/frame.py: can_return_named_tuples = sys.version_info >= (3, 7) or len(self.columns) + index < 255
python/pyspark/pandas/tests/computation/test_apply_func.py: if sys.version_info >= (3, 8):
python/pyspark/pandas/tests/test_typedef.py: if sys.version_info >= (3, 8):
python/pyspark/pandas/typedef/typehints.py: if sys.version_info >= (3, 8):
python/pyspark/sql/connect/_typing.py:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/base_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/catalog_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/commands_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/common_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/example_plugins_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/expressions_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/relations_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/types_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/tests/test_worker.py: @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617098
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
To @HyukjinKwon , do we need to reevaluate every instances like this? It looks a little counter-intuitive.
```
$ git grep 'sys.version_info >'
python/pyspark/pandas/__init__.py: if sys.version_info >= (3, 7):
python/pyspark/pandas/frame.py: can_return_named_tuples = sys.version_info >= (3, 7) or len(self.columns) + index < 255
python/pyspark/pandas/tests/computation/test_apply_func.py: if sys.version_info >= (3, 8):
python/pyspark/pandas/tests/test_typedef.py: if sys.version_info >= (3, 8):
python/pyspark/pandas/typedef/typehints.py: if sys.version_info >= (3, 8):
python/pyspark/sql/connect/_typing.py:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/base_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/catalog_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/commands_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/common_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/example_plugins_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/expressions_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/relations_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/types_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/tests/test_worker.py: @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
```
```
$ git grep 'sys.version_info <'
core/src/main/scala/org/apache/spark/TestUtils.scala: val pythonSnippet = s"import sys; sys.exit(sys.version_info < ($major, $minor, $reversion))"
dev/run-tests:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; print(sys.version_info < (3, 8, 0))')
dev/run-tests-jenkins:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; print(sys.version_info < (3, 8, 0))')
python/pyspark/cloudpickle/compat.py:if sys.version_info < (3, 8):
python/pyspark/pandas/typedef/typehints.py: if sys.version_info < (3, 11):
python/pyspark/shuffle.py: if sys.version_info < (3, 11):
python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py: @unittest.skipIf(sys.version_info < (3, 9), "Type hinting generics require Python 3.9.")
python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py: sys.version_info < (3, 9),
python/run-tests:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; print(sys.version_info < (3, 8, 0))')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408614514
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
It was my bad. I am fixing it together:
```
Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info > (3, 12)
False
>>> sys.version_info > (3, 11)
True
```
```
>>> (3, 12, 0) > (3, 12)
True
>>> (3, 11, 0) > (3, 12)
False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408626232
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
So, is it impossible to be **equal** because one is 3-item tuple and the other is 2-item tuple?
```
>>> (3, 12, 0) == (3, 12)
False
>>> (3, 12, 0) <= (3, 12)
False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #44065:
URL: https://github.com/apache/spark/pull/44065#issuecomment-1831144154
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617846
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408613472
##########
python/pyspark/ml/torch/tests/test_distributor.py:
##########
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
test_file_path, learning_rate_str
)
+ @unittest.skipIf(
+ sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with Python 3.12"
Review Comment:
```suggestion
sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with Python 3.12"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408615284
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
Oh, interesting.
##########
python/pyspark/ml/torch/tests/test_distributor.py:
##########
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
test_file_path, learning_rate_str
)
+ @unittest.skipIf(
+ sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with Python 3.12"
Review Comment:
Is this correct? The new commit seems to be unable to skip 3.12.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408614954
##########
python/pyspark/ml/torch/tests/test_distributor.py:
##########
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
test_file_path, learning_rate_str
)
+ @unittest.skipIf(
+ sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with Python 3.12"
Review Comment:
Is this correct? The new commit seems to be unable to skip 3.12.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #44065:
URL: https://github.com/apache/spark/pull/44065#issuecomment-1831051264
Build: https://github.com/HyukjinKwon/spark/actions/runs/7026809016/job/19120174720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408613472
##########
python/pyspark/ml/torch/tests/test_distributor.py:
##########
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
test_file_path, learning_rate_str
)
+ @unittest.skipIf(
+ sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with Python 3.12"
Review Comment:
```suggestion
sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with Python 3.12"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44065:
URL: https://github.com/apache/spark/pull/44065#issuecomment-1831055718
Thank you for taking care of them, @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617846
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
sure - lemme check them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408649869
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
Yeah, I think so actually. For that, let me take a separate look for individual (with reading https://docs.python.org/3/reference/expressions.html#value-comparisons carefully), and fix them if there's anything wrong (or that looks weird).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #44065: [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12
URL: https://github.com/apache/spark/pull/44065
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408612421
##########
python/pyspark/ml/torch/tests/test_distributor.py:
##########
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
test_file_path, learning_rate_str
)
+ @unittest.skipIf(
+ sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with Python 3.12"
Review Comment:
```suggestion
sys.version_info < (3, 12), "SPARK-46078: Fails with dev torch with Python 3.12"
```
##########
python/pyspark/ml/torch/tests/test_distributor.py:
##########
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
test_file_path, learning_rate_str
)
+ @unittest.skipIf(
+ sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with Python 3.12"
Review Comment:
```suggestion
sys.version_info < (3, 12), "SPARK-46078: Fails with dev torch with Python 3.12"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617098
##########
python/pyspark/tests/test_worker.py:
##########
@@ -230,7 +230,7 @@ def conf(cls):
_conf.set("spark.python.worker.faulthandler.enabled", "true")
return _conf
- @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
+ @unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with Python 3.12")
Review Comment:
To @HyukjinKwon , do we need to reevaluate every instances?
```
$ git grep 'sys.version_info >'
python/pyspark/pandas/__init__.py: if sys.version_info >= (3, 7):
python/pyspark/pandas/frame.py: can_return_named_tuples = sys.version_info >= (3, 7) or len(self.columns) + index < 255
python/pyspark/pandas/tests/computation/test_apply_func.py: if sys.version_info >= (3, 8):
python/pyspark/pandas/tests/test_typedef.py: if sys.version_info >= (3, 8):
python/pyspark/pandas/typedef/typehints.py: if sys.version_info >= (3, 8):
python/pyspark/sql/connect/_typing.py:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/base_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/catalog_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/commands_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/common_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/example_plugins_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/expressions_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/relations_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/types_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/tests/test_worker.py: @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org