You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/05/12 00:53:47 UTC
(spark) branch master updated: [SPARK-56760][PYTHON] Remove dead numpy 1.21 version check in pandas typehints
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8a05ae713833 [SPARK-56760][PYTHON] Remove dead numpy 1.21 version check in pandas typehints
8a05ae713833 is described below
commit 8a05ae713833ea98a49c34a740cf2c6f502d9286
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Tue May 12 09:53:34 2026 +0900
[SPARK-56760][PYTHON] Remove dead numpy 1.21 version check in pandas typehints
### What changes were proposed in this pull request?
Removes the dead `if LooseVersion(np.__version__) >= LooseVersion("1.21"):` guard in:
- `python/pyspark/pandas/typedef/typehints.py` (`as_spark_type`) -- inner block handling `numpy.typing.NDArray` is dedented one level.
- `python/pyspark/pandas/tests/test_typedef.py` (`test_as_spark_type_pandas_on_spark_dtype`) -- inner assertions on `ntp.NDArray[...]` are dedented one level.
Accompanying `# For NumPy typing, NumPy version should be 1.21+` comments are removed as well.
### Why are the changes needed?
[SPARK-45179](https://issues.apache.org/jira/browse/SPARK-45179) (2023-09-15) bumped `_minimum_numpy_version` to `"1.21"` in `python/setup.py`, so this version guard is always true and is dead code.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #55746 from Yicong-Huang/SPARK-56760.
Authored-by: Yicong Huang <17...@users.noreply.github.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/pandas/tests/test_typedef.py | 20 +++++++++-----------
python/pyspark/pandas/typedef/typehints.py | 20 ++++++++------------
2 files changed, 17 insertions(+), 23 deletions(-)
diff --git a/python/pyspark/pandas/tests/test_typedef.py b/python/pyspark/pandas/tests/test_typedef.py
index cb07b36a1c99..1b9db0a6a29a 100644
--- a/python/pyspark/pandas/tests/test_typedef.py
+++ b/python/pyspark/pandas/tests/test_typedef.py
@@ -407,17 +407,15 @@ class TypeHintTestsMixin:
(np.dtype("object"), ArrayType(spark_type)),
)
- # For NumPy typing, NumPy version should be 1.21+
- if LooseVersion(np.__version__) >= LooseVersion("1.21"):
- import numpy.typing as ntp
-
- self.assertEqual(
- as_spark_type(ntp.NDArray[numpy_or_python_type]), ArrayType(spark_type)
- )
- self.assertEqual(
- pandas_on_spark_type(ntp.NDArray[numpy_or_python_type]),
- (np.dtype("object"), ArrayType(spark_type)),
- )
+ import numpy.typing as ntp
+
+ self.assertEqual(
+ as_spark_type(ntp.NDArray[numpy_or_python_type]), ArrayType(spark_type)
+ )
+ self.assertEqual(
+ pandas_on_spark_type(ntp.NDArray[numpy_or_python_type]),
+ (np.dtype("object"), ArrayType(spark_type)),
+ )
with self.assertRaisesRegex(TypeError, "Type uint64 was not understood."):
as_spark_type(np.dtype("uint64"))
diff --git a/python/pyspark/pandas/typedef/typehints.py b/python/pyspark/pandas/typedef/typehints.py
index a98691efcb0d..5c3ea90f34ee 100644
--- a/python/pyspark/pandas/typedef/typehints.py
+++ b/python/pyspark/pandas/typedef/typehints.py
@@ -150,18 +150,14 @@ def as_spark_type(
- dictionaries of field_name -> type
- Python3's typing system
"""
- # For NumPy typing, NumPy version should be 1.21+
- if LooseVersion(np.__version__) >= LooseVersion("1.21"):
- if (
- hasattr(tpe, "__origin__")
- and tpe.__origin__ is np.ndarray
- and hasattr(tpe, "__args__")
- and len(tpe.__args__) > 1
- ):
- # numpy.typing.NDArray
- return types.ArrayType(
- as_spark_type(tpe.__args__[1].__args__[0], raise_error=raise_error)
- )
+ if (
+ hasattr(tpe, "__origin__")
+ and tpe.__origin__ is np.ndarray
+ and hasattr(tpe, "__args__")
+ and len(tpe.__args__) > 1
+ ):
+ # numpy.typing.NDArray
+ return types.ArrayType(as_spark_type(tpe.__args__[1].__args__[0], raise_error=raise_error))
if isinstance(tpe, np.dtype) and tpe == np.dtype("object"):
pass
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org