You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/04/01 06:35:47 UTC
(spark) branch master updated: [SPARK-54938][PYTHON][TEST][FOLLOW-UP] Fix `test_pyarrow_array_type_inference` for pandas >= 3
This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9518075a01ff [SPARK-54938][PYTHON][TEST][FOLLOW-UP] Fix `test_pyarrow_array_type_inference` for pandas >= 3
9518075a01ff is described below
commit 9518075a01ff264226cefbfe19ff7239c9ecaf33
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Wed Apr 1 14:35:25 2026 +0800
[SPARK-54938][PYTHON][TEST][FOLLOW-UP] Fix `test_pyarrow_array_type_inference` for pandas >= 3
### What changes were proposed in this pull request?
pandas 3.x changed default string dtype to use pyarrow-backed storage, causing pa.array() to infer large_string instead of string for string Series. Conditionally expect large_string on pandas >= 3.
### Why are the changes needed?
to resolve failure in https://github.com/apache/spark/actions/runs/23819581811/job/69428367355
### Does this PR introduce _any_ user-facing change?
No, test-only
### How was this patch tested?
manually check
pandas==3.0.1
```
In [3]: import pyarrow as pa
In [4]: import pandas as pd
In [5]: ser = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())
In [6]: pa.array(ser)
Out[6]:
<pyarrow.lib.LargeStringArray object at 0x103455d80>
[
"a",
"b",
"c"
]
In [7]: pa.array(ser, pa.string())
Out[7]:
<pyarrow.lib.StringArray object at 0x1095546a0>
[
"a",
"b",
"c"
]
```
pandas==2.3.3
```
In [7]: ser = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())
In [8]: pa.array(ser)
Out[8]:
<pyarrow.lib.StringArray object at 0x10ae16620>
[
"a",
"b",
"c"
]
In [9]: pa.array(ser, pa.string())
Out[9]:
<pyarrow.lib.StringArray object at 0x10ae14a00>
[
"a",
"b",
"c"
]
```
### Was this patch authored or co-authored using generative AI tooling?
Co-authored-by: Claude code (Opus 4.6)
Closes #55125 from zhengruifeng/fix_inference_p3.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
.../upstream/pyarrow/test_pyarrow_array_type_inference.py | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py
index 1c859fc9de2a..7e60f552e9c7 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py
@@ -295,6 +295,10 @@ class PyArrowArrayTypeInferenceTests(unittest.TestCase):
import numpy as np
import pandas as pd
import pyarrow as pa
+ from pyspark.loose_version import LooseVersion
+
+ # pandas >= 3 infers large_string instead of string for object-dtype string Series
+ string_type = pa.large_string() if LooseVersion(pd.__version__) >= "3.0.0" else pa.string()
sg = ZoneInfo("Asia/Singapore")
la = "America/Los_Angeles"
@@ -315,7 +319,7 @@ class PyArrowArrayTypeInferenceTests(unittest.TestCase):
(pd.Series([np.inf, 1.0, 2.0]), pa.float64()),
(pd.Series([-np.inf, 1.0, 2.0]), pa.float64()),
# String
- (pd.Series(["a", "b", "c"]), pa.string()),
+ (pd.Series(["a", "b", "c"]), string_type),
# Boolean
(pd.Series([True, False, True]), pa.bool_()),
# Temporal
@@ -356,6 +360,10 @@ class PyArrowArrayTypeInferenceTests(unittest.TestCase):
import numpy as np
import pandas as pd
import pyarrow as pa
+ from pyspark.loose_version import LooseVersion
+
+ # pandas >= 3 uses pyarrow-backed StringDtype, which infers large_string
+ string_type = pa.large_string() if LooseVersion(pd.__version__) >= "3.0.0" else pa.string()
cases = [
# Integer
@@ -379,8 +387,8 @@ class PyArrowArrayTypeInferenceTests(unittest.TestCase):
(pd.Series([True, False, True], dtype=pd.BooleanDtype()), pa.bool_()),
(pd.Series([True, False, None], dtype=pd.BooleanDtype()), pa.bool_()),
# String
- (pd.Series(["a", "b", "c"], dtype=pd.StringDtype()), pa.string()),
- (pd.Series(["a", "b", None], dtype=pd.StringDtype()), pa.string()),
+ (pd.Series(["a", "b", "c"], dtype=pd.StringDtype()), string_type),
+ (pd.Series(["a", "b", None], dtype=pd.StringDtype()), string_type),
]
self._run_inference_tests(cases)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org