You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/01/08 01:04:13 UTC
[PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Yicong-Huang opened a new pull request, #53721:
URL: https://github.com/apache/spark/pull/53721
### What changes were proposed in this pull request?
Add comprehensive tests for PyArrow's `pa.array` type coercion behavior when using explicit `type` parameter. These tests monitor upstream PyArrow behavior to ensure PySpark's assumptions remain valid across versions.
The tests cover 12 categories with 51 test cases:
| Category | Tests |
|----------|-------|
| **Missing Values** | `None`, `NaN`, `pd.NaT`, `pd.NA` handling |
| **Empty Datasets** | Empty lists, pandas Series, numpy arrays |
| **Invalid Values** | Overflow, precision loss, incompatible types |
| **Numeric Coercion** | int ↔ float, int narrowing/widening, unsigned |
| **Decimal Coercion** | int → decimal128/256 with various precision/scale |
| **String Coercion** | int → string (requires explicit `pc.cast()`) |
| **Boolean Coercion** | int → bool (requires explicit `pc.cast()`) |
| **Temporal Coercion** | timestamp, date, time, duration resolutions |
| **Timezone Coercion** | `datetime.datetime` and `pd.Timestamp` with timezones (UTC, Asia/Singapore, America/Los_Angeles) |
| **Nested Types** | list, struct, map element type coercion |
| **Input Types** | Python list/tuple/generator, Pandas Series, NumPy arrays |
| **Spark Types** | All Spark numeric, decimal, temporal, string, binary, complex types |
Key coercion behaviors documented:
| Coercion | Implicit | Notes |
|----------|----------|-------|
| int → float64/float32 | ✅ | Works directly |
| float → int64 | ✅ | Truncates fractional part |
| int → decimal128 | ✅ | Python list only |
| int → string | ❌ | Requires `pc.cast()` |
| int → bool | ❌ | Requires `pc.cast()` |
| float ↔ decimal | ❌ | Requires `pc.cast()` |
| numpy/pandas → decimal | ❌ | Requires sufficient precision for ArrowDtype |
| timezone-aware datetime | ✅ | Converts between timezones |
### Why are the changes needed?
This is part of [SPARK-54936](https://issues.apache.org/jira/browse/SPARK-54936) to monitor behavior changes from upstream dependencies. By testing PyArrow's type coercion behavior, we can detect breaking changes when upgrading PyArrow versions.
### Does this PR introduce _any_ user-facing change?
No. This PR only adds tests.
### How was this patch tested?
New unit tests added:
```bash
python -m pytest python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_coercion.py -v
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #53721: [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion
URL: https://github.com/apache/spark/pull/53721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #53721:
URL: https://github.com/apache/spark/pull/53721#issuecomment-3752608609
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53721:
URL: https://github.com/apache/spark/pull/53721#issuecomment-3721475469
## JIRA Issue Information
=== Sub-task SPARK-54939 ===
Summary: Add tests for pa.array type coercion
Assignee: None
Status: Open
Affected: ["4.2.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53721:
URL: https://github.com/apache/spark/pull/53721#discussion_r2670765060
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_coercion.py:
##########
@@ -0,0 +1,1187 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import math
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type coercion behavior when creating arrays with explicit type parameter.
+# This test monitors the behavior of PyArrow's type coercion to ensure PySpark's assumptions
+# about PyArrow behavior remain valid across versions.
+#
+# Key findings:
+# 1. Numeric coercion (int <-> float, int size narrowing/widening) works via safe casting
+# 2. String coercion (int -> string) requires explicit pc.cast(), not implicit via type param
+# 3. Decimal coercion (int -> decimal) works directly
+# 4. Boolean coercion (int -> bool) does NOT work implicitly, requires explicit casting
+#
+# Test coverage includes:
+# - Missing values (None, NaN, NaT)
+# - Empty datasets
+# - Invalid values and error handling
+# - All Spark/PyArrow/Pandas datatypes
+# - Python, Pandas, and NumPy input types
+# - PySpark pandas_options for to_pandas()
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeCoercionTests(unittest.TestCase):
+ """Test PyArrow's type coercion behavior for pa.array with explicit type parameter."""
+
+ # ============================================================
+ # SECTION 1: Missing Values Tests (None, NaN, NaT)
+ # ============================================================
+
+ def test_none_values_in_coercion(self):
Review Comment:
shall we use a golden file to cover more cases?
or a table to store expected behaviour, something like:
```
expected_results = { expected errors, expected py_lists }
for value in values:
for pa_type in pa_types:
try:
pa_array = pa.array(value, type=pa_type)
check pa_array.py_lists
except ...
check error
```
I feel at least we can use such loop for primitive types
for complex types, we can add separate tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #53721:
URL: https://github.com/apache/spark/pull/53721#issuecomment-3750984714
@zhengruifeng could you please check this again? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53721:
URL: https://github.com/apache/spark/pull/53721#discussion_r2677966422
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_coercion.py:
##########
@@ -0,0 +1,1187 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import math
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type coercion behavior when creating arrays with explicit type parameter.
+# This test monitors the behavior of PyArrow's type coercion to ensure PySpark's assumptions
+# about PyArrow behavior remain valid across versions.
+#
+# Key findings:
+# 1. Numeric coercion (int <-> float, int size narrowing/widening) works via safe casting
+# 2. String coercion (int -> string) requires explicit pc.cast(), not implicit via type param
+# 3. Decimal coercion (int -> decimal) works directly
+# 4. Boolean coercion (int -> bool) does NOT work implicitly, requires explicit casting
+#
+# Test coverage includes:
+# - Missing values (None, NaN, NaT)
+# - Empty datasets
+# - Invalid values and error handling
+# - All Spark/PyArrow/Pandas datatypes
+# - Python, Pandas, and NumPy input types
+# - PySpark pandas_options for to_pandas()
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeCoercionTests(unittest.TestCase):
+ """Test PyArrow's type coercion behavior for pa.array with explicit type parameter."""
+
+ # ============================================================
+ # SECTION 1: Missing Values Tests (None, NaN, NaT)
+ # ============================================================
+
+ def test_none_values_in_coercion(self):
+ """Test that None values are preserved during type coercion."""
+ import pyarrow as pa
+ import pyarrow.compute as pc
+
+ # int with None -> float64
+ a = pa.array([1, None, 3], type=pa.float64())
+ self.assertEqual(a.type, pa.float64())
+ self.assertEqual(a.to_pylist(), [1.0, None, 3.0])
+ self.assertEqual(a.null_count, 1)
+
+ # int with None -> decimal128
+ a = pa.array([1, None, 3], type=pa.decimal128(10, 0))
+ self.assertEqual(a.type, pa.decimal128(10, 0))
+ self.assertEqual(a.to_pylist(), [Decimal("1"), None, Decimal("3")])
+ self.assertEqual(a.null_count, 1)
+
+ # int with None -> int32 (narrowing)
+ a = pa.array([1, None, 3], type=pa.int32())
+ self.assertEqual(a.type, pa.int32())
+ self.assertEqual(a.to_pylist(), [1, None, 3])
+ self.assertEqual(a.null_count, 1)
+
+ # explicit cast to string with None
+ int_arr = pa.array([1, None, 3])
+ str_arr = pc.cast(int_arr, pa.string())
Review Comment:
removed `cast` related tests
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_coercion.py:
##########
@@ -0,0 +1,1187 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import math
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type coercion behavior when creating arrays with explicit type parameter.
+# This test monitors the behavior of PyArrow's type coercion to ensure PySpark's assumptions
+# about PyArrow behavior remain valid across versions.
+#
+# Key findings:
+# 1. Numeric coercion (int <-> float, int size narrowing/widening) works via safe casting
+# 2. String coercion (int -> string) requires explicit pc.cast(), not implicit via type param
+# 3. Decimal coercion (int -> decimal) works directly
+# 4. Boolean coercion (int -> bool) does NOT work implicitly, requires explicit casting
+#
+# Test coverage includes:
+# - Missing values (None, NaN, NaT)
+# - Empty datasets
+# - Invalid values and error handling
+# - All Spark/PyArrow/Pandas datatypes
+# - Python, Pandas, and NumPy input types
+# - PySpark pandas_options for to_pandas()
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeCoercionTests(unittest.TestCase):
+ """Test PyArrow's type coercion behavior for pa.array with explicit type parameter."""
+
+ # ============================================================
+ # SECTION 1: Missing Values Tests (None, NaN, NaT)
+ # ============================================================
+
+ def test_none_values_in_coercion(self):
Review Comment:
used for-loop and cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53721:
URL: https://github.com/apache/spark/pull/53721#discussion_r2670765060
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_coercion.py:
##########
@@ -0,0 +1,1187 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import math
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type coercion behavior when creating arrays with explicit type parameter.
+# This test monitors the behavior of PyArrow's type coercion to ensure PySpark's assumptions
+# about PyArrow behavior remain valid across versions.
+#
+# Key findings:
+# 1. Numeric coercion (int <-> float, int size narrowing/widening) works via safe casting
+# 2. String coercion (int -> string) requires explicit pc.cast(), not implicit via type param
+# 3. Decimal coercion (int -> decimal) works directly
+# 4. Boolean coercion (int -> bool) does NOT work implicitly, requires explicit casting
+#
+# Test coverage includes:
+# - Missing values (None, NaN, NaT)
+# - Empty datasets
+# - Invalid values and error handling
+# - All Spark/PyArrow/Pandas datatypes
+# - Python, Pandas, and NumPy input types
+# - PySpark pandas_options for to_pandas()
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeCoercionTests(unittest.TestCase):
+ """Test PyArrow's type coercion behavior for pa.array with explicit type parameter."""
+
+ # ============================================================
+ # SECTION 1: Missing Values Tests (None, NaN, NaT)
+ # ============================================================
+
+ def test_none_values_in_coercion(self):
Review Comment:
shall we use a golden file to cover more cases?
or a table to store expected behaviour, something like:
```
expected_results = { expected errors, expected py_lists }
for value in values:
for pa_type in pa_types:
try:
pa_array = pa.array(value, type=pa_type)
assert pa_type == pa_array.type
check pa_array.py_lists
except ...
check error
```
I feel at least we can use such loop for primitive types
for complex types, we can add separate tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53721:
URL: https://github.com/apache/spark/pull/53721#discussion_r2670765706
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_coercion.py:
##########
@@ -0,0 +1,1187 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import math
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type coercion behavior when creating arrays with explicit type parameter.
+# This test monitors the behavior of PyArrow's type coercion to ensure PySpark's assumptions
+# about PyArrow behavior remain valid across versions.
+#
+# Key findings:
+# 1. Numeric coercion (int <-> float, int size narrowing/widening) works via safe casting
+# 2. String coercion (int -> string) requires explicit pc.cast(), not implicit via type param
+# 3. Decimal coercion (int -> decimal) works directly
+# 4. Boolean coercion (int -> bool) does NOT work implicitly, requires explicit casting
+#
+# Test coverage includes:
+# - Missing values (None, NaN, NaT)
+# - Empty datasets
+# - Invalid values and error handling
+# - All Spark/PyArrow/Pandas datatypes
+# - Python, Pandas, and NumPy input types
+# - PySpark pandas_options for to_pandas()
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeCoercionTests(unittest.TestCase):
+ """Test PyArrow's type coercion behavior for pa.array with explicit type parameter."""
+
+ # ============================================================
+ # SECTION 1: Missing Values Tests (None, NaN, NaT)
+ # ============================================================
+
+ def test_none_values_in_coercion(self):
+ """Test that None values are preserved during type coercion."""
+ import pyarrow as pa
+ import pyarrow.compute as pc
+
+ # int with None -> float64
+ a = pa.array([1, None, 3], type=pa.float64())
+ self.assertEqual(a.type, pa.float64())
+ self.assertEqual(a.to_pylist(), [1.0, None, 3.0])
+ self.assertEqual(a.null_count, 1)
+
+ # int with None -> decimal128
+ a = pa.array([1, None, 3], type=pa.decimal128(10, 0))
+ self.assertEqual(a.type, pa.decimal128(10, 0))
+ self.assertEqual(a.to_pylist(), [Decimal("1"), None, Decimal("3")])
+ self.assertEqual(a.null_count, 1)
+
+ # int with None -> int32 (narrowing)
+ a = pa.array([1, None, 3], type=pa.int32())
+ self.assertEqual(a.type, pa.int32())
+ self.assertEqual(a.to_pylist(), [1, None, 3])
+ self.assertEqual(a.null_count, 1)
+
+ # explicit cast to string with None
+ int_arr = pa.array([1, None, 3])
+ str_arr = pc.cast(int_arr, pa.string())
Review Comment:
let's test `cast` in separate PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org