You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/01/07 22:57:57 UTC
[PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Yicong-Huang opened a new pull request, #53718:
URL: https://github.com/apache/spark/pull/53718
### What changes were proposed in this pull request?
Add tests for PyArrow's `pa.array` type inference behavior. These tests monitor upstream PyArrow behavior to ensure PySpark's assumptions remain valid across versions.
The tests cover type inference across four input categories:
1. **Nullable data** - with `None` values
2. **Plain Python instances** - `list`, `tuple`
3. **Pandas instances** - `pd.Series`
4. **Numpy instances** - `np.array`
Types tested include:
- Primitive types: `int`, `float`, `string`, `bool`
- Temporal types: `date`, `datetime`, `time`, `timedelta`
- Binary and decimal types
- Various numpy dtypes (`int8/16/32/64`, `uint8/16/32/64`, `float32/64`, `datetime64`, `timedelta64`)
### Why are the changes needed?
This is part of [SPARK-54936](https://issues.apache.org/jira/browse/SPARK-54936) to monitor behavior changes from upstream dependencies. By testing PyArrow's type inference behavior, we can detect breaking changes when upgrading PyArrow versions.
### Does this PR introduce _any_ user-facing change?
No. This PR only adds tests.
### How was this patch tested?
New unit tests added:
```bash
python -m pytest python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py -v
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2687682027
##########
dev/sparktestsupport/modules.py:
##########
@@ -492,6 +492,7 @@ def __hash__(self):
# unittests for upstream projects
"pyspark.tests.upstream.pyarrow.test_pyarrow_ignore_timezone",
"pyspark.tests.upstream.pyarrow.test_pyarrow_scalar_type_inference",
+ "pyspark.tests.upstream.pyarrow.test_pyarrow_type_inference",
Review Comment:
renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670510456
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values) for all types."""
+ import pyarrow as pa
+
+ # Single None value
+ a = pa.array([None])
+ self.assertEqual(a.type, pa.null())
+
+ # Integers with None
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+
+ a = pa.array([1, 2, 3, None])
+ self.assertEqual(a.type, pa.int64())
+
+ # Floats with None
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ a = pa.array([1.0, 2.0, None])
+ self.assertEqual(a.type, pa.float64())
+
+ # Strings with None
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ a = pa.array(["a", "b", None])
+ self.assertEqual(a.type, pa.string())
+
+ # Booleans with None
+ a = pa.array([True, False, None])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Date with None
+ a = pa.array([datetime.date(2024, 1, 1), None])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime with None
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), None])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time with None
+ a = pa.array([datetime.time(12, 30, 0), None])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta with None
+ a = pa.array([datetime.timedelta(days=1), None])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary with None
+ a = pa.array([b"hello", None])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal with None
+ a = pa.array([Decimal("1.23"), None])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ def test_plain_python_list(self):
+ """Test type inference from Python lists for all types."""
+ import pyarrow as pa
+
+ # Integer list
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Float list
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ # Mixed int and float
+ a = pa.array([1, 2.0, 3])
+ self.assertEqual(a.type, pa.float64())
+
+ # String list
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean list
+ a = pa.array([True, False, True])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nested list (list of lists)
+ a = pa.array([[1, 2], [3, 4]])
+ self.assertEqual(a.type, pa.list_(pa.int64()))
+
+ # Date list
+ a = pa.array([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime list (naive - no timezone)
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time list
+ a = pa.array([datetime.time(12, 30, 0), datetime.time(13, 45, 0)])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta list (duration)
+ a = pa.array([datetime.timedelta(days=1), datetime.timedelta(hours=2)])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary list
+ a = pa.array([b"hello", b"world"])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal list
+ a = pa.array([Decimal("1.23"), Decimal("4.56")])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ a = pa.array([Decimal("123456.789"), Decimal("987654.321")])
+ self.assertEqual(a.type, pa.decimal128(9, 3))
+
+ def test_plain_python_tuple(self):
+ """Test type inference from Python tuples for all types."""
+ import pyarrow as pa
+
+ # Tuple of integers
+ a = pa.array((1, 2, 3))
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Tuple of floats
+ a = pa.array((1.0, 2.0, 3.0))
+ self.assertEqual(a.type, pa.float64())
+
+ # Tuple of strings
+ a = pa.array(("a", "b", "c"))
+ self.assertEqual(a.type, pa.string())
+
+ # Tuple of booleans
+ a = pa.array((True, False, True))
+ self.assertEqual(a.type, pa.bool_())
+
+ # Tuple of dates
+ a = pa.array((datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)))
+ self.assertEqual(a.type, pa.date32())
+
+ # Tuple of datetimes
+ a = pa.array((datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)))
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Tuple of times
+ a = pa.array((datetime.time(12, 30, 0), datetime.time(13, 45, 0)))
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Tuple of timedeltas
+ a = pa.array((datetime.timedelta(days=1), datetime.timedelta(hours=2)))
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Tuple of bytes
+ a = pa.array((b"hello", b"world"))
+ self.assertEqual(a.type, pa.binary())
+
+ # Tuple of decimals
+ a = pa.array((Decimal("1.23"), Decimal("4.56")))
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ @unittest.skipIf(not have_pandas, pandas_requirement_message)
+ def test_pandas_series(self):
+ """Test type inference from pandas Series for all types."""
+ import pyarrow as pa
+ import pandas as pd
+
+ # ========== numpy-backed Series (default) ==========
+
+ # Integer Series (numpy int64)
+ s = pd.Series([1, 2, 3])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Float Series (numpy float64)
+ s = pd.Series([1.0, 2.0, 3.0])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # String Series (object dtype)
+ s = pd.Series(["a", "b", "c"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean Series (numpy bool)
+ s = pd.Series([True, False, True])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Series with None (converts to float due to numpy)
+ s = pd.Series([1, 2, None])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Date Series (object dtype with date objects)
+ s = pd.Series([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime Series (pandas Timestamp, numpy datetime64[ns])
+ s = pd.Series(pd.to_datetime(["2024-01-01", "2024-01-02"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.timestamp("ns"))
+
+ # Timedelta Series (numpy timedelta64[ns])
+ s = pd.Series(pd.to_timedelta(["1 day", "2 hours"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.duration("ns"))
+
+ # Binary Series (object dtype with bytes)
+ s = pd.Series([b"hello", b"world"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.binary())
+
+ # ========== Pandas nullable extension types ==========
+
+ # Nullable Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.Int8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Nullable Integer with None
+ s = pd.Series([1, 2, None], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Unsigned Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.UInt8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint64())
+
+ # Nullable Float types
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ s = pd.Series([1.0, 2.0, None], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Nullable Boolean type
+ s = pd.Series([True, False, True], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ s = pd.Series([True, False, None], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nullable String type
+ s = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ s = pd.Series(["a", "b", None], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # ========== PyArrow-backed Series (pd.ArrowDtype) ==========
+
+ # ArrowDtype - integers
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int8()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # ArrowDtype - floats
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float32()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # ArrowDtype - string (large_string)
+ s = pd.Series(["a", "b", "c"], dtype=pd.ArrowDtype(pa.large_string()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_string())
+
+ # ArrowDtype - binary (large_binary)
+ s = pd.Series([b"hello", b"world"], dtype=pd.ArrowDtype(pa.large_binary()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_binary())
+
+ # ArrowDtype - date
+ s = pd.Series(
+ [datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)],
+ dtype=pd.ArrowDtype(pa.date32()),
+ )
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # ArrowDtype - timestamp with timezone
+ tz = "UTC"
+ pa_type = pa.timestamp("us", tz=tz)
+ s = pd.Series(
+ [datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)],
+ dtype=pd.ArrowDtype(pa_type),
+ )
+ a = pa.array(s)
+ self.assertEqual(a.type, pa_type)
Review Comment:
shall we also test nested pandas series?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670544998
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
Review Comment:
> it would be harder to find the failed pair
assertions supports error message like `self.assertEqual(a, b, f"{a} {b}")`
> some of the cases need special comments
the comments can be interleaved with codes
```
for value, expected_type in [
# comments
([None], pa.null()),
# comments
...
]:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2674393221
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values) for all types."""
+ import pyarrow as pa
+
+ # Single None value
+ a = pa.array([None])
+ self.assertEqual(a.type, pa.null())
+
+ # Integers with None
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+
+ a = pa.array([1, 2, 3, None])
+ self.assertEqual(a.type, pa.int64())
+
+ # Floats with None
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ a = pa.array([1.0, 2.0, None])
+ self.assertEqual(a.type, pa.float64())
+
+ # Strings with None
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ a = pa.array(["a", "b", None])
+ self.assertEqual(a.type, pa.string())
+
+ # Booleans with None
+ a = pa.array([True, False, None])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Date with None
+ a = pa.array([datetime.date(2024, 1, 1), None])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime with None
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), None])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time with None
+ a = pa.array([datetime.time(12, 30, 0), None])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta with None
+ a = pa.array([datetime.timedelta(days=1), None])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary with None
+ a = pa.array([b"hello", None])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal with None
+ a = pa.array([Decimal("1.23"), None])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ def test_plain_python_list(self):
+ """Test type inference from Python lists for all types."""
+ import pyarrow as pa
+
+ # Integer list
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Float list
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ # Mixed int and float
+ a = pa.array([1, 2.0, 3])
+ self.assertEqual(a.type, pa.float64())
+
+ # String list
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean list
+ a = pa.array([True, False, True])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nested list (list of lists)
+ a = pa.array([[1, 2], [3, 4]])
+ self.assertEqual(a.type, pa.list_(pa.int64()))
+
+ # Date list
+ a = pa.array([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime list (naive - no timezone)
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time list
+ a = pa.array([datetime.time(12, 30, 0), datetime.time(13, 45, 0)])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta list (duration)
+ a = pa.array([datetime.timedelta(days=1), datetime.timedelta(hours=2)])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary list
+ a = pa.array([b"hello", b"world"])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal list
+ a = pa.array([Decimal("1.23"), Decimal("4.56")])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ a = pa.array([Decimal("123456.789"), Decimal("987654.321")])
+ self.assertEqual(a.type, pa.decimal128(9, 3))
+
+ def test_plain_python_tuple(self):
+ """Test type inference from Python tuples for all types."""
+ import pyarrow as pa
+
+ # Tuple of integers
+ a = pa.array((1, 2, 3))
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Tuple of floats
+ a = pa.array((1.0, 2.0, 3.0))
+ self.assertEqual(a.type, pa.float64())
+
+ # Tuple of strings
+ a = pa.array(("a", "b", "c"))
+ self.assertEqual(a.type, pa.string())
+
+ # Tuple of booleans
+ a = pa.array((True, False, True))
+ self.assertEqual(a.type, pa.bool_())
+
+ # Tuple of dates
+ a = pa.array((datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)))
+ self.assertEqual(a.type, pa.date32())
+
+ # Tuple of datetimes
+ a = pa.array((datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)))
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Tuple of times
+ a = pa.array((datetime.time(12, 30, 0), datetime.time(13, 45, 0)))
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Tuple of timedeltas
+ a = pa.array((datetime.timedelta(days=1), datetime.timedelta(hours=2)))
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Tuple of bytes
+ a = pa.array((b"hello", b"world"))
+ self.assertEqual(a.type, pa.binary())
+
+ # Tuple of decimals
+ a = pa.array((Decimal("1.23"), Decimal("4.56")))
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ @unittest.skipIf(not have_pandas, pandas_requirement_message)
+ def test_pandas_series(self):
+ """Test type inference from pandas Series for all types."""
+ import pyarrow as pa
+ import pandas as pd
+
+ # ========== numpy-backed Series (default) ==========
+
+ # Integer Series (numpy int64)
+ s = pd.Series([1, 2, 3])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Float Series (numpy float64)
+ s = pd.Series([1.0, 2.0, 3.0])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # String Series (object dtype)
+ s = pd.Series(["a", "b", "c"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean Series (numpy bool)
+ s = pd.Series([True, False, True])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Series with None (converts to float due to numpy)
+ s = pd.Series([1, 2, None])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Date Series (object dtype with date objects)
+ s = pd.Series([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime Series (pandas Timestamp, numpy datetime64[ns])
+ s = pd.Series(pd.to_datetime(["2024-01-01", "2024-01-02"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.timestamp("ns"))
+
+ # Timedelta Series (numpy timedelta64[ns])
+ s = pd.Series(pd.to_timedelta(["1 day", "2 hours"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.duration("ns"))
+
+ # Binary Series (object dtype with bytes)
+ s = pd.Series([b"hello", b"world"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.binary())
+
+ # ========== Pandas nullable extension types ==========
+
+ # Nullable Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.Int8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Nullable Integer with None
+ s = pd.Series([1, 2, None], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Unsigned Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.UInt8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint64())
+
+ # Nullable Float types
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ s = pd.Series([1.0, 2.0, None], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Nullable Boolean type
+ s = pd.Series([True, False, True], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ s = pd.Series([True, False, None], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nullable String type
+ s = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ s = pd.Series(["a", "b", None], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # ========== PyArrow-backed Series (pd.ArrowDtype) ==========
+
+ # ArrowDtype - integers
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int8()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # ArrowDtype - floats
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float32()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # ArrowDtype - string (large_string)
+ s = pd.Series(["a", "b", "c"], dtype=pd.ArrowDtype(pa.large_string()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_string())
+
+ # ArrowDtype - binary (large_binary)
+ s = pd.Series([b"hello", b"world"], dtype=pd.ArrowDtype(pa.large_binary()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_binary())
+
+ # ArrowDtype - date
+ s = pd.Series(
+ [datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)],
+ dtype=pd.ArrowDtype(pa.date32()),
+ )
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # ArrowDtype - timestamp with timezone
+ tz = "UTC"
+ pa_type = pa.timestamp("us", tz=tz)
+ s = pd.Series(
+ [datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)],
+ dtype=pd.ArrowDtype(pa_type),
+ )
+ a = pa.array(s)
+ self.assertEqual(a.type, pa_type)
Review Comment:
added some cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2674392942
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670725135
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
Review Comment:
let's also consider `dict`
```
pa.array([{}]).type
pa.array([{"a":3,"b":None,"c":"s"}]).type
pa.array([{"a":3,"b":"s"}, {"x":5, "a":1.0}]).type
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #53718:
URL: https://github.com/apache/spark/pull/53718#issuecomment-3747018999
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2686188258
##########
dev/sparktestsupport/modules.py:
##########
@@ -492,6 +492,7 @@ def __hash__(self):
# unittests for upstream projects
"pyspark.tests.upstream.pyarrow.test_pyarrow_ignore_timezone",
"pyspark.tests.upstream.pyarrow.test_pyarrow_scalar_type_inference",
+ "pyspark.tests.upstream.pyarrow.test_pyarrow_type_inference",
Review Comment:
let's rename the file `test_pyarrow_array_type_inference`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2683587202
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Test pa.array type inference for various input types.
+
+This test monitors the behavior of PyArrow's type inference to ensure
+PySpark's assumptions about PyArrow behavior remain valid across versions.
+
+Test cases are organized by input category:
+1. Nullable data - None value handling
+2. Python list - Plain Python lists
+3. Python tuple - Plain Python tuples
+4. Python dict - Plain Python dicts (inferred as struct)
+5. Pandas Series (numpy-backed)
+6. Pandas nullable extension types
+7. Pandas ArrowDtype
+8. NumPy array
+9. Nested list types
+10. Explicit type specification
+"""
+
+import datetime
+import unittest
+from decimal import Decimal
+from zoneinfo import ZoneInfo
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ # =========================================================================
+ # Helper method
+ # =========================================================================
+ def _run_inference_tests(self, cases, use_expected_as_explicit=False):
+ """Run type inference tests from a list of (name, data, expected_type) tuples.
+
+ If use_expected_as_explicit is True, pass expected_type as explicit type to pa.array().
+ """
+ import pyarrow as pa
+
+ for name, data, expected_type in cases:
+ with self.subTest(name=name):
+ if use_expected_as_explicit:
+ arr = pa.array(data, type=expected_type)
+ else:
+ arr = pa.array(data)
+ self.assertEqual(arr.type, expected_type, f"Failed for {name}")
+
+ # =========================================================================
+ # 1. NULLABLE DATA - Test None value handling
+ # =========================================================================
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values)."""
+ import pyarrow as pa
+
+ cases = [
+ # fmt: off
Review Comment:
sure, I have removed `name` field and also applied the format.
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Test pa.array type inference for various input types.
+
+This test monitors the behavior of PyArrow's type inference to ensure
+PySpark's assumptions about PyArrow behavior remain valid across versions.
+
+Test cases are organized by input category:
+1. Nullable data - None value handling
+2. Python list - Plain Python lists
+3. Python tuple - Plain Python tuples
+4. Python dict - Plain Python dicts (inferred as struct)
+5. Pandas Series (numpy-backed)
+6. Pandas nullable extension types
+7. Pandas ArrowDtype
+8. NumPy array
+9. Nested list types
+10. Explicit type specification
+"""
+
+import datetime
+import unittest
+from decimal import Decimal
+from zoneinfo import ZoneInfo
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ # =========================================================================
+ # Helper method
+ # =========================================================================
+ def _run_inference_tests(self, cases, use_expected_as_explicit=False):
+ """Run type inference tests from a list of (name, data, expected_type) tuples.
+
+ If use_expected_as_explicit is True, pass expected_type as explicit type to pa.array().
+ """
+ import pyarrow as pa
+
+ for name, data, expected_type in cases:
+ with self.subTest(name=name):
Review Comment:
removed both!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670549778
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
Review Comment:
Those feel more like workarounds than clear benefits. Is there a concrete advantage to switching to a for loop in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53718:
URL: https://github.com/apache/spark/pull/53718#issuecomment-3721173464
## JIRA Issue Information
=== Sub-task SPARK-54938 ===
Summary: Add tests for pa.array type inference
Assignee: None
Status: Open
Affected: ["4.2.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670488536
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
Review Comment:
@Yicong-Huang shall we check the inference with a for loop like
```
for value, expected_type in [
([None], pa.null())
...
]:
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2674071329
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values) for all types."""
+ import pyarrow as pa
+
+ # Single None value
+ a = pa.array([None])
+ self.assertEqual(a.type, pa.null())
+
+ # Integers with None
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+
+ a = pa.array([1, 2, 3, None])
+ self.assertEqual(a.type, pa.int64())
+
+ # Floats with None
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ a = pa.array([1.0, 2.0, None])
+ self.assertEqual(a.type, pa.float64())
+
+ # Strings with None
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ a = pa.array(["a", "b", None])
+ self.assertEqual(a.type, pa.string())
+
+ # Booleans with None
+ a = pa.array([True, False, None])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Date with None
+ a = pa.array([datetime.date(2024, 1, 1), None])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime with None
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), None])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time with None
+ a = pa.array([datetime.time(12, 30, 0), None])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta with None
+ a = pa.array([datetime.timedelta(days=1), None])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary with None
+ a = pa.array([b"hello", None])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal with None
+ a = pa.array([Decimal("1.23"), None])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ def test_plain_python_list(self):
+ """Test type inference from Python lists for all types."""
+ import pyarrow as pa
+
+ # Integer list
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Float list
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ # Mixed int and float
+ a = pa.array([1, 2.0, 3])
+ self.assertEqual(a.type, pa.float64())
+
+ # String list
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean list
+ a = pa.array([True, False, True])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nested list (list of lists)
+ a = pa.array([[1, 2], [3, 4]])
+ self.assertEqual(a.type, pa.list_(pa.int64()))
+
+ # Date list
+ a = pa.array([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime list (naive - no timezone)
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time list
+ a = pa.array([datetime.time(12, 30, 0), datetime.time(13, 45, 0)])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta list (duration)
+ a = pa.array([datetime.timedelta(days=1), datetime.timedelta(hours=2)])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary list
+ a = pa.array([b"hello", b"world"])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal list
+ a = pa.array([Decimal("1.23"), Decimal("4.56")])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ a = pa.array([Decimal("123456.789"), Decimal("987654.321")])
+ self.assertEqual(a.type, pa.decimal128(9, 3))
+
+ def test_plain_python_tuple(self):
+ """Test type inference from Python tuples for all types."""
+ import pyarrow as pa
+
+ # Tuple of integers
+ a = pa.array((1, 2, 3))
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Tuple of floats
+ a = pa.array((1.0, 2.0, 3.0))
+ self.assertEqual(a.type, pa.float64())
+
+ # Tuple of strings
+ a = pa.array(("a", "b", "c"))
+ self.assertEqual(a.type, pa.string())
+
+ # Tuple of booleans
+ a = pa.array((True, False, True))
+ self.assertEqual(a.type, pa.bool_())
+
+ # Tuple of dates
+ a = pa.array((datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)))
+ self.assertEqual(a.type, pa.date32())
+
+ # Tuple of datetimes
+ a = pa.array((datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)))
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Tuple of times
+ a = pa.array((datetime.time(12, 30, 0), datetime.time(13, 45, 0)))
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Tuple of timedeltas
+ a = pa.array((datetime.timedelta(days=1), datetime.timedelta(hours=2)))
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Tuple of bytes
+ a = pa.array((b"hello", b"world"))
+ self.assertEqual(a.type, pa.binary())
+
+ # Tuple of decimals
+ a = pa.array((Decimal("1.23"), Decimal("4.56")))
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ @unittest.skipIf(not have_pandas, pandas_requirement_message)
+ def test_pandas_series(self):
+ """Test type inference from pandas Series for all types."""
+ import pyarrow as pa
+ import pandas as pd
+
+ # ========== numpy-backed Series (default) ==========
+
+ # Integer Series (numpy int64)
+ s = pd.Series([1, 2, 3])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Float Series (numpy float64)
+ s = pd.Series([1.0, 2.0, 3.0])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # String Series (object dtype)
+ s = pd.Series(["a", "b", "c"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean Series (numpy bool)
+ s = pd.Series([True, False, True])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Series with None (converts to float due to numpy)
+ s = pd.Series([1, 2, None])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Date Series (object dtype with date objects)
+ s = pd.Series([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime Series (pandas Timestamp, numpy datetime64[ns])
+ s = pd.Series(pd.to_datetime(["2024-01-01", "2024-01-02"]))
Review Comment:
added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2674439495
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Test pa.array type inference for various input types.
+
+This test monitors the behavior of PyArrow's type inference to ensure
+PySpark's assumptions about PyArrow behavior remain valid across versions.
+
+Test cases are organized by input category:
+1. Nullable data - None value handling
+2. Python list - Plain Python lists
+3. Python tuple - Plain Python tuples
+4. Python dict - Plain Python dicts (inferred as struct)
+5. Pandas Series (numpy-backed)
+6. Pandas nullable extension types
+7. Pandas ArrowDtype
+8. NumPy array
+9. Nested list types
+10. Explicit type specification
+"""
+
+import datetime
+import unittest
+from decimal import Decimal
+from zoneinfo import ZoneInfo
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ # =========================================================================
+ # Helper method
+ # =========================================================================
+ def _run_inference_tests(self, cases, use_expected_as_explicit=False):
+ """Run type inference tests from a list of (name, data, expected_type) tuples.
+
+ If use_expected_as_explicit is True, pass expected_type as explicit type to pa.array().
+ """
+ import pyarrow as pa
+
+ for name, data, expected_type in cases:
+ with self.subTest(name=name):
Review Comment:
`subTest` will generate a lot of log in the file, I feel we can just remove it.
and the `name` field is not necessary.
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,434 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Test pa.array type inference for various input types.
+
+This test monitors the behavior of PyArrow's type inference to ensure
+PySpark's assumptions about PyArrow behavior remain valid across versions.
+
+Test cases are organized by input category:
+1. Nullable data - None value handling
+2. Python list - Plain Python lists
+3. Python tuple - Plain Python tuples
+4. Python dict - Plain Python dicts (inferred as struct)
+5. Pandas Series (numpy-backed)
+6. Pandas nullable extension types
+7. Pandas ArrowDtype
+8. NumPy array
+9. Nested list types
+10. Explicit type specification
+"""
+
+import datetime
+import unittest
+from decimal import Decimal
+from zoneinfo import ZoneInfo
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ # =========================================================================
+ # Helper method
+ # =========================================================================
+ def _run_inference_tests(self, cases, use_expected_as_explicit=False):
+ """Run type inference tests from a list of (name, data, expected_type) tuples.
+
+ If use_expected_as_explicit is True, pass expected_type as explicit type to pa.array().
+ """
+ import pyarrow as pa
+
+ for name, data, expected_type in cases:
+ with self.subTest(name=name):
+ if use_expected_as_explicit:
+ arr = pa.array(data, type=expected_type)
+ else:
+ arr = pa.array(data)
+ self.assertEqual(arr.type, expected_type, f"Failed for {name}")
+
+ # =========================================================================
+ # 1. NULLABLE DATA - Test None value handling
+ # =========================================================================
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values)."""
+ import pyarrow as pa
+
+ cases = [
+ # fmt: off
Review Comment:
I feel enabling the fmt won't hurt the readability too much, if we remove the `name` field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670499442
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values) for all types."""
+ import pyarrow as pa
+
+ # Single None value
+ a = pa.array([None])
+ self.assertEqual(a.type, pa.null())
+
+ # Integers with None
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+
+ a = pa.array([1, 2, 3, None])
+ self.assertEqual(a.type, pa.int64())
+
+ # Floats with None
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ a = pa.array([1.0, 2.0, None])
+ self.assertEqual(a.type, pa.float64())
+
+ # Strings with None
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ a = pa.array(["a", "b", None])
+ self.assertEqual(a.type, pa.string())
+
+ # Booleans with None
+ a = pa.array([True, False, None])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Date with None
+ a = pa.array([datetime.date(2024, 1, 1), None])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime with None
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), None])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time with None
+ a = pa.array([datetime.time(12, 30, 0), None])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta with None
+ a = pa.array([datetime.timedelta(days=1), None])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary with None
+ a = pa.array([b"hello", None])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal with None
+ a = pa.array([Decimal("1.23"), None])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ def test_plain_python_list(self):
+ """Test type inference from Python lists for all types."""
+ import pyarrow as pa
+
+ # Integer list
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Float list
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ # Mixed int and float
+ a = pa.array([1, 2.0, 3])
+ self.assertEqual(a.type, pa.float64())
+
+ # String list
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean list
+ a = pa.array([True, False, True])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nested list (list of lists)
+ a = pa.array([[1, 2], [3, 4]])
+ self.assertEqual(a.type, pa.list_(pa.int64()))
+
+ # Date list
+ a = pa.array([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime list (naive - no timezone)
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time list
+ a = pa.array([datetime.time(12, 30, 0), datetime.time(13, 45, 0)])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta list (duration)
+ a = pa.array([datetime.timedelta(days=1), datetime.timedelta(hours=2)])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary list
+ a = pa.array([b"hello", b"world"])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal list
+ a = pa.array([Decimal("1.23"), Decimal("4.56")])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ a = pa.array([Decimal("123456.789"), Decimal("987654.321")])
+ self.assertEqual(a.type, pa.decimal128(9, 3))
+
+ def test_plain_python_tuple(self):
+ """Test type inference from Python tuples for all types."""
+ import pyarrow as pa
+
+ # Tuple of integers
+ a = pa.array((1, 2, 3))
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Tuple of floats
+ a = pa.array((1.0, 2.0, 3.0))
+ self.assertEqual(a.type, pa.float64())
+
+ # Tuple of strings
+ a = pa.array(("a", "b", "c"))
+ self.assertEqual(a.type, pa.string())
+
+ # Tuple of booleans
+ a = pa.array((True, False, True))
+ self.assertEqual(a.type, pa.bool_())
+
+ # Tuple of dates
+ a = pa.array((datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)))
+ self.assertEqual(a.type, pa.date32())
+
+ # Tuple of datetimes
+ a = pa.array((datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)))
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Tuple of times
+ a = pa.array((datetime.time(12, 30, 0), datetime.time(13, 45, 0)))
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Tuple of timedeltas
+ a = pa.array((datetime.timedelta(days=1), datetime.timedelta(hours=2)))
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Tuple of bytes
+ a = pa.array((b"hello", b"world"))
+ self.assertEqual(a.type, pa.binary())
+
+ # Tuple of decimals
+ a = pa.array((Decimal("1.23"), Decimal("4.56")))
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ @unittest.skipIf(not have_pandas, pandas_requirement_message)
+ def test_pandas_series(self):
+ """Test type inference from pandas Series for all types."""
+ import pyarrow as pa
+ import pandas as pd
+
+ # ========== numpy-backed Series (default) ==========
+
+ # Integer Series (numpy int64)
+ s = pd.Series([1, 2, 3])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Float Series (numpy float64)
+ s = pd.Series([1.0, 2.0, 3.0])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # String Series (object dtype)
+ s = pd.Series(["a", "b", "c"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean Series (numpy bool)
+ s = pd.Series([True, False, True])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Series with None (converts to float due to numpy)
+ s = pd.Series([1, 2, None])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Date Series (object dtype with date objects)
+ s = pd.Series([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime Series (pandas Timestamp, numpy datetime64[ns])
+ s = pd.Series(pd.to_datetime(["2024-01-01", "2024-01-02"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.timestamp("ns"))
+
+ # Timedelta Series (numpy timedelta64[ns])
+ s = pd.Series(pd.to_timedelta(["1 day", "2 hours"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.duration("ns"))
+
+ # Binary Series (object dtype with bytes)
+ s = pd.Series([b"hello", b"world"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.binary())
+
+ # ========== Pandas nullable extension types ==========
+
+ # Nullable Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.Int8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Nullable Integer with None
+ s = pd.Series([1, 2, None], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Unsigned Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.UInt8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint64())
+
+ # Nullable Float types
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ s = pd.Series([1.0, 2.0, None], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Nullable Boolean type
+ s = pd.Series([True, False, True], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ s = pd.Series([True, False, None], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nullable String type
+ s = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ s = pd.Series(["a", "b", None], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # ========== PyArrow-backed Series (pd.ArrowDtype) ==========
+
+ # ArrowDtype - integers
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int8()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # ArrowDtype - floats
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float32()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # ArrowDtype - string (large_string)
+ s = pd.Series(["a", "b", "c"], dtype=pd.ArrowDtype(pa.large_string()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_string())
+
+ # ArrowDtype - binary (large_binary)
+ s = pd.Series([b"hello", b"world"], dtype=pd.ArrowDtype(pa.large_binary()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_binary())
+
+ # ArrowDtype - date
+ s = pd.Series(
+ [datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)],
+ dtype=pd.ArrowDtype(pa.date32()),
+ )
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # ArrowDtype - timestamp with timezone
+ tz = "UTC"
Review Comment:
for timezone, let's also test a non-UTC timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670506072
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values) for all types."""
+ import pyarrow as pa
+
+ # Single None value
+ a = pa.array([None])
+ self.assertEqual(a.type, pa.null())
+
+ # Integers with None
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+
+ a = pa.array([1, 2, 3, None])
+ self.assertEqual(a.type, pa.int64())
+
+ # Floats with None
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ a = pa.array([1.0, 2.0, None])
+ self.assertEqual(a.type, pa.float64())
+
+ # Strings with None
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ a = pa.array(["a", "b", None])
+ self.assertEqual(a.type, pa.string())
+
+ # Booleans with None
+ a = pa.array([True, False, None])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Date with None
+ a = pa.array([datetime.date(2024, 1, 1), None])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime with None
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), None])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time with None
+ a = pa.array([datetime.time(12, 30, 0), None])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta with None
+ a = pa.array([datetime.timedelta(days=1), None])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary with None
+ a = pa.array([b"hello", None])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal with None
+ a = pa.array([Decimal("1.23"), None])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ def test_plain_python_list(self):
+ """Test type inference from Python lists for all types."""
+ import pyarrow as pa
+
+ # Integer list
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Float list
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ # Mixed int and float
+ a = pa.array([1, 2.0, 3])
+ self.assertEqual(a.type, pa.float64())
+
+ # String list
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean list
+ a = pa.array([True, False, True])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nested list (list of lists)
+ a = pa.array([[1, 2], [3, 4]])
+ self.assertEqual(a.type, pa.list_(pa.int64()))
+
+ # Date list
+ a = pa.array([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime list (naive - no timezone)
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time list
+ a = pa.array([datetime.time(12, 30, 0), datetime.time(13, 45, 0)])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta list (duration)
+ a = pa.array([datetime.timedelta(days=1), datetime.timedelta(hours=2)])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary list
+ a = pa.array([b"hello", b"world"])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal list
+ a = pa.array([Decimal("1.23"), Decimal("4.56")])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ a = pa.array([Decimal("123456.789"), Decimal("987654.321")])
+ self.assertEqual(a.type, pa.decimal128(9, 3))
+
+ def test_plain_python_tuple(self):
+ """Test type inference from Python tuples for all types."""
+ import pyarrow as pa
+
+ # Tuple of integers
+ a = pa.array((1, 2, 3))
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Tuple of floats
+ a = pa.array((1.0, 2.0, 3.0))
+ self.assertEqual(a.type, pa.float64())
+
+ # Tuple of strings
+ a = pa.array(("a", "b", "c"))
+ self.assertEqual(a.type, pa.string())
+
+ # Tuple of booleans
+ a = pa.array((True, False, True))
+ self.assertEqual(a.type, pa.bool_())
+
+ # Tuple of dates
+ a = pa.array((datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)))
+ self.assertEqual(a.type, pa.date32())
+
+ # Tuple of datetimes
+ a = pa.array((datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)))
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Tuple of times
+ a = pa.array((datetime.time(12, 30, 0), datetime.time(13, 45, 0)))
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Tuple of timedeltas
+ a = pa.array((datetime.timedelta(days=1), datetime.timedelta(hours=2)))
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Tuple of bytes
+ a = pa.array((b"hello", b"world"))
+ self.assertEqual(a.type, pa.binary())
+
+ # Tuple of decimals
+ a = pa.array((Decimal("1.23"), Decimal("4.56")))
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ @unittest.skipIf(not have_pandas, pandas_requirement_message)
+ def test_pandas_series(self):
+ """Test type inference from pandas Series for all types."""
+ import pyarrow as pa
+ import pandas as pd
+
+ # ========== numpy-backed Series (default) ==========
+
+ # Integer Series (numpy int64)
+ s = pd.Series([1, 2, 3])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Float Series (numpy float64)
+ s = pd.Series([1.0, 2.0, 3.0])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # String Series (object dtype)
+ s = pd.Series(["a", "b", "c"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean Series (numpy bool)
+ s = pd.Series([True, False, True])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Series with None (converts to float due to numpy)
+ s = pd.Series([1, 2, None])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Date Series (object dtype with date objects)
+ s = pd.Series([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime Series (pandas Timestamp, numpy datetime64[ns])
+ s = pd.Series(pd.to_datetime(["2024-01-01", "2024-01-02"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.timestamp("ns"))
+
+ # Timedelta Series (numpy timedelta64[ns])
+ s = pd.Series(pd.to_timedelta(["1 day", "2 hours"]))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.duration("ns"))
+
+ # Binary Series (object dtype with bytes)
+ s = pd.Series([b"hello", b"world"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.binary())
+
+ # ========== Pandas nullable extension types ==========
+
+ # Nullable Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.Int8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Nullable Integer with None
+ s = pd.Series([1, 2, None], dtype=pd.Int64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Unsigned Integer types (all sizes)
+ s = pd.Series([1, 2, 3], dtype=pd.UInt8Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt16Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint16())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint32())
+
+ s = pd.Series([1, 2, 3], dtype=pd.UInt64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.uint64())
+
+ # Nullable Float types
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float32Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ s = pd.Series([1.0, 2.0, None], dtype=pd.Float64Dtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Nullable Boolean type
+ s = pd.Series([True, False, True], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ s = pd.Series([True, False, None], dtype=pd.BooleanDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nullable String type
+ s = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ s = pd.Series(["a", "b", None], dtype=pd.StringDtype())
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # ========== PyArrow-backed Series (pd.ArrowDtype) ==========
+
+ # ArrowDtype - integers
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int8()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int8())
+
+ s = pd.Series([1, 2, 3], dtype=pd.ArrowDtype(pa.int64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # ArrowDtype - floats
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float32()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float32())
+
+ s = pd.Series([1.0, 2.0, 3.0], dtype=pd.ArrowDtype(pa.float64()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # ArrowDtype - string (large_string)
+ s = pd.Series(["a", "b", "c"], dtype=pd.ArrowDtype(pa.large_string()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_string())
+
+ # ArrowDtype - binary (large_binary)
+ s = pd.Series([b"hello", b"world"], dtype=pd.ArrowDtype(pa.large_binary()))
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.large_binary())
+
+ # ArrowDtype - date
+ s = pd.Series(
+ [datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)],
+ dtype=pd.ArrowDtype(pa.date32()),
+ )
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # ArrowDtype - timestamp with timezone
+ tz = "UTC"
Review Comment:
sure will add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670503818
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
+ """Test type inference with nullable data (None values) for all types."""
+ import pyarrow as pa
+
+ # Single None value
+ a = pa.array([None])
+ self.assertEqual(a.type, pa.null())
+
+ # Integers with None
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+
+ a = pa.array([1, 2, 3, None])
+ self.assertEqual(a.type, pa.int64())
+
+ # Floats with None
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ a = pa.array([1.0, 2.0, None])
+ self.assertEqual(a.type, pa.float64())
+
+ # Strings with None
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ a = pa.array(["a", "b", None])
+ self.assertEqual(a.type, pa.string())
+
+ # Booleans with None
+ a = pa.array([True, False, None])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Date with None
+ a = pa.array([datetime.date(2024, 1, 1), None])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime with None
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), None])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time with None
+ a = pa.array([datetime.time(12, 30, 0), None])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta with None
+ a = pa.array([datetime.timedelta(days=1), None])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary with None
+ a = pa.array([b"hello", None])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal with None
+ a = pa.array([Decimal("1.23"), None])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ def test_plain_python_list(self):
+ """Test type inference from Python lists for all types."""
+ import pyarrow as pa
+
+ # Integer list
+ a = pa.array([1, 2, 3])
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Float list
+ a = pa.array([1.0, 2.0, 3.0])
+ self.assertEqual(a.type, pa.float64())
+
+ # Mixed int and float
+ a = pa.array([1, 2.0, 3])
+ self.assertEqual(a.type, pa.float64())
+
+ # String list
+ a = pa.array(["a", "b", "c"])
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean list
+ a = pa.array([True, False, True])
+ self.assertEqual(a.type, pa.bool_())
+
+ # Nested list (list of lists)
+ a = pa.array([[1, 2], [3, 4]])
+ self.assertEqual(a.type, pa.list_(pa.int64()))
+
+ # Date list
+ a = pa.array([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime list (naive - no timezone)
+ a = pa.array([datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)])
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Time list
+ a = pa.array([datetime.time(12, 30, 0), datetime.time(13, 45, 0)])
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Timedelta list (duration)
+ a = pa.array([datetime.timedelta(days=1), datetime.timedelta(hours=2)])
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Binary list
+ a = pa.array([b"hello", b"world"])
+ self.assertEqual(a.type, pa.binary())
+
+ # Decimal list
+ a = pa.array([Decimal("1.23"), Decimal("4.56")])
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ a = pa.array([Decimal("123456.789"), Decimal("987654.321")])
+ self.assertEqual(a.type, pa.decimal128(9, 3))
+
+ def test_plain_python_tuple(self):
+ """Test type inference from Python tuples for all types."""
+ import pyarrow as pa
+
+ # Tuple of integers
+ a = pa.array((1, 2, 3))
+ self.assertEqual(a.type, pa.int64())
+ self.assertEqual(a.to_pylist(), [1, 2, 3])
+
+ # Tuple of floats
+ a = pa.array((1.0, 2.0, 3.0))
+ self.assertEqual(a.type, pa.float64())
+
+ # Tuple of strings
+ a = pa.array(("a", "b", "c"))
+ self.assertEqual(a.type, pa.string())
+
+ # Tuple of booleans
+ a = pa.array((True, False, True))
+ self.assertEqual(a.type, pa.bool_())
+
+ # Tuple of dates
+ a = pa.array((datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)))
+ self.assertEqual(a.type, pa.date32())
+
+ # Tuple of datetimes
+ a = pa.array((datetime.datetime(2024, 1, 1, 12, 0, 0), datetime.datetime(2024, 1, 2, 12, 0, 0)))
+ self.assertEqual(a.type, pa.timestamp("us"))
+
+ # Tuple of times
+ a = pa.array((datetime.time(12, 30, 0), datetime.time(13, 45, 0)))
+ self.assertEqual(a.type, pa.time64("us"))
+
+ # Tuple of timedeltas
+ a = pa.array((datetime.timedelta(days=1), datetime.timedelta(hours=2)))
+ self.assertEqual(a.type, pa.duration("us"))
+
+ # Tuple of bytes
+ a = pa.array((b"hello", b"world"))
+ self.assertEqual(a.type, pa.binary())
+
+ # Tuple of decimals
+ a = pa.array((Decimal("1.23"), Decimal("4.56")))
+ self.assertEqual(a.type, pa.decimal128(3, 2))
+
+ @unittest.skipIf(not have_pandas, pandas_requirement_message)
+ def test_pandas_series(self):
+ """Test type inference from pandas Series for all types."""
+ import pyarrow as pa
+ import pandas as pd
+
+ # ========== numpy-backed Series (default) ==========
+
+ # Integer Series (numpy int64)
+ s = pd.Series([1, 2, 3])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.int64())
+
+ # Float Series (numpy float64)
+ s = pd.Series([1.0, 2.0, 3.0])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # String Series (object dtype)
+ s = pd.Series(["a", "b", "c"])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.string())
+
+ # Boolean Series (numpy bool)
+ s = pd.Series([True, False, True])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.bool_())
+
+ # Series with None (converts to float due to numpy)
+ s = pd.Series([1, 2, None])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.float64())
+
+ # Date Series (object dtype with date objects)
+ s = pd.Series([datetime.date(2024, 1, 1), datetime.date(2024, 1, 2)])
+ a = pa.array(s)
+ self.assertEqual(a.type, pa.date32())
+
+ # Datetime Series (pandas Timestamp, numpy datetime64[ns])
+ s = pd.Series(pd.to_datetime(["2024-01-01", "2024-01-02"]))
Review Comment:
lets also test datetime.datetime and pd.Timestamp with timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670500949
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
Review Comment:
I'd prefer explicit checks instead of group input and expected value with a for loop.
- the current version is easier to read. for for loop, if anything breaks, it would be harder to find the failed pair.
- some of the cases need special comments. commenting in a list of pairs would also be harder to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #53718:
URL: https://github.com/apache/spark/pull/53718#issuecomment-3721242081
cc @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #53718:
URL: https://github.com/apache/spark/pull/53718#discussion_r2670572642
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py:
##########
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import datetime
+from decimal import Decimal
+import unittest
+
+from pyspark.testing.utils import (
+ have_pandas,
+ have_pyarrow,
+ pandas_requirement_message,
+ pyarrow_requirement_message,
+)
+
+
+# Test pa.array type inference for various input types.
+# This test monitors the behavior of PyArrow's type inference to ensure
+# PySpark's assumptions about PyArrow behavior remain valid across versions.
+#
+# Key input types tested:
+# 1. nullable data (with None values)
+# 2. plain Python instances (list, tuple, array)
+# 3. pandas instances (pd.Series)
+# 4. numpy instances (np.array)
+
+
+@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+class PyArrowTypeInferenceTests(unittest.TestCase):
+ """Test PyArrow's type inference behavior for pa.array."""
+
+ def test_nullable_data(self):
Review Comment:
to make code concise, so that developers can review more cases in one screen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #53718: [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference
URL: https://github.com/apache/spark/pull/53718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org