You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "luc-pimentel (via GitHub)" <gi...@apache.org> on 2026/10/09 16:51:09 UTC
[PR] [SPARK-60009][PYTHON] Honor Arrow-backed pandas conversion types and safety [spark]
luc-pimentel opened a new pull request, #59326:
URL: https://github.com/apache/spark/pull/59326
### What changes were proposed in this pull request?
Use a shared pandas-to-Arrow array constructor for DataFrame input and pandas-UDF output. Arrow-array protocol results are extracted first, then explicitly cast with the requested type and safe-conversion setting.
Add tests for requested types, nullable/chunked inputs, safe and unsafe narrowing, and string/binary offset widths. Add a PySpark migration note.
### Why are the changes needed?
SPARK-60009: on PyArrow 18, protocol-backed pandas columns can ignore the requested type and bypass overflow checks. With safe conversion enabled, converting an Arrow-backed value of 128 to ByteType silently returns -128. A pandas UDF declared as LongType can also fail with ARROW_TYPE_MISMATCH.
On PyArrow 19, the implicit protocol cast uses safe conversion even when Spark's setting is disabled.
### Does this PR introduce _any_ user-facing change?
Yes. Arrow-backed pandas columns honor requested schema types and spark.sql.execution.pandas.convertToArrowArraySafely. Safe overflow conversions are rejected, and explicitly unsafe conversions are allowed.
Null handling, NumPy-backed conversion, and string/binary offset-width behavior are preserved.
### How was this patch tested?
- Built Spark with Hive support using build/sbt -Phive package.
- Added four tests that fail on the original code with PyArrow 18 and pass with the fix. A fifth test checks that string and binary conversions still work correctly.
- All 54 conversion-module tests passed on PyArrow 18.0.0, 19.0.0, and 25.0.1.
- Local JVM checks passed for DataFrame inputs, pandas-UDF output, overflow rejection, and explicitly unsafe casts on all three Arrow versions.
- Spark's Python compilation, custom-error checks, Ruff checks, and formatting passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenCode 1.18.35 (openai/gpt-6.1-sol)
This contribution is my original work, and I license it under the Apache License 2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60009][PYTHON] Honor Arrow-backed pandas conversion types and safety [spark]
Posted by "luc-pimentel (via GitHub)" <gi...@apache.org>.
luc-pimentel commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4237990630
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,33 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pyarrow as pa
+
+ if mask is None:
Review Comment:
Good catch, thanks. The branch now applies only to `pd.arrays.ArrowExtensionArray`, so `Int64`, `Float64`, `boolean` and `string[python]` keep the typed `from_pandas` call and mismatches raise as before. Added `test_nullable_dtype_type_mismatch` and `test_nullable_dtype_requested_type`, both going through `PandasToArrowConversion.from_pandas` and `create_arrow_array_from_pandas`.
The mismatch test fails on the previous head and passes now. The full conversion module (56 tests) passes on PyArrow 18.0.0 and 19.0.0 with pandas 2.2.3, on 25.0.1 with pandas 2.3.3, and on 25.0.1 with pandas 3.0.6. The six local JVM checks pass on all three Arrow versions.
---
Generated-by: Claude Code 2.1.296 (claude-fable-5-1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60009][PYTHON] Honor Arrow-backed pandas conversion types and safety [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4237806939
##########
python/pyspark/sql/tests/test_conversion.py:
##########
@@ -439,6 +440,73 @@ def test_from_pandas_arrow_cast(self):
result = PandasToArrowConversion.from_pandas(df, schema, arrow_cast=True)
self.assertEqual(result.column(0).to_pylist(), [1, 2, 3])
+ def test_from_pandas_arrow_dtype_requested_type(self):
Review Comment:
**Nit (P3):** All five new tests use `pd.ArrowDtype` series, but `_create_array_from_pandas` takes the same branch for every `__arrow_array__` input, including `Int64`, `Float64`, `boolean`, and `string[python]`. Nothing here, in `arrow/test_arrow.py`, or in the coercion goldens converts those dtypes to a mismatched type or uses `safecheck=False`. As a result, the nullable-dtype coercion change in the helper passes the suite. Could you add nullable-dtype cases that pin the intended behavior, through both `PandasToArrowConversion.from_pandas` and `create_arrow_array_from_pandas`?
See **Shared repair plan 1** in the review body.
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,33 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pyarrow as pa
+
+ if mask is None:
Review Comment:
**Non-blocking (P2):** Here `mask is None` means "the array has `__arrow_array__`", which is broader than "the array is Arrow-backed". It is also true for pandas' NumPy-backed nullable arrays (`Int64`, `Float64`, `boolean`, ...) and for `string[python]`. Their `__arrow_array__` already honors the requested type (`pa.array(self._data, mask=self._mask, type=type)`), so before this change a mismatched type raised `ArrowTypeError`. They now go through `Array.cast` instead, on every PyArrow version:
- A pandas UDF declared `StringType` that returns `pd.Series([1, None], dtype="Int64")` used to fail with `PySparkTypeError`. It now returns `['1', None]`; `Float64` gives `'1.5'` and `boolean` gives `'true'`. A NumPy `int64` series with the same values still fails. This bypasses the non-legacy rule in `PandasToArrowConversion._convert_column_legacy` ("ArrowTypeError must NOT be silently cast").
- `createDataFrame` with a `StringType` schema accepts these columns the same way, and `string[python]` values are now parsed into integer columns.
- `Int64` for `BinaryType` or `StructType` now escapes as a raw `ArrowNotImplementedError` instead of a `PySpark*Error` or the previous result.
I confirmed these by running the old and new code paths on pyarrow 25.0.1 / pandas 3.0.6. Could this path apply only when `isinstance(series.array, pd.arrays.ArrowExtensionArray)`? That covers `pd.ArrowDtype`, `string[pyarrow]`, and pandas 3's default `str`. Every other `__arrow_array__` implementer would keep the typed `pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)` call.
See **Shared repair plan 1** in the review body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60009][PYTHON] Honor Arrow-backed pandas conversion types and safety [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4238252067
##########
python/pyspark/sql/conversion.py:
##########
@@ -543,7 +544,9 @@ def _convert_column_legacy(
# trigger the cast fallback when arrow_cast is enabled.
try:
try:
- return pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)
+ return _create_array_from_pandas(
Review Comment:
**Nit (P3):** No test covers this legacy-mode call with a pyarrow-backed series. It is the branch `worker.py`'s `convert_df_to_arrow` uses for Arrow UDTF output. The new SPARK-60009 tests either go through `create_arrow_array_from_pandas` or call `from_pandas` with the default `use_legacy_error_handling=False`. `test_from_pandas_use_legacy_error_handling` only uses inputs that fail under both the old and new code.
If this line went back to `pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)`, a `pd.ArrowDtype(pa.int32())` series for a `LongType` field would come back as int32 again on PyArrow 18. The protocol ignores the type and nothing raises, so the `arrow_cast` fallback never runs, and no test would fail.
Could you add a `use_legacy_error_handling=True, arrow_cast=True` variant of `test_from_pandas_arrow_dtype_requested_type`? Like the other requested-type checks, it would only fail on the PyArrow 18 job, but it would cover this branch.
**Verification:**
- **Regression:** Legacy-mode conversion of a pyarrow-backed series yields the requested Arrow type and values, and the check fails on pyarrow < 19 when that call site bypasses the shared helper.
- **Compatibility:** The existing conversion tests still pass, including the legacy error-format test and the non-legacy requested-type, safety, offset-width, and nullable-dtype tests.
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,33 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pyarrow as pa
+
+ if mask is None:
Review Comment:
Thanks, confirmed. `_create_array_from_pandas` now takes the extract-then-cast path only for `pd.arrays.ArrowExtensionArray`, so `Int64`, `Float64`, `boolean`, and `string[python]` keep the typed `pa.Array.from_pandas(..., type=...)` call and still raise on mismatches. `test_nullable_dtype_type_mismatch` and `test_nullable_dtype_requested_type` cover this through both `PandasToArrowConversion.from_pandas` and `create_arrow_array_from_pandas`, and they pass for me on pyarrow 25.0.1 / pandas 3.0.6.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4237806937","thread_id":"inline:4237806937","verdict_sha256":"88252b55a6909ad22fa8d910e12351cff39c30e7209785e39c45de82d7b0151b"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60009][PYTHON] Honor Arrow-backed pandas conversion types and safety [spark]
Posted by "luc-pimentel (via GitHub)" <gi...@apache.org>.
luc-pimentel commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4239342359
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,34 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pandas as pd
+ import pyarrow as pa
+
+ if isinstance(series.array, pd.arrays.ArrowExtensionArray):
+ # pyarrow-backed arrays can ignore the requested type or force a safe cast.
+ # Extract first, then cast explicitly. Other dtypes already honor the type.
+ result = pa.Array.from_pandas(series, safe=safecheck)
+ if arrow_type is not None and result.type != arrow_type:
+ result = result.cast(arrow_type, safe=safecheck)
Review Comment:
Confirmed the shift here too: a tz-naive `pd.ArrowDtype` timestamp under `America/Sao_Paulo` comes out three hours early while the NumPy series is right. The localization you describe is what #58765 (SPARK-59467) adds in `_check_series_convert_timestamps_internal`, including the `ambiguous` handling, so I'd rather not duplicate it here. Once that lands, the cast in this helper only relabels the timezone and the instants line up. I'll keep this PR to the type and safecheck handling.
---
Generated-by: Claude Code 2.1.296 (claude-fable-5-1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60009][PYTHON] Honor Arrow-backed pandas conversion types and safety [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4238550089
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,34 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pandas as pd
+ import pyarrow as pa
+
+ if isinstance(series.array, pd.arrays.ArrowExtensionArray):
+ # pyarrow-backed arrays can ignore the requested type or force a safe cast.
+ # Extract first, then cast explicitly. Other dtypes already honor the type.
+ result = pa.Array.from_pandas(series, safe=safecheck)
+ if arrow_type is not None and result.type != arrow_type:
+ result = result.cast(arrow_type, safe=safecheck)
Review Comment:
**Non-blocking (P2):** Take a tz-naive `pd.ArrowDtype` timestamp series and a `TimestampType` field. This line casts `timestamp[ns]` to `timestamp('us', tz=<session tz>)` by changing only the metadata, so Arrow reads the wall-clock values as UTC. NumPy `datetime64` series get localized to the session timezone first, in `_check_series_convert_timestamps_internal`. That function only checks `is_datetime64_dtype` and `DatetimeTZDtype`, and both are False for `ArrowDtype`.
Example with session timezone `America/Los_Angeles`: `pd.Series(pd.to_datetime(['2020-01-01 10:00']), dtype=pd.ArrowDtype(pa.timestamp('ns')))` becomes 1577872800000000 (10:00 UTC). The same NumPy series becomes 1577901600000000 (18:00 UTC). I reproduced this with the helper's calls on pyarrow 25.0.1 / pandas 3.0.6.
PyArrow >= 19 already gave the same result through its protocol cast before this PR. On PyArrow 18, the supported minimum, the requested type used to be ignored and the `timestamp[ns]` column failed downstream. Classic `createDataFrame` and pandas UDF/UDTF results now succeed with the shifted value instead. None of the new tests uses a timestamp target. tz-aware Arrow timestamps and `TimestampNTZType` targets are not affected.
**Recommended change:** In the shared pandas timestamp converter, localize tz-naive Arrow-backed timestamp series to the session (or local) timezone and normalize them to UTC, using the same DST-ambiguity choice (standard time) and nonexistent-time handling as the NumPy path. Leave tz-aware Arrow-backed timestamps unchanged. Add PySpark conversion tests for Arrow-backed TimestampType targets, and extend the PySpark migration note to cover this behavior.
**Why this works:** Once the converter emits tz-aware UTC instants for Arrow-backed naive input, the cast in `_create_array_from_pandas` from that tz-aware source to `timestamp('us', tz=<session>)` only changes the timezone label and preserves the instant. Arrow-backed and NumPy-backed inputs then produce the same values on every PyArrow version and entry point.
**Scope:** Localize tz-naive Arrow-backed timestamp columns in the shared pandas timestamp converter, so the shared Arrow constructor's cast preserves instants for every pandas backing and PyArrow version.
**Compatibility:** NumPy-backed datetime64 and DatetimeTZDtype localization, including the standard-time DST choice, stays unchanged.
**Risks:** On PyArrow >= 19 the merge target already returned shifted values for this input, so the fix changes observable results there too. The migration note must say so. pandas' ArrowDtype `.dt.tz_localize` does not accept `ambiguous=False`, so the implementation must reproduce the standard-time choice for ambiguous DST times another way.
**Constraints:** Do not change NumPy-backed `datetime64` or `DatetimeTZDtype` handling. Keep the timestamp resolution, and do not overflow dates outside the nanosecond range for microsecond-unit Arrow inputs.
**Success:** With a non-UTC session timezone, a tz-naive `pd.ArrowDtype` timestamp series converted for TimestampType through `create_arrow_array_from_pandas` and through `PandasToArrowConversion.from_pandas` (both error-handling modes) yields the same UTC instants as the equivalent NumPy `datetime64` series, on every supported PyArrow version. tz-aware Arrow-backed timestamps keep their instants, TimestampNTZType targets keep their wall-clock values, and nulls are preserved.
**Verification:**
- **Regression:** A tz-naive Arrow-backed timestamp series converted for TimestampType under a non-UTC session timezone produces the same epoch values as the equivalent NumPy-backed series, through both the createDataFrame helper and the pandas UDF output conversion.
- **Behavior:** tz-aware Arrow-backed timestamps keep their instants, and TimestampNTZType targets keep their wall-clock values, including nulls.
- **Compatibility:** The existing NumPy-backed timestamp conversion and SPARK-60009 requested-type, safety, offset-width, nullable-dtype, and legacy-path conversion checks still pass.
##########
python/pyspark/sql/conversion.py:
##########
@@ -543,7 +544,9 @@ def _convert_column_legacy(
# trigger the cast fallback when arrow_cast is enabled.
try:
try:
- return pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)
+ return _create_array_from_pandas(
Review Comment:
Thanks, confirmed. `test_from_pandas_arrow_dtype_requested_type_legacy_error_handling` runs the `use_legacy_error_handling=True, arrow_cast=True` branch with a `pd.ArrowDtype(pa.int32())` series and asserts the `int64` result type. So if that call site went back to the raw `pa.Array.from_pandas(..., type=...)` call, the test would fail on the PyArrow 18 job.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4238252067","thread_id":"inline:4238252067","verdict_sha256":"b5a8c6db037983c56343f6fda30b5b312ccbc91ff9dc23933c51b7143516b533"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60009][PYTHON] Honor Arrow-backed pandas conversion types and safety [spark]
Posted by "luc-pimentel (via GitHub)" <gi...@apache.org>.
luc-pimentel commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4238312966
##########
python/pyspark/sql/conversion.py:
##########
@@ -543,7 +544,9 @@ def _convert_column_legacy(
# trigger the cast fallback when arrow_cast is enabled.
try:
try:
- return pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)
+ return _create_array_from_pandas(
Review Comment:
Added `test_from_pandas_arrow_dtype_requested_type_legacy_error_handling` with `use_legacy_error_handling=True, arrow_cast=True`. With the legacy call site reverted to the raw constructor it fails on PyArrow 18 with int32 instead of int64; on the current code the module (57 tests) passes on PyArrow 18.0.0, 19.0.0 and 25.0.1.
---
Generated-by: Claude Code 2.1.296 (claude-fable-5-1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org