You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/07/07 23:48:14 UTC
[PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
viirya opened a new pull request, #57099:
URL: https://github.com/apache/spark/pull/57099
### What changes were proposed in this pull request?
Add `ArrowTableToRowsConversion._to_pylist`, which converts Arrow list-typed columns to Python values in bulk: the flattened child values are converted with a single recursive `to_pylist` call, and the resulting Python list is sliced per row using the offsets and the validity bitmap. It is used in the Arrow-to-rows conversion paths (Spark Connect `collect()`, Arrow batch UDF inputs, Arrow UDTF inputs). Non-list columns, map columns and environments without NumPy fall back to plain `column.to_pylist()`.
Leaf values are still converted by Arrow's own `to_pylist`, so results are exactly identical to `column.to_pylist()`: `None` stays `None` and values inside numeric lists stay Python ints. NumPy is used only for the offsets (non-null integers) and the validity bitmap (booleans), never for the values, so the type coercion problems of a pandas round trip (`list<int32>` of `[1, None, 3]` becoming `[1.0, nan, 3.0]`) cannot occur.
This is an interim measure for a PyArrow-side inefficiency: `Array.to_pylist()` materializes one Scalar per element, and for list types each row additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array wrapper before converting elements one by one (root-caused in apache/arrow#50326, fix proposed in apache/arrow#50327). The helper documents this and can be removed once the minimum supported PyArrow version includes the upstream fix.
### Why are the changes needed?
Converting Arrow list columns to Python rows is the hot path of Arrow-optimized Python UDF inputs and Spark Connect `collect()`. With plain `to_pylist()` it is several times slower than necessary, which caused a performance regression on array columns when Arrow serialization became the default for regular Python UDFs.
ASV microbenchmark (`python/benchmarks/bench_arrow.py::ArrowListColumnToRowsBenchmark`, added in this PR; 1M rows, macOS arm64, PyArrow 24.0.0):
| case | `to_pylist()` | this PR | speedup |
|---|---|---|---|
| `list<string>` | 769 ms | 507 ms | 1.5x |
| `list<list<int32>>` with nulls | 1.86 s | 537 ms | 3.5x |
Peak memory is unchanged.
### Does this PR introduce _any_ user-facing change?
No. Only performance; conversion results are byte-identical (covered by exact-type tests).
### How was this patch tested?
New `ArrowColumnToPylistTests` in `python/pyspark/sql/tests/test_conversion.py`, comparing `_to_pylist` against `column.to_pylist()` with exact element-type assertions across list/large_list/nested/struct/map/fixed-size-list columns, sliced and chunked variants, plus a dedicated test that `list<int32>` of `[1, None, 3]` stays ints, and an end-to-end `ArrowTableToRowsConversion.convert` test with array columns. Full `test_conversion.py` passes. The new ASV benchmark class parametrizes `baseline` vs `bulk`, so the comparison above is reproducible with `./python/asv run --python=same --quick -b 'bench_arrow.ArrowListColumnToRowsBenchmark'`.
### Was this patch authored or co-authored using generative AI tooling?
Yes. This pull request and its description were written by Isaac (Claude Code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4939189513
Thanks @viirya for the fix! I will cut RC6 as soon as this is in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3561857051
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
Review Comment:
Let's not do this. Just use `pa.types`. We use `pa_type` a lot in this file as a pyarrow datatype variable. It would be confusing to developers when this now becomes a module. LLM sometimes overestimate the cost to access an attribute of an object - 99% of the time this does not make a difference at all.
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ if _has_fast_native_to_pylist() or not _is_numpy_available():
+ # Recent PyArrow converts without per-element Scalars natively
+ # (apache/arrow#50326); without NumPy the bulk paths below are
+ # unavailable. Either way, use the native conversion.
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
Review Comment:
I don't think this is really necessary.
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ if _has_fast_native_to_pylist() or not _is_numpy_available():
+ # Recent PyArrow converts without per-element Scalars natively
+ # (apache/arrow#50326); without NumPy the bulk paths below are
+ # unavailable. Either way, use the native conversion.
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
Review Comment:
Is the type hint here necessary? I thought `[]` indicates `List[Any]`. Maybe I'm wrong. It's possible that due to the closure this requires some annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4939594733
@gaogaotiantian Both done in 5ce205e5d3a:
1. The module-level block is gone — the version and NumPy checks are folded into a single `ArrowTableToRowsConversion._should_manual_bulk()` staticmethod with `@functools.cache` (I put it on `ArrowTableToRowsConversion` since that's the class that owns `_to_pylist`). Tests patch the method directly instead of poking module globals, so removing the whole thing later is a clean two-site deletion.
2. Both docstrings now state the helpers are internal-only and that they (together with the manual bulk paths) *should* be removed once the minimum supported PyArrow version contains the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3546858615
##########
python/benchmarks/bench_arrow.py:
##########
@@ -114,3 +114,43 @@ def time_long_with_nulls_to_pandas_ext(self, n_rows, method):
def peakmem_long_with_nulls_to_pandas_ext(self, n_rows, method):
self.run_long_with_nulls_to_pandas_ext(n_rows, method)
+
+
+class ArrowListColumnToRowsBenchmark:
+ """
+ Benchmark for converting Arrow list-typed columns to Python rows, the hot
+ path of Arrow-optimized Python UDF inputs and Spark Connect collect().
+
+ ``baseline`` measures plain ``column.to_pylist()``; ``bulk`` measures
+ ``ArrowTableToRowsConversion._to_pylist`` (see apache/arrow#50326).
+ """
+
+ params = [
+ [100000, 1000000],
+ ["baseline", "bulk"],
+ ]
+ param_names = ["n_rows", "method"]
+
+ def setup(self, n_rows, method):
+ from pyspark.sql.conversion import ArrowTableToRowsConversion
+
+ self.list_of_strings = pa.array(
+ [[f"s{i}", f"t{i}"] for i in range(n_rows)], type=pa.list_(pa.string())
+ )
+ self.nested_ints_with_nulls = pa.array(
+ [[[i, i + 1], None, [i + 2]] if i % 10 != 0 else None for i in range(n_rows)],
+ type=pa.list_(pa.list_(pa.int32())),
+ )
+ if method == "bulk":
+ self.convert = ArrowTableToRowsConversion._to_pylist
+ else:
+ self.convert = lambda column: column.to_pylist()
+
+ def time_list_of_strings_to_rows(self, n_rows, method):
+ self.convert(self.list_of_strings)
+
+ def time_nested_ints_with_nulls_to_rows(self, n_rows, method):
+ self.convert(self.nested_ints_with_nulls)
+
+ def peakmem_list_of_strings_to_rows(self, n_rows, method):
Review Comment:
Could we add a peakmem benchmark for the nested list case, and ideally one nested `array<struct<...>>` like case too?
The new path materializes the flattened child values first and do slices to create rows. this is supposed to cause the memory usage higher.
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +980,63 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ try:
+ import numpy # noqa: F401
+ except ImportError:
+ return column.to_pylist()
Review Comment:
minor: since we are recursively calling this method, this try except and import can be executed multiple times. do you think we can improve here to treat with or without numpy as a argument, instead of try to import it as detection in every invocation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4938706991
Please give me some time to review this. Will do this asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4940332379
Thanks @dbtsai @dongjoon-hyun @Yicong-Huang @huaxingao @gaogaotiantian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3562024514
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ if _has_fast_native_to_pylist() or not _is_numpy_available():
+ # Recent PyArrow converts without per-element Scalars natively
+ # (apache/arrow#50326); without NumPy the bulk paths below are
+ # unavailable. Either way, use the native conversion.
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
Review Comment:
I meant assigning `column.type` to `column_type`. I think it's also some optimization that tries to save attribute access cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4940325881
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/484342ab403271283b2298c43e5e93c3d6d0bc87
- merged into branch-4.x https://github.com/apache/spark/commit/ad0eebf8c3aa3fe34f86fc38082c6ecc731bb9f7
- merged into branch-4.2 https://github.com/apache/spark/commit/d0856c78ee1891987065aec6a28dca4d5a6c903d
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3562068032
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ if _has_fast_native_to_pylist() or not _is_numpy_available():
+ # Recent PyArrow converts without per-element Scalars natively
+ # (apache/arrow#50326); without NumPy the bulk paths below are
+ # unavailable. Either way, use the native conversion.
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
Review Comment:
Ah, I misread your comment as being about the ChunkedArray block above. Removed in a96c82b83a9 — `column.type` is used inline now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4910075217
cc @Yicong-Huang FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3540595159
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +980,60 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ try:
+ import numpy # noqa: F401
+ except ImportError:
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
+ if (pa_types.is_list(column_type) or pa_types.is_large_list(column_type)) and len(
+ column
+ ) > 0:
+ n = len(column)
+ offsets = column.offsets.to_numpy(zero_copy_only=True).tolist()
Review Comment:
Good point — added the comment in eebbc4fd059. `zero_copy_only=True` also doubles as an assertion: if a future Arrow list variant ever produced offsets with a validity bitmap, this would fail loudly instead of silently copying.
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +980,60 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ try:
+ import numpy # noqa: F401
+ except ImportError:
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
+ if (pa_types.is_list(column_type) or pa_types.is_large_list(column_type)) and len(
+ column
+ ) > 0:
+ n = len(column)
+ offsets = column.offsets.to_numpy(zero_copy_only=True).tolist()
+ start = offsets[0]
+ flat = ArrowTableToRowsConversion._to_pylist(
+ column.values.slice(start, offsets[-1] - start)
+ )
+ if column.null_count == 0:
+ return [flat[offsets[i] - start : offsets[i + 1] - start] for i in range(n)]
+ valid = column.is_valid().to_numpy(zero_copy_only=False).tolist()
+ return [
Review Comment:
Right — row assembly is still an O(rows) Python loop; the win is that each row costs one list slice instead of a C++ scalar + Python Scalar wrapper + Python Array wrapper + generator. "Bulk" refers to the child-values conversion being a single call, not to vectorized row assembly. The fully vectorized fix is the upstream one (apache/arrow#50327); this stays intentionally simple as the interim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3561735675
##########
python/pyspark/sql/conversion.py:
##########
@@ -506,6 +506,43 @@ def convert_column(
return pa.RecordBatch.from_arrays(arrays, schema.names)
+# The pure-Python bulk conversion in ArrowTableToRowsConversion._to_pylist is
+# a workaround for PyArrow materializing one Scalar per element (see
+# apache/arrow#50326). PyArrow releases containing the fix convert natively
+# without per-element Scalars, in which case the native conversion is used
+# directly. Bump this constant if the fix ships in a different release.
+_MIN_PYARROW_NATIVE_TO_PYLIST_VERSION = "25.0.1"
+
+_pyarrow_native_to_pylist_is_fast: Optional[bool] = None
+
+
+def _has_fast_native_to_pylist() -> bool:
+ global _pyarrow_native_to_pylist_is_fast
+ if _pyarrow_native_to_pylist_is_fast is None:
+ import pyarrow as pa
+ from pyspark.loose_version import LooseVersion
+
+ _pyarrow_native_to_pylist_is_fast = LooseVersion(pa.__version__) >= LooseVersion(
+ _MIN_PYARROW_NATIVE_TO_PYLIST_VERSION
+ )
+ return _pyarrow_native_to_pylist_is_fast
+
+
+_numpy_available: Optional[bool] = None
Review Comment:
change to `has_numpy` and `np` to align with other files? for reference: https://github.com/apache/spark/pull/57163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3541724906
##########
python/pyspark/sql/tests/test_conversion.py:
##########
@@ -844,6 +844,83 @@ def test_geometry_convert_numpy(self):
self.assertEqual(len(result), 0)
Review Comment:
Good catch — added in 784974ceece, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3561992123
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ if _has_fast_native_to_pylist() or not _is_numpy_available():
+ # Recent PyArrow converts without per-element Scalars natively
+ # (apache/arrow#50326); without NumPy the bulk paths below are
+ # unavailable. Either way, use the native conversion.
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
Review Comment:
It is actually needed: `ArrowTableToRowsConversion.convert` passes `table.columns`, and `pa.Table.columns` are `ChunkedArray`s (the Spark Connect collect path). Without this branch a chunked list column falls into the list path and fails with `AttributeError: 'pyarrow.lib.ChunkedArray' object has no attribute 'offsets'`. The worker.py call sites pass `RecordBatch` columns (plain `Array`s), so only the `convert` path exercises it; the chunked variants in `ArrowColumnToPylistTests` cover it.
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
Review Comment:
Done in 8e24c2fa2ef — dropped the alias and use `pa.types.is_list` / `pa.types.is_large_list` directly. Agreed that `pa_types` reads confusingly close to the `pa_type` variables in this file.
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +1017,64 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ if _has_fast_native_to_pylist() or not _is_numpy_available():
+ # Recent PyArrow converts without per-element Scalars natively
+ # (apache/arrow#50326); without NumPy the bulk paths below are
+ # unavailable. Either way, use the native conversion.
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
Review Comment:
You're right — mypy infers `list[Any]` from the `extend` in the same scope. Removed in 8e24c2fa2ef and verified with the pinned mypy 1.19.1.
##########
python/pyspark/sql/conversion.py:
##########
@@ -506,6 +506,43 @@ def convert_column(
return pa.RecordBatch.from_arrays(arrays, schema.names)
+# The pure-Python bulk conversion in ArrowTableToRowsConversion._to_pylist is
+# a workaround for PyArrow materializing one Scalar per element (see
+# apache/arrow#50326). PyArrow releases containing the fix convert natively
+# without per-element Scalars, in which case the native conversion is used
+# directly. Bump this constant if the fix ships in a different release.
+_MIN_PYARROW_NATIVE_TO_PYLIST_VERSION = "25.0.1"
+
+_pyarrow_native_to_pylist_is_fast: Optional[bool] = None
+
+
+def _has_fast_native_to_pylist() -> bool:
+ global _pyarrow_native_to_pylist_is_fast
+ if _pyarrow_native_to_pylist_is_fast is None:
+ import pyarrow as pa
+ from pyspark.loose_version import LooseVersion
+
+ _pyarrow_native_to_pylist_is_fast = LooseVersion(pa.__version__) >= LooseVersion(
+ _MIN_PYARROW_NATIVE_TO_PYLIST_VERSION
+ )
+ return _pyarrow_native_to_pylist_is_fast
+
+
+_numpy_available: Optional[bool] = None
Review Comment:
Renamed the cached flag to `has_numpy` in 8e24c2fa2ef to align with #57163. I didn't cache an `np` module global though: unlike `stateful_processor_api_client.py`, this file never references the numpy module directly — numpy is only needed because `pyarrow.Array.to_numpy()` requires it — so `np` would have no user here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4938438883
I will go to merge this. @gaogaotiantian Do you want to do a review before that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3541183382
##########
python/pyspark/sql/tests/test_conversion.py:
##########
@@ -844,6 +844,83 @@ def test_geometry_convert_numpy(self):
self.assertEqual(len(result), 0)
Review Comment:
We need `skipIf` like the other arrow-based test cases.
```suggestion
@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3540541738
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +980,60 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ try:
+ import numpy # noqa: F401
+ except ImportError:
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
+ if (pa_types.is_list(column_type) or pa_types.is_large_list(column_type)) and len(
+ column
+ ) > 0:
+ n = len(column)
+ offsets = column.offsets.to_numpy(zero_copy_only=True).tolist()
+ start = offsets[0]
+ flat = ArrowTableToRowsConversion._to_pylist(
+ column.values.slice(start, offsets[-1] - start)
+ )
+ if column.null_count == 0:
+ return [flat[offsets[i] - start : offsets[i + 1] - start] for i in range(n)]
+ valid = column.is_valid().to_numpy(zero_copy_only=False).tolist()
+ return [
Review Comment:
Minor (no change needed): the per-row slicing here is still O(rows) Python (`offsets.tolist()` + a comprehension slicing `flat`), so the win is fewer object allocations, not vectorization -- consistent with the benchmark numbers. Fine as the documented interim tradeoff; noting it so the "bulk" framing isn't read as fully vectorized.
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +980,60 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ try:
+ import numpy # noqa: F401
+ except ImportError:
+ return column.to_pylist()
+
+ if isinstance(column, pa.ChunkedArray):
+ result: List[Any] = []
+ for chunk in column.chunks:
+ result.extend(ArrowTableToRowsConversion._to_pylist(chunk))
+ return result
+
+ column_type = column.type
+ if (pa_types.is_list(column_type) or pa_types.is_large_list(column_type)) and len(
+ column
+ ) > 0:
+ n = len(column)
+ offsets = column.offsets.to_numpy(zero_copy_only=True).tolist()
Review Comment:
Minor: `zero_copy_only=True` here is load-bearing and works only because list offset buffers never carry a validity bitmap. That invariant is correct, but non-obvious -- a one-line comment would stop a future reader from "fixing" it to `zero_copy_only=False` (and would flag the assumption if some new Arrow list variant ever violated it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #57099:
URL: https://github.com/apache/spark/pull/57099#discussion_r3547004147
##########
python/pyspark/sql/conversion.py:
##########
@@ -980,6 +980,63 @@ class ArrowTableToRowsConversion:
Conversion from Arrow Table to Rows.
"""
+ @staticmethod
+ def _to_pylist(column: Union["pa.Array", "pa.ChunkedArray"]) -> List[Any]:
+ """
+ Equivalent to ``column.to_pylist()``, but converts (nested) list columns in bulk
+ instead of one scalar at a time.
+
+ ``Array.to_pylist()`` materializes one Scalar per element; for list types each row
+ additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array
+ wrapper for the row's values before converting elements one by one, which is
+ several times slower than converting the flattened child values in a single pass
+ and slicing the resulting Python list per row (see apache/arrow#50326). The values
+ themselves are still converted by Arrow's own ``to_pylist``, so results are exactly
+ identical: ``None`` stays ``None`` and values inside numeric lists stay Python ints,
+ unlike a pandas round trip which would coerce them to floats/NaN. NumPy is used
+ only for the offsets (non-null integers) and the validity bitmap (booleans), so no
+ value coercion can occur.
+
+ This can be removed once the minimum supported PyArrow version includes the fix
+ for apache/arrow#50326.
+ """
+ import pyarrow as pa
+ import pyarrow.types as pa_types
+
+ try:
+ import numpy # noqa: F401
+ except ImportError:
+ return column.to_pylist()
Review Comment:
Good point — moved it to a module-level cached helper (`_is_numpy_available()`) in 1fd52c8e351, so the try/import runs at most once per process instead of per (recursive) invocation.
##########
python/benchmarks/bench_arrow.py:
##########
@@ -114,3 +114,43 @@ def time_long_with_nulls_to_pandas_ext(self, n_rows, method):
def peakmem_long_with_nulls_to_pandas_ext(self, n_rows, method):
self.run_long_with_nulls_to_pandas_ext(n_rows, method)
+
+
+class ArrowListColumnToRowsBenchmark:
+ """
+ Benchmark for converting Arrow list-typed columns to Python rows, the hot
+ path of Arrow-optimized Python UDF inputs and Spark Connect collect().
+
+ ``baseline`` measures plain ``column.to_pylist()``; ``bulk`` measures
+ ``ArrowTableToRowsConversion._to_pylist`` (see apache/arrow#50326).
+ """
+
+ params = [
+ [100000, 1000000],
+ ["baseline", "bulk"],
+ ]
+ param_names = ["n_rows", "method"]
+
+ def setup(self, n_rows, method):
+ from pyspark.sql.conversion import ArrowTableToRowsConversion
+
+ self.list_of_strings = pa.array(
+ [[f"s{i}", f"t{i}"] for i in range(n_rows)], type=pa.list_(pa.string())
+ )
+ self.nested_ints_with_nulls = pa.array(
+ [[[i, i + 1], None, [i + 2]] if i % 10 != 0 else None for i in range(n_rows)],
+ type=pa.list_(pa.list_(pa.int32())),
+ )
+ if method == "bulk":
+ self.convert = ArrowTableToRowsConversion._to_pylist
+ else:
+ self.convert = lambda column: column.to_pylist()
+
+ def time_list_of_strings_to_rows(self, n_rows, method):
+ self.convert(self.list_of_strings)
+
+ def time_nested_ints_with_nulls_to_rows(self, n_rows, method):
+ self.convert(self.nested_ints_with_nulls)
+
+ def peakmem_list_of_strings_to_rows(self, n_rows, method):
Review Comment:
Added `peakmem` variants for the nested case plus a new `array<struct<i:int,s:string>>` case (time + peakmem) in 1fd52c8e351. Measured (1M rows): nested `list<list<int32>>` 862M → 862M (no change); `array<struct>` 885M → 921M (~4%). The flattened child conversion only adds a transient pointer list — the leaf objects themselves are shared between the flat list and the per-row slices — so the peak increase stays small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #57099: [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk
URL: https://github.com/apache/spark/pull/57099
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4939537181
Thanks @huaxingao! Addressing review comments now — will keep this moving so RC6 isn't blocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57099:
URL: https://github.com/apache/spark/pull/57099#issuecomment-4928697283
Added one more commit (2e405eb6ffb) after discussion: since Spark 4.2 ships with this code frozen, `_to_pylist` now checks the installed PyArrow version and short-circuits to the native `column.to_pylist()` when it contains the apache/arrow#50326 fix (planned for PyArrow 25.0.1 per the dev-list discussion). This way Spark 4.2 users automatically pick up the native fast path — and any future PyArrow-side improvements — just by upgrading PyArrow, while older PyArrow keeps the bulk conversion here. The version constant is documented and can be bumped in a patch release if the fix ships in a different version. Tests pin the gate off so the bulk paths stay covered regardless of the CI's PyArrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org