You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/07/08 19:49:33 UTC
[PR] [SPARK-58050][PYTHON] Fuse per-row converters into bulk Arrow conversion for Arrow Python UDFs [spark]
viirya opened a new pull request, #57137:
URL: https://github.com/apache/spark/pull/57137
### What changes were proposed in this pull request?
Follow-up of SPARK-58019 (#57099), SPARK-58023 (#57104) and SPARK-58024 (#57105); **only the last commit is new — this PR is stacked on #57105 and will be rebased as its predecessors merge.**
Even with the bulk `_to_pylist` conversions from the predecessor PRs, the Arrow Python UDF worker still runs **per-row Python converters** on both sides of the UDF. This PR fuses them into bulk operations:
- **Input**: new `ArrowTableToRowsConversion._to_rows_column` is equivalent to `[conv(v) for v in _to_pylist(column)]` but builds map rows as dicts and struct rows as `Row`s directly from the flattened child columns, applying child converters per flattened column instead of per row. Anything not covered falls back to the per-row converter unchanged.
- **Output**: when the element/field/value converters are identity, UDF results are assembled in bulk — arrays are passed to `pa.array` directly (skipping the per-row defensive copy), map results are passed as dicts directly (PyArrow accepts dicts for map types), and struct results (tuples/`Row`s) are transposed and assembled via `pa.StructArray.from_arrays` with a null mask. Any shape/validation mismatch falls back to the existing per-row path, preserving error behavior.
### Why are the changes needed?
Worker-side profiling shows the per-row converters dominate the remaining performance gap between Arrow Python UDFs and pickled Python UDFs on nested types. End-to-end (`bench_python_udf`-style workload, 6.4M rows, macOS arm64, `local[4]`, on top of the three predecessor PRs):
| case | pickled UDF | Arrow UDF before | Arrow UDF after | delta |
|---|---|---|---|---|
| `array<string>` | 1.10 s | 2.39 s | 1.77 s | 1.35x |
| `map<string,int>` | 1.18 s | 3.65 s | 2.40 s | 1.53x |
| `struct<i:int,s:string,d:double>` | 4.32 s | 6.03 s | 4.87 s | 1.24x |
Microbenchmarks of the fused chains (400k rows, byte-identical outputs): input map→dict 3.0x, input struct→Row 1.8x, output array 7.5x, output map 4.2x, output struct 4.0x.
The remaining input-side cost is PyArrow's per-element `to_pylist`, addressed upstream in apache/arrow#50326 / apache/arrow#50327.
### Does this PR introduce _any_ user-facing change?
No. Only performance; conversion results are identical (covered by exact-type tests and an arrow-vs-pickle end-to-end comparison with injected nulls).
### How was this patch tested?
New `ArrowToRowsColumnTests` compares `_to_rows_column` against the per-row converter chain with exact-type assertions across map (incl. nested values), struct (incl. nested map), array-of-struct/map, fallback types (timestamp/binary), sliced and chunked views, plus dedicated tests that struct rows are `Row` objects and map rows preserve Arrow entry order. Full `test_conversion.py` passes (43 tests, 200 subtests). End-to-end: arrow-vs-pickle `collect()` results verified identical for all nested benchmark cases with 10% injected nulls.
### Was this patch authored or co-authored using generative AI tooling?
Yes. This pull request and its description were written by Isaac (Claude Code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58050][PYTHON] Bulk-assemble Arrow Python UDF results [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57137:
URL: https://github.com/apache/spark/pull/57137#issuecomment-4920071728
Thanks for the honest feedback — I agree with the principle, and I've scoped this PR down accordingly.
On "this should be done in arrow": fully agreed, and that's where the real fix is happening — apache/arrow#50327 (reworked per Antoine's review) moves the scalar-free conversion into PyArrow itself, with apache/arrow#50429 covering the map→dict form Spark consumes. The Spark-side `_to_pylist` shims from SPARK-58019/58023/58024 are explicitly interim, each with a removal note for when the minimum PyArrow version includes the fix.
I've now **dropped the input-side converter fusion from this PR** — you're right that it stacked more machinery on the interim layer, and once the PyArrow fix ships it reduces to a few lines anyway, with no NumPy involved at all. Deferring it until then both answers the maintainability concern and makes it simpler.
What remains is the **output side only**, which I'd argue is a different category from "taking over arrow's work": it removes *Spark's own* per-row result converters (defensive list copies, dict→entry-list, `Row`→dict) that run before `pa.array`. Arrow can never do this for us — it doesn't know what a `Row` is — and this half uses no NumPy and doesn't depend on the PyArrow version. It's a permanent simplification of Spark's own hot loop, guarded so that anything unexpected falls back to the existing per-row path with identical error behavior.
On "we don't know what happens on other data types": we measured — scalar types are already *faster* on the arrow path than pickle (string 2.2x, date 2.4x, timestamp_ntz 2.9x, decimal 2.2x at 6.4M rows). The regression is specific to nested types, which is exactly what this line of work targets, and where the remaining gap closes when the PyArrow fix lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58050][PYTHON] Bulk-assemble Arrow Python UDF results [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #57137:
URL: https://github.com/apache/spark/pull/57137#issuecomment-4930590267
First of all, I believe the input part is not removed.
Then, I don't believe the output part is meaningful.
> Spark's own per-row result converters (defensive list copies, dict→entry-list, Row→dict) that run before pa.array. Arrow can never do this for us — it doesn't know what a Row is — and this half uses no NumPy and doesn't depend on the PyArrow version. It's a permanent simplification of Spark's own hot loop, guarded so that anything unexpected falls back to the existing per-row path with identical error behavior.
I don't believe this is correct because the row to arrow converter already deal with cases where the data can be directly converted. The converter would be `None` and no copy will be performed.
Could you name a case where the new mechanism works but the old one doesn't?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58050][PYTHON] Bulk-assemble Arrow Python UDF results [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #57137: [SPARK-58050][PYTHON] Bulk-assemble Arrow Python UDF results
URL: https://github.com/apache/spark/pull/57137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58050][PYTHON] Fuse per-row converters into bulk Arrow conversion for Arrow Python UDFs [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #57137:
URL: https://github.com/apache/spark/pull/57137#issuecomment-4919401038
Honestly I don't think this is worth it. The code is really not maintainable and I'm not even sure this will pass on all arrow/numpy version matrix that we support. The code is cheap to generate now with LLM, but the maintenance effort is just too high. We don't know if there will corner cases in type coercion with `numpy`. This is something that should be done in low level libraries like `arrow`, not `pyspark`. The perf improvement is observable on certain dataset, but the issue is not "solved" - there is still a gap between arrow vs pickle, even with this much code. More importantly, we don't know what happens on other data types, or even how much this matters to our users.
In general, I don't think taking over the work from arrow is a good idea. The benefit can't justify the cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58050][PYTHON] Bulk-assemble Arrow Python UDF results [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57137:
URL: https://github.com/apache/spark/pull/57137#issuecomment-4931247811
Closing this PR. After investigating your objections in depth, you are right on both counts, and my earlier reply to you was incorrect — I should have verified before arguing back. You called this correctly before we did.
What we found when we re-measured properly:
1. **For identity children, the existing code already avoids per-row work** — the converter is `None` and results go straight to `pa.array`, exactly as you said. My "defensive copy" claim was wrong for those cases.
2. **For string-bearing types, the per-row converter is `str()` coercion** — a semantic contract (e.g. a UDF returning `bytes` for a string field yields `"b'x'"` today), which a bulk path cannot safely skip: `pa.array` would silently decode the bytes instead, with no error to trigger a fallback.
3. **Consequently, with the identity-only guard this PR shipped with, the bulk paths never actually engage for realistic types.** A strategy probe on the live worker shows every benchmark case takes the fallback, which is equivalent to the existing code.
4. The end-to-end deltas I previously cited were measurement artifacts: stage-level instrumentation shows the difference spread uniformly across stages whose code is byte-identical (including the UDF invocation itself), so it cannot be attributed to this change. I withdraw those numbers.
Apologies for the review time this consumed. The corrective takeaway on our side: performance claims must come with verification that the intended fast path actually executes and that the gain is localized to it — output equivalence and a benchmark delta alone are not evidence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org