You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/08/04 23:19:58 UTC
[PR] [SPARK-58577][PYTHON] Inline wrap_udf into the batched UDF dispatch branch in worker.py [spark]
Yicong-Huang opened a new pull request, #57775:
URL: https://github.com/apache/spark/pull/57775
### What changes were proposed in this pull request?
This PR inlines the `wrap_udf` helper into its single caller, the `SQL_BATCHED_UDF` branch of `read_single_udf` in `python/pyspark/worker.py`, and deletes the `wrap_udf` definition.
`wrap_udf` was a thin wrapper doing two things: calling `wrap_kwargs_support`, then conditionally wrapping the result with `return_type.toInternal` based on `needConversion()`. It had exactly one caller. Both of these steps are already open-coded inline in their dispatch branches elsewhere in `worker.py` -- `wrap_kwargs_support` has 8 call sites and 7 of them call it directly inside their branch rather than through a wrapper, so "inline in branch" is the prevailing idiom and `wrap_udf` was the lone outlier. Inlining it matches the style of the neighboring grouped-map branches. A short comment is added above the branch to preserve the semantic label that the function name carried.
### Why are the changes needed?
`wrap_udf` is a thin single-caller wrapper that diverges from the established pattern in `read_single_udf`, where each eval type assembles its return value inline in its own branch. Removing the indirection makes the dispatch branch self-contained and consistent with its neighbors.
### Does this PR introduce _any_ user-facing change?
No. This is a pure internal refactor with exact behavioral equivalence: the lambda closure capture of `func`/`toInternal` and the `needConversion()` vs else split are preserved unchanged.
### How was this patch tested?
Existing `pyspark.sql.tests.test_udf UDFTests` suite passes. Additionally verified end-to-end that plain Python UDFs still behave correctly across both the `needConversion()` path (e.g. `DateType`, `DecimalType`) and the non-conversion path (e.g. `IntegerType`, `StringType`), as well as the keyword-argument path through `wrap_kwargs_support`.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58577][PYTHON] Inline wrap_udf into the batched UDF dispatch branch in worker.py [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #57775:
URL: https://github.com/apache/spark/pull/57775#issuecomment-5188555954
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/94ecd044ac05d2de513ebf74b1a135c647df7b26
- merged into branch-4.x https://github.com/apache/spark/commit/34822d0c75f1ac67a014662bf27b4a976d97e2d4
- merged into branch-4.3 https://github.com/apache/spark/commit/85565f37fcd620df0c705ed7e1d0b531309906c2
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58577][PYTHON] Inline wrap_udf into the batched UDF dispatch branch in worker.py [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #57775: [SPARK-58577][PYTHON] Inline wrap_udf into the batched UDF dispatch branch in worker.py
URL: https://github.com/apache/spark/pull/57775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org