You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/10/13 01:13:50 UTC
[PR] [SPARK-60209][PYTHON] Migrate SQL_ARROW_TABLE_UDF to the EvalTypeHandler pipeline [spark]
Yicong-Huang opened a new pull request, #59418:
URL: https://github.com/apache/spark/pull/59418
### What changes were proposed in this pull request?
Add variant registration to the eval type handler registry, then move `SQL_ARROW_TABLE_UDF` onto it as two handlers.
- Registry (`_base.py`): a handler may declare `variant_conf` (a `RunnerConf` property name) and `variant_value`. The registry maps `eval_type -> (variant_conf, {variant_value: handler})`; handlers without `variant_conf` register under `None`, so existing handlers are unchanged.
- `get_eval_type_handler(eval_type, runner_conf)` picks the variant per lookup, since the conf is per task and workers are reused across tasks. A conf value with no registered variant raises.
- Registration rejects a duplicate value and handlers of one eval type keyed on different confs.
- `_udtf.py`: `ArrowTableUDTFHandler` base (per-row `eval`/`terminate` and error wrapping), with two variants keyed on `use_legacy_pandas_udtf_conversion`:
- `ArrowTableUDTFLegacyPandasHandler` (`True`): the legacy pandas path.
- `ArrowTableUDTFArrowHandler` (`False`): the Arrow path.
- Move `verify_arrow_result`, used only by the Arrow path, from `worker.py` to `eval_handlers/verification.py`.
- `read_udtf` keeps only `SQL_TABLE_UDF`; its body is dedented (review with `?w=1`).
```
UDTFEvalTypeHandler
├─ ArrowUDTFHandler SQL_ARROW_UDTF
└─ ArrowTableUDTFHandler
├─ ArrowTableUDTFLegacyPandasHandler SQL_ARROW_TABLE_UDF, legacy pandas conversion = True
└─ ArrowTableUDTFArrowHandler SQL_ARROW_TABLE_UDF, legacy pandas conversion = False
```
### Why are the changes needed?
Step toward running UDTF and UDF eval types through one handler dispatch path and removing `read_udtf` (SPARK-59415). `SQL_ARROW_BATCHED_UDF` has the same legacy-conversion split and can reuse the variant registration later.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New registry tests in `test_base_eval_type_handlers` and handler tests for both variants in `test_udtf_eval_type_handlers`; existing `test_udtf` (incl. `LegacyUDTFArrowTests` and `UDTFArrowTests`), `test_arrow_udtf` and `test_parity_udtf` pass. ASV `ArrowTableUDFTimeBench`, master vs this PR (`repeat=(5,10,10.0)`, one representative run; a second run agreed):
```text
scenario udtf arrow legacy_pandas
sm_batch_few_col identity_udtf 69.8+-0.4ms -> 70.8+-0.3ms 293+-7ms -> 289+-1ms
sm_batch_many_col stringify_udtf 24.5+-0.2ms -> 24.7+-0.1ms 76.1+-0.6ms -> 76.2+-0.3ms
lg_batch_few_col filter_udtf 135+-1ms -> 137+-0.7ms 817+-9ms -> 816+-10ms
lg_batch_many_col explode_udtf 98.2+-0.3ms -> 98.6+-0.4ms 293+-3ms -> 296+-4ms
pure_ints identity_udtf 177+-0.9ms -> 180+-2ms 716+-3ms -> 719+-5ms
pure_strings stringify_udtf 187+-1ms -> 187+-1ms 737+-6ms -> 731+-4ms
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60209][PYTHON] Migrate SQL_ARROW_TABLE_UDF to the EvalTypeHandler pipeline [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #59418:
URL: https://github.com/apache/spark/pull/59418#issuecomment-6148081779
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/7d92cce351b2b3b15a746344bff3c6240304af87
- merged into branch-4.x https://github.com/apache/spark/commit/68356de63a0a6171b444decdeb103ee6a2979ff3
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60209][PYTHON] Migrate SQL_ARROW_TABLE_UDF to the EvalTypeHandler pipeline [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #59418: [SPARK-60209][PYTHON] Migrate SQL_ARROW_TABLE_UDF to the EvalTypeHandler pipeline
URL: https://github.com/apache/spark/pull/59418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org