You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/09/11 03:30:18 UTC
[PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Yicong-Huang opened a new pull request, #58729:
URL: https://github.com/apache/spark/pull/58729
### What changes were proposed in this pull request?
This PR introduces an extensible, typed execution model for Arrow/Pandas UDF eval types in the PySpark worker, and migrates the first eval type (`SQL_SCALAR_ARROW_UDF`) onto it. It is the foundation for the umbrella refactor tracked in [SPARK-59415](https://issues.apache.org/jira/browse/SPARK-59415) (design: "1DD: Python UDF Eval Handler API Design", Design A).
Every Arrow/Pandas UDF eval type runs the same three-stage pipeline in the worker: prepare the input, invoke user code, then validate and normalize the result. Historically each eval type spelled this out inline in `read_udfs`, which has grown into a single ~2,500-line central `if/elif` dispatcher. This PR makes that lifecycle explicit:
- `EvalTypeHandler[InputBatch, OutputBatch]`: a generic interface with three abstract stages -- `pre_process` (argument extraction / conversions), `process` (user-code invocation only), `post_process` (schema enforcement, row-count checks) -- plus a `run(split_index, data)` runner that chains them lazily as generators. `run` matches the `func(split_index, data)` shape `read_udfs` already returns.
- Three typed category bases that fix the input-stream element type and the default serializer: `BatchEvalTypeHandler` (`Iterator[pa.RecordBatch]`, `ArrowStreamSerializer`), `GroupedEvalTypeHandler` (`Iterator[GroupedBatch]`, `ArrowStreamGroupSerializer`), and `CoGroupedEvalTypeHandler` (`Iterator[CoGroupedBatch]`, `ArrowStreamCoGroupSerializer`).
- Automatic registration: a concrete handler declares its `eval_type`, and `EvalTypeHandler.__init_subclass__` registers it in `_EVAL_TYPE_HANDLERS`. `read_udfs` looks the eval type up first and delegates to the handler, so a later eval type attaches as a self-contained subclass without editing any central branch. Duplicate registrations are rejected.
- `ArrowScalarUDFHandler` migrates `SQL_SCALAR_ARROW_UDF` as the first handler; its old inline branch and serializer-tuple entry in `read_udfs` are removed.
The remaining 28 Arrow/Pandas eval types are left on the existing `if/elif` path and will be migrated incrementally in follow-up PRs, each independently revertible.
Class hierarchy:
```
EvalTypeHandler[InputBatch, OutputBatch] (ABC: pre_process / process / post_process / run)
|- BatchEvalTypeHandler (Iterator[pa.RecordBatch])
| |- ArrowScalarUDFHandler (SQL_SCALAR_ARROW_UDF)
|- GroupedEvalTypeHandler (Iterator[GroupedBatch])
|- CoGroupedEvalTypeHandler (Iterator[CoGroupedBatch])
```
### Why are the changes needed?
`read_udfs` is a single central dispatcher whose implicit per-eval-type lifecycle is hard to read, extend, and test. Making the three stages explicit and letting eval types register themselves removes the central `if/elif` chain (for both function construction and serializer selection), keeps a strong per-category type contract, and lets each subsequent eval type be migrated and reverted on its own.
### Does this PR introduce _any_ user-facing change?
No. This is an internal worker refactor. There is no change to any user-facing UDF API, the on-the-wire format, or JVM-side code, and the migrated `SQL_SCALAR_ARROW_UDF` path is behavior-identical.
### How was this patch tested?
- New `pyspark.sql.tests.test_eval_type_handlers` unit suite (no JVM required): handler registration, category-base abstractness, non-registration of the abstract bases, per-category default serializers, `run` stage-chaining, duplicate-registration rejection, and end-to-end `ArrowScalarUDFHandler` output including schema coercion.
- Existing `pyspark.sql.tests.arrow.test_arrow_udf_scalar` (55 tests + 20 subtests) and `pyspark.sql.tests.arrow.test_arrow_udf_typehints` (15 tests) pass unchanged against a local `-Phive` build, confirming zero behavior change on the migrated path.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Spenserrrr (via GitHub)" <gi...@apache.org>.
Spenserrrr commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4022083086
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,129 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional, cast
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by _EvalTypeHandlerMeta at class definition.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
+
+
+class _EvalTypeHandlerMeta(ABCMeta):
Review Comment:
I'm wondering if we can pupulate this mapping explicitly instead of through the metaclass, given Tian suggests that we can keep this registry private. Right now, these handlers are interal to Spark, so there is no dynamic third-party discovery requirement. Also, each concrete handler module already has to be imported, so the metaclass is not doing discovery for us. I'm wondering if we can add a _registry.py file, and do something like:
```
from pyspark.sql.eval_handlers._arrow import ArrowScalarUDFHandler
_eval_type_handlers = {
ArrowScalarUDFHandler.eval_type: ArrowScalarUDFHandler,
}
def get_eval_type_handler(eval_type):
return _eval_type_handlers.get(eval_type)
```
and in the worker.py and we call `handler_cls = get_eval_type_handler(eval_type)`. This can make the supported eval type clearer. Do you see a case where class-time self-registration would help as more handlers are added?
##########
python/pyspark/worker.py:
##########
@@ -2053,6 +1881,18 @@ def _elementwise_result_to_arrow(result, return_type, arrow_element_type, is_pan
def read_udfs(pickleSer, udf_info_list, eval_type, runner_conf, eval_conf):
+ # Eval types that have been migrated to a handler are dispatched here without
+ # walking the if/elif chain below. The handler owns the whole lifecycle,
+ # including serializer selection.
+ handler_cls = EVAL_TYPE_HANDLERS.get(eval_type)
+ if handler_cls is not None:
+ udfs = [
+ read_single_udf(pickleSer, udf_info, eval_type, runner_conf, udf_index=udf_index)
Review Comment:
One suggestion for the new handler path. Do you think it is better to split the common UDF loading from the eval-type prepartation? Right now, read_single_udf does both common work such as deserialization and it also does eval-type-speicifc prepartion that returns different typle layouts.
The next line passes these values through the generic handler boundary as list[tuple[Any, ...]], so the meaning of each position is known only by read_single_udf and the concrete handler. For example, can we split this function into two:
```
loaded_udf = load_single_udf(...) # common work
prepared_udf = handler_cls.prepare_udf(loaded_udf) # eval-type-specific work
```
and the scalar handler can define the shape it needs, like:
```
class PreparedScalarUDF(NamedTuple):
func: Callable[..., Any]
args_offsets: list[int]
kwargs_offsets: dict[str, int]
return_type: DataType
```
##########
python/pyspark/sql/eval_handlers/_typing.py:
##########
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Type aliases and variables for the eval type handlers.
+
+The Arrow element types are forward refs so pyarrow stays a type-checking-only
+import at runtime, as elsewhere in the worker.
+"""
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, TypeVar
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+# Grouped-category input element, matching what the serializer yields: a group
+# serializer yields a lazy iterator; a co-group serializer materializes each side.
+GroupedBatch = Iterator["pa.RecordBatch"]
Review Comment:
Do you think Iterable[RecordBatch] is better here? The group serializer yields an iterator normally, but pipelined_process materializes each group into a list before passing it to run, so a future handler relying on iterator-only operations could fail only with pipelining enabled. A list-backed grouped case would cover that representation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4013925817
##########
python/pyspark/sql/eval_handlers/__init__.py:
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Handlers for the Arrow/Pandas UDF eval types.
+
+Each eval type is an ``EvalTypeHandler`` subclass that declares its ``eval_type``
+and self-registers in ``EVAL_TYPE_HANDLERS`` via ``__init_subclass__``; the
+worker's ``read_udfs`` dispatches on the registry. Base classes live in
+``_base``; concrete handlers live in private per-family submodules (``_arrow``),
+imported below so importing this package registers them.
+"""
+
+# Re-export the concrete handlers from their private submodules so callers reach
+# them from this package (or via EVAL_TYPE_HANDLERS), never from the ``_``
+# submodules. The import also registers them.
+from pyspark.sql.eval_handlers._arrow import ArrowScalarUDFHandler
+from pyspark.sql.eval_handlers._base import (
+ EVAL_TYPE_HANDLERS,
+ BatchEvalTypeHandler,
+ CoGroupedEvalTypeHandler,
+ EvalTypeHandler,
+ GroupedEvalTypeHandler,
+)
+
+__all__ = [
Review Comment:
This exports an implementor-facing API from a non-underscore package, but external subclasses cannot populate the executor-local registry before `read_udfs` consults it: user code is deserialized only afterward. Please keep the package and symbols internal for this refactor; a supported extension point would need an explicit driver-to-worker registration mechanism.
##########
python/pyspark/sql/eval_handlers/__init__.py:
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Handlers for the Arrow/Pandas UDF eval types.
+
+Each eval type is an ``EvalTypeHandler`` subclass that declares its ``eval_type``
Review Comment:
This should describe the key-to-handler relationship and limit the claim to the incrementally migrated eval types.
```suggestion
Each migrated eval type has an ``EvalTypeHandler`` subclass that declares its ``eval_type``
```
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,115 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by EvalTypeHandler.__init_subclass__.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
+
+
+class EvalTypeHandler(Generic[InputBatch, OutputBatch], metaclass=ABCMeta):
+ """Base class for the Arrow/Pandas UDF execution model.
+
+ A handler declares the ``serializer`` for the input/output streams and
+ implements ``run``, which consumes the input stream and yields the output
+ stream. Concrete handlers subclass a typed category base
+ (``BatchEvalTypeHandler``, ``GroupedEvalTypeHandler``,
+ ``CoGroupedEvalTypeHandler``), which fixes the input type and serializer.
+ """
+
+ # PythonEvalType this handler serves; None on the abstract bases.
+ eval_type: ClassVar[Optional[int]] = None
+
+ def __init_subclass__(cls, **kwargs: Any) -> None:
+ super().__init_subclass__(**kwargs)
+ # Register concrete handlers, i.e. those that declare an eval type.
+ eval_type = cls.__dict__.get("eval_type")
Review Comment:
This registers a class solely because it declares `eval_type`, even when `ABCMeta` still considers `run` abstract. `read_udfs` then fails when it instantiates that handler. Please reject abstract declarations here and add a focused case so an invalid handler fails during class definition instead of task startup.
##########
python/pyspark/sql/eval_handlers/_typing.py:
##########
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Type aliases and variables for the eval type handlers.
+
+The Arrow element types are forward refs so pyarrow stays a type-checking-only
+import at runtime, as elsewhere in the worker.
+"""
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, TypeVar
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+# Input stream element type for the grouped categories (the batch category's
+# element is a plain ``pa.RecordBatch``).
+GroupedBatch = Iterator["pa.RecordBatch"] # one group of batches
+CoGroupedBatch = tuple[Iterator["pa.RecordBatch"], Iterator["pa.RecordBatch"]] # a co-group pair
Review Comment:
`ArrowStreamCoGroupSerializer.load_stream` eagerly materializes both sides as lists, so this annotation gives future handlers the wrong contract. Please cover one deserialized co-group in the category tests as well.
```suggestion
CoGroupedBatch = tuple[list["pa.RecordBatch"], list["pa.RecordBatch"]]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4040511319
##########
python/pyspark/eval_handlers/_arrow.py:
##########
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Review Comment:
I don't have super strong opinion on this, but eval type handlers is not part of sql now? I mean our serializers are in sql. I agree that maybe eventually our udf can handle requests from everywhere, but for now, do we have any case where this runs outside of sql?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #58729: [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF
URL: https://github.com/apache/spark/pull/58729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4036722918
##########
python/pyspark/sql/eval_handlers/_arrow.py:
##########
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Handlers for the Arrow-native UDF eval types (the UDF exchanges ``pa.Array`` /
+``pa.RecordBatch`` values directly, without a pandas conversion)."""
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any
+
+from pyspark.sql.conversion import ArrowBatchTransformer
+from pyspark.sql.eval_handlers._base import BatchEvalTypeHandler
+from pyspark.sql.eval_handlers.verification import verify_scalar_result
+from pyspark.sql.pandas.types import to_arrow_schema
+from pyspark.sql.types import StructField, StructType
+from pyspark.util import PythonEvalType
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+
+class ArrowScalarUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
Review Comment:
nit: we have 'PythonUDFRunner' in the jvm side, and this class just provide a `run` method, not a 'handle'-like one, so shall we rename it `ArrowScalarUDFRunner`? otherwise LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4017060817
##########
python/pyspark/sql/eval_handlers/__init__.py:
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Handlers for the Arrow/Pandas UDF eval types.
+
+Each eval type is an ``EvalTypeHandler`` subclass that declares its ``eval_type``
+and self-registers in ``EVAL_TYPE_HANDLERS`` via ``__init_subclass__``; the
+worker's ``read_udfs`` dispatches on the registry. Base classes live in
+``_base``; concrete handlers live in private per-family submodules (``_arrow``),
+imported below so importing this package registers them.
+"""
+
+# Re-export the concrete handlers from their private submodules so callers reach
+# them from this package (or via EVAL_TYPE_HANDLERS), never from the ``_``
+# submodules. The import also registers them.
+from pyspark.sql.eval_handlers._arrow import ArrowScalarUDFHandler
+from pyspark.sql.eval_handlers._base import (
+ EVAL_TYPE_HANDLERS,
+ BatchEvalTypeHandler,
+ CoGroupedEvalTypeHandler,
+ EvalTypeHandler,
+ GroupedEvalTypeHandler,
+)
+
+__all__ = [
Review Comment:
Agreed, this isn't a real cross-process extension point yet. I removed `__all__` and marked the package internal in the docstring so it isn't advertised as a supported API, and consumers now import from the private submodules. A supported extension API can come later with a proper driver-to-worker registration mechanism.
##########
python/pyspark/sql/eval_handlers/__init__.py:
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Handlers for the Arrow/Pandas UDF eval types.
+
+Each eval type is an ``EvalTypeHandler`` subclass that declares its ``eval_type``
Review Comment:
Done. Updated the docstring to scope it to migrated eval types and spell out the eval-type-to-handler mapping.
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,115 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by EvalTypeHandler.__init_subclass__.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
+
+
+class EvalTypeHandler(Generic[InputBatch, OutputBatch], metaclass=ABCMeta):
+ """Base class for the Arrow/Pandas UDF execution model.
+
+ A handler declares the ``serializer`` for the input/output streams and
+ implements ``run``, which consumes the input stream and yields the output
+ stream. Concrete handlers subclass a typed category base
+ (``BatchEvalTypeHandler``, ``GroupedEvalTypeHandler``,
+ ``CoGroupedEvalTypeHandler``), which fixes the input type and serializer.
+ """
+
+ # PythonEvalType this handler serves; None on the abstract bases.
+ eval_type: ClassVar[Optional[int]] = None
+
+ def __init_subclass__(cls, **kwargs: Any) -> None:
+ super().__init_subclass__(**kwargs)
+ # Register concrete handlers, i.e. those that declare an eval type.
+ eval_type = cls.__dict__.get("eval_type")
Review Comment:
Good catch. Registration now happens in a metaclass that runs after `ABCMeta` computes `__abstractmethods__`, so a class declaring an `eval_type` while leaving `run`/`serializer` abstract is rejected at definition time. Added a focused test.
##########
python/pyspark/sql/eval_handlers/_typing.py:
##########
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Type aliases and variables for the eval type handlers.
+
+The Arrow element types are forward refs so pyarrow stays a type-checking-only
+import at runtime, as elsewhere in the worker.
+"""
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, TypeVar
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+# Input stream element type for the grouped categories (the batch category's
+# element is a plain ``pa.RecordBatch``).
+GroupedBatch = Iterator["pa.RecordBatch"] # one group of batches
+CoGroupedBatch = tuple[Iterator["pa.RecordBatch"], Iterator["pa.RecordBatch"]] # a co-group pair
Review Comment:
Right, the co-group serializer materializes each side as a list. Fixed the alias to `tuple[list[...], list[...]]` and added a category test that deserializes one co-group and asserts both sides are lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58729:
URL: https://github.com/apache/spark/pull/58729#issuecomment-5685326518
thanks @zhengruifeng @HyukjinKwon, I've resolved comments and the PR is ready for another pass. also @gaogaotiantian please take a look if you have time, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4020537209
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,129 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional, cast
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by _EvalTypeHandlerMeta at class definition.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
+
+
+class _EvalTypeHandlerMeta(ABCMeta):
+ """Registers a concrete handler under its ``eval_type`` at definition time.
+
+ Runs after ``ABCMeta`` sets ``__abstractmethods__``, so a class that declares
+ an ``eval_type`` while leaving ``run``/``serializer`` abstract is rejected here
+ instead of failing when ``read_udfs`` instantiates it.
+ """
+
+ def __new__(mcs, name: str, bases: tuple, namespace: dict, **kwargs: Any) -> type:
+ cls = cast("type[EvalTypeHandler]", super().__new__(mcs, name, bases, namespace, **kwargs))
Review Comment:
We can probably do a runtime check here like `assert issubclass(cls, EvalTypeHandler)` then mypy should be happy.
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,129 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional, cast
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by _EvalTypeHandlerMeta at class definition.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
Review Comment:
All cap normally means "constant" in Python. This should be `_eval_type_handlers`. Let's expose a function in this module to get the handler, instead of directly using this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4040567427
##########
python/pyspark/eval_handlers/_arrow.py:
##########
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Review Comment:
Currently no other invokers. I think we don't have to limit it by saying only sql will invoke eval types so eval types are under `sql/`. Serializers also would be relocated accordingly in my mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4041117713
##########
python/pyspark/eval_handlers/_arrow.py:
##########
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Review Comment:
I don't think we need to limit it to sql, but I also think it's not a horrible idea to keep this inside sql when we only have use cases there. On the other hand, `worker.py` is at top level so maybe it's not too bad. I'll leave the decision to you, I don't have a super strong opinion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58729:
URL: https://github.com/apache/spark/pull/58729#issuecomment-5720769418
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/4a63c76ba1a01e21387337dd8bd24b8e44a6780d
- merged into branch-4.x https://github.com/apache/spark/commit/7d975394473f3534e911ca5633b16205d7056978
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58729:
URL: https://github.com/apache/spark/pull/58729#issuecomment-5675235666
Thanks for the review, Hyukjin!
1. Import cycle: good catch. I moved the base classes into a `_base.py` leaf module that both `__init__` and `_arrow` import, so the cycle is gone now.
2. `assert`: I actually started with `raise AssertionError`, but the `check_pyspark_custom_errors` linter rejects raising builtins, and adding a dedicated error condition felt like overkill for an internal dev-time invariant, so I kept `assert` to match the worker's style. Happy to switch to a custom error if you'd rather it survive `-O`.
cc @zhengruifeng @gaogaotiantian, would appreciate your eyes on this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4022916029
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,129 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional, cast
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by _EvalTypeHandlerMeta at class definition.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
+
+
+class _EvalTypeHandlerMeta(ABCMeta):
+ """Registers a concrete handler under its ``eval_type`` at definition time.
+
+ Runs after ``ABCMeta`` sets ``__abstractmethods__``, so a class that declares
+ an ``eval_type`` while leaving ``run``/``serializer`` abstract is rejected here
+ instead of failing when ``read_udfs`` instantiates it.
+ """
+
+ def __new__(mcs, name: str, bases: tuple, namespace: dict, **kwargs: Any) -> type:
+ cls = cast("type[EvalTypeHandler]", super().__new__(mcs, name, bases, namespace, **kwargs))
Review Comment:
Done, replaced the cast with `assert issubclass(cls, EvalTypeHandler)`. It narrows the type for mypy and doubles as a guard.
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,129 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional, cast
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by _EvalTypeHandlerMeta at class definition.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
Review Comment:
Good point. Renamed it to `_eval_type_handlers` and added `get_eval_type_handler(eval_type)`; `read_udfs` now goes through the function instead of touching the module variable.
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,129 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional, cast
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by _EvalTypeHandlerMeta at class definition.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
+
+
+class _EvalTypeHandlerMeta(ABCMeta):
Review Comment:
I'd like to keep the class-time self-registration. It keeps adding an eval type to a single self-contained subclass, with no separate registry file to update in lockstep, which is the main maintenance win as more handlers land. The registry is now private with a `get_eval_type_handler` accessor (per Tian's comment), so it isn't a public discovery API. Happy to revisit if an explicit table turns out clearer later.
##########
python/pyspark/worker.py:
##########
@@ -2053,6 +1881,18 @@ def _elementwise_result_to_arrow(result, return_type, arrow_element_type, is_pan
def read_udfs(pickleSer, udf_info_list, eval_type, runner_conf, eval_conf):
+ # Eval types that have been migrated to a handler are dispatched here without
+ # walking the if/elif chain below. The handler owns the whole lifecycle,
+ # including serializer selection.
+ handler_cls = EVAL_TYPE_HANDLERS.get(eval_type)
+ if handler_cls is not None:
+ udfs = [
+ read_single_udf(pickleSer, udf_info, eval_type, runner_conf, udf_index=udf_index)
Review Comment:
I like this, but it changes `read_single_udf`, which is shared by every eval type, so I'd rather do it as its own change once more handlers are migrated and the per-handler prepared shapes are clear. Keeping the `list[tuple[...]]` boundary for now; let's discuss it separately.
##########
python/pyspark/sql/eval_handlers/_typing.py:
##########
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Type aliases and variables for the eval type handlers.
+
+The Arrow element types are forward refs so pyarrow stays a type-checking-only
+import at runtime, as elsewhere in the worker.
+"""
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, TypeVar
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+# Grouped-category input element, matching what the serializer yields: a group
+# serializer yields a lazy iterator; a co-group serializer materializes each side.
+GroupedBatch = Iterator["pa.RecordBatch"]
Review Comment:
Keeping `Iterator` here intentionally. The group serializer yields a lazy iterator, and grouped funcs consume it via `for batch in group` / `list(group)`, so the migrated handlers won't rely on iterator-only ops. The intended contract is a real iterator, so I'd rather keep the annotation as `Iterator` and tighten the pipelined path toward it than widen to `Iterable`. (CoGroupedBatch stays a list pair since its serializer materializes both sides.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Spenserrrr (via GitHub)" <gi...@apache.org>.
Spenserrrr commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4023085804
##########
python/pyspark/sql/eval_handlers/_base.py:
##########
@@ -0,0 +1,129 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Base classes and registry for the eval type handlers.
+
+Kept in a leaf module (importing only ``_typing`` and serializers) so both the
+package ``__init__`` and the concrete-handler submodules can import it without a
+circular import.
+"""
+
+from abc import ABCMeta, abstractmethod
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional, cast
+
+from pyspark.serializers import Serializer
+from pyspark.sql.eval_handlers._typing import (
+ CoGroupedBatch,
+ GroupedBatch,
+ InputBatch,
+ OutputBatch,
+)
+from pyspark.sql.pandas.serializers import (
+ ArrowStreamCoGroupSerializer,
+ ArrowStreamGroupSerializer,
+ ArrowStreamSerializer,
+)
+
+if TYPE_CHECKING:
+ import pyarrow as pa # noqa: F401 # only in the batch category's forward-ref subscript
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+# eval type -> handler class, populated by _EvalTypeHandlerMeta at class definition.
+EVAL_TYPE_HANDLERS: "dict[int, type[EvalTypeHandler]]" = {}
+
+
+class _EvalTypeHandlerMeta(ABCMeta):
Review Comment:
That makes sense. Keeping the eval type and its registration in the same subclass avoids a second table to maintain, and the private getter addresses the registry exposure. Thanks for explaining.
##########
python/pyspark/worker.py:
##########
@@ -2053,6 +1881,18 @@ def _elementwise_result_to_arrow(result, return_type, arrow_element_type, is_pan
def read_udfs(pickleSer, udf_info_list, eval_type, runner_conf, eval_conf):
+ # Eval types that have been migrated to a handler are dispatched here without
+ # walking the if/elif chain below. The handler owns the whole lifecycle,
+ # including serializer selection.
+ handler_cls = EVAL_TYPE_HANDLERS.get(eval_type)
+ if handler_cls is not None:
+ udfs = [
+ read_single_udf(pickleSer, udf_info, eval_type, runner_conf, udf_index=udf_index)
Review Comment:
Makes sense. Keeping this PR focused is reasonable. We can revisit the preparation boundary under SPARK-59415 once another handler family makes the required shapes clearer.
##########
python/pyspark/sql/eval_handlers/_typing.py:
##########
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Type aliases and variables for the eval type handlers.
+
+The Arrow element types are forward refs so pyarrow stays a type-checking-only
+import at runtime, as elsewhere in the worker.
+"""
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, TypeVar
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+# Grouped-category input element, matching what the serializer yields: a group
+# serializer yields a lazy iterator; a co-group serializer materializes each side.
+GroupedBatch = Iterator["pa.RecordBatch"]
Review Comment:
That makes sense. Keeping `Iterator` as the intended contract and tightening the pipelined path separately sounds reasonable. Thanks for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Spenserrrr (via GitHub)" <gi...@apache.org>.
Spenserrrr commented on PR #58729:
URL: https://github.com/apache/spark/pull/58729#issuecomment-5704053737
LGTM, thanks for addressing the feedback and explaining the design choices!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58729:
URL: https://github.com/apache/spark/pull/58729#issuecomment-5702477805
Thanks @zhengruifeng @gaogaotiantian @Spenserrrr, please kindly have another pass!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4039039054
##########
python/pyspark/eval_handlers/_arrow.py:
##########
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Handlers for the Arrow-native UDF eval types (the UDF exchanges ``pa.Array`` /
+``pa.RecordBatch`` values directly, without a pandas conversion)."""
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING, Any
+
+from pyspark.sql.conversion import ArrowBatchTransformer
+from pyspark.sql.eval_handlers._base import BatchEvalTypeHandler
+from pyspark.sql.eval_handlers.verification import verify_scalar_result
+from pyspark.sql.pandas.types import to_arrow_schema
+from pyspark.sql.types import StructField, StructType
+from pyspark.util import PythonEvalType
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+ from pyspark.worker_util import EvalConf, RunnerConf
+
+
+class ArrowScalarUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
Review Comment:
Thanks! I'd rather keep `Handler` here. Semantically the whole worker is the runner, so naming one eval-type class a `Runner` would blur that, which I'd like to avoid. Keeping `EvalTypeHandler` for the abstraction.
##########
python/pyspark/eval_handlers/_arrow.py:
##########
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Review Comment:
Good point on the location. There's no `pyspark/udf` package today (the UDF Python code lives under `pyspark/sql`), but I agree it doesn't belong under `sql`. I moved it out to top-level `pyspark/eval_handlers`, next to `worker.py`, so it can sit alongside future UDF-related packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #58729:
URL: https://github.com/apache/spark/pull/58729#discussion_r4036743927
##########
python/pyspark/sql/eval_handlers/_arrow.py:
##########
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Review Comment:
another nit: we have udf maven module in jvm side, why the new code be placed in `pyspark/sql` than `pyspark/udf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org