You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/04/02 22:33:15 UTC
[PR] [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py [spark]
Yicong-Huang opened a new pull request, #55178:
URL: https://github.com/apache/spark/pull/55178
### What changes were proposed in this pull request?
Tighten the type hints for `def func` signatures in `read_udfs()` for eval types that have been refactored to be self-contained. Specifically:
- Replace `Iterator[Any]` with `Iterator["GroupedBatch"]` for grouped eval types (`SQL_GROUPED_AGG_ARROW_UDF`, `SQL_GROUPED_AGG_ARROW_ITER_UDF`, `SQL_WINDOW_AGG_ARROW_UDF`)
- Rename the `batches` parameter to `data` across all refactored eval types for consistency, since the same call site passes either a flat stream or grouped stream
- Use `for batch in data` for flat streams and `for group in data` for grouped streams
- Define `GroupedBatch` and `CoGroupedBatch` type aliases in `pyspark.sql.pandas._typing`
### Why are the changes needed?
After refactoring eval types to be self-contained in `read_udfs()`, the `def func` signatures used `Iterator[Any]` for grouped eval types, losing type information. Tightening these hints improves readability and enables better static analysis.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55178:
URL: https://github.com/apache/spark/pull/55178#issuecomment-4194796565
> Do we need to change `func` to `group_func`? In my imagination eventually they will be the methods with the same name of corresponding eval classes. I don't think we need to use a different name for the function at this point - it is kind of an anonymous function I think.
yeah it is needed, this is a compromise to satisfy mypy. in the future when we move those code out of read_udf and build classes, we can rename them a gain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #55178: [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py
URL: https://github.com/apache/spark/pull/55178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55178:
URL: https://github.com/apache/spark/pull/55178#issuecomment-4186422069
cc @gaogaotiantian @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #55178:
URL: https://github.com/apache/spark/pull/55178#issuecomment-4194177431
Do we need to change `func` to `group_func`? In my imagination eventually they will be the methods with the same name of corresponding eval classes. I don't think we need to use a different name for the function at this point - it is kind of an anonymous function I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #55178:
URL: https://github.com/apache/spark/pull/55178#issuecomment-4195405867
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org