You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "RogelioKG (via GitHub)" <gi...@apache.org> on 2026/10/11 07:10:01 UTC
[PR] [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] in pyspark.sql [spark]
RogelioKG opened a new pull request, #59364:
URL: https://github.com/apache/spark/pull/59364
### What changes were proposed in this pull request?
Add type annotations to four internal functions in `pyspark.sql` and remove the corresponding `# type: ignore[no-untyped-def]` comments:
| Function | Annotation | Based on |
|---|---|---|
| `check_string_field` in `pyspark/sql/functions/builtin.py` (`window`) | `(field: Any, fieldName: str) -> None` | It validates arbitrary user input at runtime, so `field` is intentionally `Any` |
| `infer_eval_type_for_udf` in `pyspark/sql/pandas/typehints.py` | `f: Callable[..., Any]` | Both callers (`_create_py_udf` in classic and Connect) declare `f: Callable[..., Any]` |
| `process` in `pyspark/sql/connect/streaming/worker/foreach_batch_worker.py` | `(df_id: str, batch_id: int, cloned_session_id: str) -> None` | Arguments come from `utf8_deserializer.loads(...)` and `int(read_long(...))`; the JVM side writes them with `PythonWorkerUtils.writeUTF` |
| `process` in `pyspark/sql/connect/streaming/worker/listener_worker.py` | `(listener_event_str: str, listener_event_type: int) -> None` | Arguments come from `utf8_deserializer.loads(...)` and `int(read_int(...))`; the JVM side writes the event JSON with `PythonWorkerUtils.writeUTF` |
After this change there is no `type: ignore[no-untyped-def]` left in `pyspark.sql`.
### Why are the changes needed?
Polish the type annotations for PySpark. With the suppressions in place, mypy skipped the bodies of these functions entirely; now they are type-checked.
### Does this PR introduce _any_ user-facing change?
No. Only type annotations of internal functions are changed.
### How was this patch tested?
- `dev/lint-python --mypy` equivalent (`mypy --namespace-packages --config-file python/mypy.ini python/pyspark`): the output is identical before and after this change, and there are no errors in the four modified files.
- `ruff check` and `ruff format --check` pass on the modified files.
- The following tests pass locally:
- `python/run-tests --testnames 'pyspark.sql.functions.builtin'` (doctests, including `window`)
- `python/run-tests --testnames 'pyspark.sql.tests.test_functions FunctionsTests.test_window_time'`
- `python/run-tests --testnames 'pyspark.sql.tests.test_unified_udf'` (exercises `infer_eval_type_for_udf` via `@udf`)
- `python/run-tests --testnames 'pyspark.sql.tests.connect.streaming.test_parity_foreach_batch'`
- `python/run-tests --testnames 'pyspark.sql.tests.connect.streaming.test_parity_listener'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] in pyspark.sql [spark]
Posted by "gaogaotiantian (via GitHub) via reviews" <re...@spark.apache.org>.
gaogaotiantian commented on PR #59364:
URL: https://github.com/apache/spark/pull/59364#issuecomment-6591936936
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/676e64ba0eeb400173d70889ddad92bf0ee5d420
- merged into branch-4.x https://github.com/apache/spark/commit/0ff392a0ed23cd570942e219b25d78e1ceb92a6b
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] in pyspark.sql [spark]
Posted by "gaogaotiantian (via GitHub) via reviews" <re...@spark.apache.org>.
gaogaotiantian closed pull request #59364: [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] in pyspark.sql
URL: https://github.com/apache/spark/pull/59364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] in pyspark.sql [spark]
Posted by "gaogaotiantian (via GitHub) via reviews" <re...@spark.apache.org>.
gaogaotiantian commented on PR #59364:
URL: https://github.com/apache/spark/pull/59364#issuecomment-6591951074
Thank you! Congrats to your first commit to Spark!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org