[jira] [Resolved] (SPARK-60173) Remove unnecessary type: ignore[union-attr] in pyspark.ml

["Ruifeng Zheng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruifeng Zheng resolved SPARK-60173.
-----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 59377
[https://github.com/apache/spark/pull/59377]

> Remove unnecessary type: ignore[union-attr] in pyspark.ml
> ---------------------------------------------------------
>
>                 Key: SPARK-60173
>                 URL: https://issues.apache.org/jira/browse/SPARK-60173
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML, PySpark
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Vito Lin
>            Assignee: Vito Lin
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> pyspark.ml module has five "type: ignore[union-attr]" comments that can be replaced with proper type narrowing or corrected annotations:
> 1. ml/torch/torch_run_process_wrapper.py: task.stdin and task.stdout are created with subprocess.PIPE so they are never None; narrow with an assert.
> 2. ml/connect/io_utils.py: narrow sparkContext._gateway with an assert.
> 3. ml/wrapper.py: check SparkContext._active_spark_context._gateway is not None before calling detach, instead of relying on the surrounding except.
> 4. ml/connect/tuning.py: param_map comes from Sequence[ParamMap], so annotate it as ParamMap instead of Any.
> 5. ml/connect/base.py: Estimator.fit only accepts a single ParamMap and never returns a list, but its return type is declared as Union[M, List[M]]. Correct it to M. This is what caused the ignore in tuning.py.
> No behavior change.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org