[jira] [Created] (SPARK-33002) Post-port removal of non-API stubs

["Maciej Szymkiewicz (Jira)" <ji...@apache.org>]

Maciej Szymkiewicz created SPARK-33002:
------------------------------------------

             Summary: Post-port removal of non-API stubs
                 Key: SPARK-33002
                 URL: https://issues.apache.org/jira/browse/SPARK-33002
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 3.1.0
            Reporter: Maciej Szymkiewicz


To simplify initial port we merge all existing stubs.

However, some of these cover non-API components and are usually dynamically annotated (generated with stubgen).

This includes modules like {{serializers}}, {{utils}}, {{shell}}, {{worker}}, etc.

These can be safely removed as:

- MyPy can infer types from the source, where stub is not present.
- No longer provide value, when corresponding modules are present in the same directory structure.
- Annotations are here primarily to help end users, not Spark developers and many of the annotations cannot be meaningfully refined.

It should also reduce overhead of maintaining annotations (especially when places where we don't guarantee signature stability).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org