[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py

[GitBox <gi...@apache.org>]

dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r745237218



##########
File path: python/pyspark/sql/conf.py
##########
@@ -39,7 +40,7 @@ def set(self, key: str, value: str) -> None:
         self._jconf.set(key, value)
 
     @since(2.0)
-    def get(self, key: str, default: Optional[str] = _NoValue) -> str:
+    def get(self, key: str, default: Union[Optional[str], _NoValueType] = _NoValue) -> str:

Review comment:
       files in `pyspark/sql` need to be changed because mypy check type of `_NoValue` after we inline type hint for `pyspark/__init__.py`

##########
File path: python/pyspark/__init__.py
##########
@@ -63,41 +64,54 @@
 from pyspark.version import __version__
 from pyspark._globals import _NoValue  # noqa: F401
 
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
 
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
     """
     A decorator that annotates a function to append the version of Spark the function was added.
     """
     import re
     indent_p = re.compile(r'\n( +)')
 
-    def deco(f):
-        indents = indent_p.findall(f.__doc__)
+    def deco(f: T) -> T:
+        indents = indent_p.findall(cast(str, f.__doc__))
         indent = ' ' * (min(len(m) for m in indents) if indents else 0)
-        f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+        f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
         return f
     return deco
 
 
-def copy_func(f, name=None, sinceversion=None, doc=None):
+def copy_func(
+    f: F,
+    name: Optional[str] = None,
+    sinceversion: Optional[Union[str, float]] = None,
+    doc: Optional[str] = None,
+) -> F:
     """
     Returns a function with same code, globals, defaults, closure, and
     name (or provide a new name).
     """
     # See
     # http://stackoverflow.com/questions/6527633/how-can-i-make-a-deepcopy-of-a-function-in-python
-    fn = types.FunctionType(f.__code__, f.__globals__, name or f.__name__, f.__defaults__,
-                            f.__closure__)
+    fn = types.FunctionType(
+        f.__code__,
+        f.__globals__,  # type: ignore[attr-defined]
+        name or f.__name__,
+        f.__defaults__,  # type: ignore[attr-defined]
+        f.__closure__  # type: ignore[attr-defined]

Review comment:
       mypy doesn't support ```__defaults__```, ```__globals__``` and ```__closure__```
   https://github.com/python/mypy/issues/1923




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org