You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/29 06:12:16 UTC
[GitHub] [spark] dchvn opened a new pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
dchvn opened a new pull request #34433:
URL: https://github.com/apache/spark/pull/34433
### What changes were proposed in this pull request?
Inline type hints for python/pyspark/\_\_init\_\_.py
### Why are the changes needed?
We can take advantage of static type checking within the functions by inlining the type hints.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993167036
**[Test build #146165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146165/testReport)** for PR 34433 at commit [`8d3be75`](https://github.com/apache/spark/commit/8d3be758efd4996737db297ee4f1742ed99488cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993193980
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50638/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954488737
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144738/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r768054438
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2530,7 +2531,9 @@ def replace( # type: ignore[misc]
to_replace: Union[
"LiteralType", List["LiteralType"], Dict["LiteralType", "OptionalPrimitiveType"]
],
- value: Optional[Union["OptionalPrimitiveType", List["OptionalPrimitiveType"]]] = _NoValue,
+ value: Optional[
+ Union["OptionalPrimitiveType", List["OptionalPrimitiveType"], _NoValueType]
+ ] = _NoValue,
Review comment:
OK, I gave it another look and any changes to `pyspark.sql` seem to unnecessary for the core task (in-lining of `pyspark/__init__.py`) ‒ the whole type checking pipeline passes just fine when omit `pyspark.sql` changes.
So let's resolve the conflicts, revert whatever is not strictly necessary, double check and get this merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969692863
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49729/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954482857
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49207/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-962729288
ping @HyukjinKwon and @zero323 again! Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966050120
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145097/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966050120
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145097/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993180399
**[Test build #146165 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146165/testReport)** for PR 34433 at commit [`8d3be75`](https://github.com/apache/spark/commit/8d3be758efd4996737db297ee4f1742ed99488cd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class TimedeltaOps(DataTypeOps):`
* `class TimedeltaIndex(Index):`
* `class MissingPandasLikeTimedeltaIndex(MissingPandasLikeIndex):`
* `class PandasSQLStringFormatter(string.Formatter):`
* `class UDFBasicProfiler(BasicProfiler):`
* `class CloudPickleSerializer(FramedSerializer):`
* `class SQLStringFormatter(string.Formatter):`
* `class DayTimeIntervalType(AtomicType):`
* `class DayTimeIntervalTypeConverter(object):`
* `class ExecutorPodsPollingSnapshotSource(`
* `class ExecutorPodsWatchSnapshotSource(`
* `class ExecutorRollPlugin extends SparkPlugin `
* `class ExecutorRollDriverPlugin extends DriverPlugin with Logging `
* ` class AnsiCombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends`
* `case class RelationTimeTravel(`
* `case class AsOfTimestamp(timestamp: Long) extends TimeTravelSpec`
* `case class AsOfVersion(version: String) extends TimeTravelSpec`
* ` class CombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends TypeCoercionRule `
* `case class PrettyPythonUDF(`
* `case class MapContainsKey(`
* `case class TryElementAt(left: Expression, right: Expression, child: Expression)`
* `case class ConvertTimezone(`
* `case class AesEncrypt(`
* `case class AesDecrypt(`
* `case class UnclosedCommentProcessor(`
* `case class CreateTable(`
* `case class TableSpec(`
* `case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements)`
* ` // When this is enabled, this class does additional lookup on write operations (put/delete) to`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966249692
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r747255941
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,55 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
+ indents = indent_p.findall(cast(str, f.__doc__))
indent = " " * (min(len(m) for m in indents) if indents else 0)
- f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+ f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
return f
return deco
-def copy_func(f, name=None, sinceversion=None, doc=None):
+def copy_func(
+ f: F,
+ name: Optional[str] = None,
+ sinceversion: Optional[Union[str, float]] = None,
+ doc: Optional[str] = None,
+) -> F:
"""
Returns a function with same code, globals, defaults, closure, and
name (or provide a new name).
"""
# See
# http://stackoverflow.com/questions/6527633/how-can-i-make-a-deepcopy-of-a-function-in-python
fn = types.FunctionType(
- f.__code__, f.__globals__, name or f.__name__, f.__defaults__, f.__closure__
+ f.__code__,
+ f.__globals__, # type: ignore[attr-defined]
Review comment:
mypy doesn't support ```__defaults__```, ```__globals__``` and ```__closure__```
https://github.com/python/mypy/issues/1923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r747417666
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
+ indents = indent_p.findall(cast(str, f.__doc__))
indent = " " * (min(len(m) for m in indents) if indents else 0)
- f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+ f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
Review comment:
Thank you! update
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
+ indents = indent_p.findall(cast(str, f.__doc__))
indent = " " * (min(len(m) for m in indents) if indents else 0)
- f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+ f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
Review comment:
Thank you! updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969620591
**[Test build #145259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145259/testReport)** for PR 34433 at commit [`3684fc4`](https://github.com/apache/spark/commit/3684fc43a2bed3cb74672729bf5a6264fabd55c8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969698393
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145259/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r748950832
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2530,7 +2531,9 @@ def replace( # type: ignore[misc]
to_replace: Union[
"LiteralType", List["LiteralType"], Dict["LiteralType", "OptionalPrimitiveType"]
],
- value: Optional[Union["OptionalPrimitiveType", List["OptionalPrimitiveType"]]] = _NoValue,
+ value: Optional[
+ Union["OptionalPrimitiveType", List["OptionalPrimitiveType"], _NoValueType]
+ ] = _NoValue,
Review comment:
Should I change these with `ignore`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969672838
**[Test build #145259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145259/testReport)** for PR 34433 at commit [`3684fc4`](https://github.com/apache/spark/commit/3684fc43a2bed3cb74672729bf5a6264fabd55c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966075658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145099/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966330370
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966069953
**[Test build #145099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145099/testReport)** for PR 34433 at commit [`b293718`](https://github.com/apache/spark/commit/b2937185bc248c4806fc6bcc7366033ec304a547).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954459604
**[Test build #144738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144738/testReport)** for PR 34433 at commit [`233b5ac`](https://github.com/apache/spark/commit/233b5ac02ec6d6e929131e713089cc5a3066aff5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r753875232
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2530,7 +2531,9 @@ def replace( # type: ignore[misc]
to_replace: Union[
"LiteralType", List["LiteralType"], Dict["LiteralType", "OptionalPrimitiveType"]
],
- value: Optional[Union["OptionalPrimitiveType", List["OptionalPrimitiveType"]]] = _NoValue,
+ value: Optional[
+ Union["OptionalPrimitiveType", List["OptionalPrimitiveType"], _NoValueType]
+ ] = _NoValue,
Review comment:
@ueshin I meant `_NoValueType`, but I honestly don't know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r745237218
##########
File path: python/pyspark/sql/conf.py
##########
@@ -39,7 +40,7 @@ def set(self, key: str, value: str) -> None:
self._jconf.set(key, value)
@since(2.0)
- def get(self, key: str, default: Optional[str] = _NoValue) -> str:
+ def get(self, key: str, default: Union[Optional[str], _NoValueType] = _NoValue) -> str:
Review comment:
files in `pyspark/sql` need to be changed because mypy check type of `_NoValue` after we inline type hint for `pyspark/__init__.py`
##########
File path: python/pyspark/__init__.py
##########
@@ -63,41 +64,54 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r'\n( +)')
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
+ indents = indent_p.findall(cast(str, f.__doc__))
indent = ' ' * (min(len(m) for m in indents) if indents else 0)
- f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+ f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
return f
return deco
-def copy_func(f, name=None, sinceversion=None, doc=None):
+def copy_func(
+ f: F,
+ name: Optional[str] = None,
+ sinceversion: Optional[Union[str, float]] = None,
+ doc: Optional[str] = None,
+) -> F:
"""
Returns a function with same code, globals, defaults, closure, and
name (or provide a new name).
"""
# See
# http://stackoverflow.com/questions/6527633/how-can-i-make-a-deepcopy-of-a-function-in-python
- fn = types.FunctionType(f.__code__, f.__globals__, name or f.__name__, f.__defaults__,
- f.__closure__)
+ fn = types.FunctionType(
+ f.__code__,
+ f.__globals__, # type: ignore[attr-defined]
+ name or f.__name__,
+ f.__defaults__, # type: ignore[attr-defined]
+ f.__closure__ # type: ignore[attr-defined]
Review comment:
mypy doesn't support ```__defaults__```, ```__globals__``` and ```__closure__```
https://github.com/python/mypy/issues/1923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954459604
**[Test build #144738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144738/testReport)** for PR 34433 at commit [`233b5ac`](https://github.com/apache/spark/commit/233b5ac02ec6d6e929131e713089cc5a3066aff5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966165825
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49568/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966224339
**[Test build #145108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145108/testReport)** for PR 34433 at commit [`a1671be`](https://github.com/apache/spark/commit/a1671be1b73a61c99274edfad1b1a4d07f73f87b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969698393
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145259/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 closed pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
zero323 closed pull request #34433:
URL: https://github.com/apache/spark/pull/34433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993226750
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50638/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993194066
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146165/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r747418144
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,55 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
+ indents = indent_p.findall(cast(str, f.__doc__))
indent = " " * (min(len(m) for m in indents) if indents else 0)
- f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+ f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
return f
return deco
-def copy_func(f, name=None, sinceversion=None, doc=None):
+def copy_func(
+ f: F,
+ name: Optional[str] = None,
+ sinceversion: Optional[Union[str, float]] = None,
+ doc: Optional[str] = None,
+) -> F:
"""
Returns a function with same code, globals, defaults, closure, and
name (or provide a new name).
"""
# See
# http://stackoverflow.com/questions/6527633/how-can-i-make-a-deepcopy-of-a-function-in-python
fn = types.FunctionType(
- f.__code__, f.__globals__, name or f.__name__, f.__defaults__, f.__closure__
+ f.__code__,
+ f.__globals__, # type: ignore[attr-defined]
Review comment:
Thank you! updated with `assert`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r747411405
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
Review comment:
I think we expect the output is ```Callable[[T], T]``` so `deco` should be ```T -> T```. Did I misunderstand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966224339
**[Test build #145108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145108/testReport)** for PR 34433 at commit [`a1671be`](https://github.com/apache/spark/commit/a1671be1b73a61c99274edfad1b1a4d07f73f87b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966330370
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966200949
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969755506
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49729/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-994191308
Thanks all. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966050102
**[Test build #145097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145097/testReport)** for PR 34433 at commit [`11de75f`](https://github.com/apache/spark/commit/11de75f24520885e91833ea6592e86aa23dbae81).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966049451
**[Test build #145097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145097/testReport)** for PR 34433 at commit [`11de75f`](https://github.com/apache/spark/commit/11de75f24520885e91833ea6592e86aa23dbae81).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r747411405
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
Review comment:
I think we expect the output is ```Callable[[T], T]``` so deco should be ```T -> ```. Did I misunderstand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966212107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966306242
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-994091268
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993226750
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50638/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993194066
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146165/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954613563
CC @HyukjinKwon @zero323 @ueshin FYI. Many thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993167036
**[Test build #146165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146165/testReport)** for PR 34433 at commit [`8d3be75`](https://github.com/apache/spark/commit/8d3be758efd4996737db297ee4f1742ed99488cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-993214326
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50638/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966049451
**[Test build #145097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145097/testReport)** for PR 34433 at commit [`11de75f`](https://github.com/apache/spark/commit/11de75f24520885e91833ea6592e86aa23dbae81).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r749023767
##########
File path: python/pyspark/__init__.py
##########
@@ -111,13 +124,13 @@ def keyword_only(func):
"""
@wraps(func)
- def wrapper(self, *args, **kwargs):
+ def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
if len(args) > 0:
raise TypeError("Method %s forces keyword arguments." % func.__name__)
self._input_kwargs = kwargs
return func(self, **kwargs)
- return wrapper
+ return cast(F, wrapper)
Review comment:
Be we expect output is `F` type so I think we should just cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r749023767
##########
File path: python/pyspark/__init__.py
##########
@@ -111,13 +124,13 @@ def keyword_only(func):
"""
@wraps(func)
- def wrapper(self, *args, **kwargs):
+ def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
if len(args) > 0:
raise TypeError("Method %s forces keyword arguments." % func.__name__)
self._input_kwargs = kwargs
return func(self, **kwargs)
- return wrapper
+ return cast(F, wrapper)
Review comment:
Because we expect output is `F` type so I think we should just cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r749601611
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
Review comment:
The `deco` will take a function to be decorated and return the decorated function.
So it should be `def deco(f: F) -> F:` and the return type of `since` should be `Callable[[F], F]`.
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2530,7 +2531,9 @@ def replace( # type: ignore[misc]
to_replace: Union[
"LiteralType", List["LiteralType"], Dict["LiteralType", "OptionalPrimitiveType"]
],
- value: Optional[Union["OptionalPrimitiveType", List["OptionalPrimitiveType"]]] = _NoValue,
+ value: Optional[
+ Union["OptionalPrimitiveType", List["OptionalPrimitiveType"], _NoValueType]
+ ] = _NoValue,
Review comment:
@zero323 Which part do you mean you don't want to leak?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r749831676
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
Review comment:
Thank you! updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969620591
**[Test build #145259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145259/testReport)** for PR 34433 at commit [`3684fc4`](https://github.com/apache/spark/commit/3684fc43a2bed3cb74672729bf5a6264fabd55c8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954488737
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144738/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954472695
**[Test build #144738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144738/testReport)** for PR 34433 at commit [`233b5ac`](https://github.com/apache/spark/commit/233b5ac02ec6d6e929131e713089cc5a3066aff5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954543194
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49207/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969790311
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49729/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-969790311
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49729/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954543194
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49207/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-954520286
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49207/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966053218
**[Test build #145099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145099/testReport)** for PR 34433 at commit [`b293718`](https://github.com/apache/spark/commit/b2937185bc248c4806fc6bcc7366033ec304a547).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r747340446
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
Review comment:
Shouldn't this be `F -> F`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34433:
URL: https://github.com/apache/spark/pull/34433#discussion_r747342243
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,56 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
+ indents = indent_p.findall(cast(str, f.__doc__))
indent = " " * (min(len(m) for m in indents) if indents else 0)
- f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+ f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
Review comment:
Instead of repeated casts one might `assert` instead:
```python
assert f.__doc__ is not None
indents = indent_p.findall(f.__doc__)
indent = " " * (min(len(m) for m in indents) if indents else 0)
f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
```
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2530,7 +2531,9 @@ def replace( # type: ignore[misc]
to_replace: Union[
"LiteralType", List["LiteralType"], Dict["LiteralType", "OptionalPrimitiveType"]
],
- value: Optional[Union["OptionalPrimitiveType", List["OptionalPrimitiveType"]]] = _NoValue,
+ value: Optional[
+ Union["OptionalPrimitiveType", List["OptionalPrimitiveType"], _NoValueType]
+ ] = _NoValue,
Review comment:
I don't think we want to leak this to the end user here, but let's see what others have to say about it.
FYI @ueshin @xinrong-databricks @itholic
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -3337,7 +3340,9 @@ def replace(
def replace( # type: ignore[misc]
self,
to_replace: Union[List["LiteralType"], Dict["LiteralType", "OptionalPrimitiveType"]],
- value: Optional[Union["OptionalPrimitiveType", List["OptionalPrimitiveType"]]] = _NoValue,
+ value: Optional[
+ Union["OptionalPrimitiveType", List["OptionalPrimitiveType"], _NoValueType]
+ ] = _NoValue,
Review comment:
Ditto.
##########
File path: python/pyspark/__init__.py
##########
@@ -63,44 +64,55 @@
from pyspark.version import __version__
from pyspark._globals import _NoValue # noqa: F401
+T = TypeVar("T")
+F = TypeVar("F", bound=Callable)
-def since(version):
+def since(version: Union[str, float]) -> Callable[[T], T]:
"""
A decorator that annotates a function to append the version of Spark the function was added.
"""
import re
indent_p = re.compile(r"\n( +)")
- def deco(f):
- indents = indent_p.findall(f.__doc__)
+ def deco(f: T) -> T:
+ indents = indent_p.findall(cast(str, f.__doc__))
indent = " " * (min(len(m) for m in indents) if indents else 0)
- f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
+ f.__doc__ = cast(str, f.__doc__).rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
return f
return deco
-def copy_func(f, name=None, sinceversion=None, doc=None):
+def copy_func(
+ f: F,
+ name: Optional[str] = None,
+ sinceversion: Optional[Union[str, float]] = None,
+ doc: Optional[str] = None,
+) -> F:
"""
Returns a function with same code, globals, defaults, closure, and
name (or provide a new name).
"""
# See
# http://stackoverflow.com/questions/6527633/how-can-i-make-a-deepcopy-of-a-function-in-python
fn = types.FunctionType(
- f.__code__, f.__globals__, name or f.__name__, f.__defaults__, f.__closure__
+ f.__code__,
+ f.__globals__, # type: ignore[attr-defined]
Review comment:
Have your tried to follow this suggestion?
> FWIW casting to types.FunctionType should do it, that seems to have all of these.
https://github.com/python/mypy/issues/1923#issuecomment-234669486
In general, if we have expect `FunctionType` on output, and we clearly expect it on input, we might consider redefining `TypeVar` here.
##########
File path: python/pyspark/__init__.py
##########
@@ -111,13 +124,13 @@ def keyword_only(func):
"""
@wraps(func)
- def wrapper(self, *args, **kwargs):
+ def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
if len(args) > 0:
raise TypeError("Method %s forces keyword arguments." % func.__name__)
self._input_kwargs = kwargs
return func(self, **kwargs)
- return wrapper
+ return cast(F, wrapper)
Review comment:
Seems like the way to go, but might require some comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966162345
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966194302
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49568/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966246059
**[Test build #145108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145108/testReport)** for PR 34433 at commit [`a1671be`](https://github.com/apache/spark/commit/a1671be1b73a61c99274edfad1b1a4d07f73f87b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966212107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966249692
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966277720
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966053218
**[Test build #145099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145099/testReport)** for PR 34433 at commit [`b293718`](https://github.com/apache/spark/commit/b2937185bc248c4806fc6bcc7366033ec304a547).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34433: [SPARK-37146][PYTHON] Inline type hints for python/pyspark/__init__.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34433:
URL: https://github.com/apache/spark/pull/34433#issuecomment-966075658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145099/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org