You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/14 07:08:45 UTC
[GitHub] [spark] ueshin opened a new pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
ueshin opened a new pull request #32871:
URL: https://github.com/apache/spark/pull/32871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32871:
URL: https://github.com/apache/spark/pull/32871#issuecomment-859176870
**[Test build #139664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139664/testReport)** for PR 32871 at commit [`81a1042`](https://github.com/apache/spark/commit/81a10425a0d4feaa08532e0138c293e9a013337e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32871:
URL: https://github.com/apache/spark/pull/32871#discussion_r650161060
##########
File path: python/pyspark/pandas/namespace.py
##########
@@ -2480,6 +2531,7 @@ def isna(obj):
isnull = isna
+@no_type_check
Review comment:
These could be used as a udf for `transform` or other functions, although usually it shouldn't happen.
In that case, it has to know the return type but if there is a return type annotation, it can't infer the return type.
##########
File path: python/pyspark/pandas/utils.py
##########
@@ -600,7 +600,7 @@ def column_labels_level(column_labels: List[Tuple]) -> int:
return list(levels)[0]
-def name_like_string(name: Optional[Union[str, Tuple]]) -> str:
+def name_like_string(name: Optional[Union[Any, Tuple]]) -> str:
Review comment:
Good point!
Actually `str` here was a mistake because the `name` in pandas-on-Spark can be non-string values, like `int`, `float`, etc., and `mypy` reported an error.
##########
File path: python/pyspark/pandas/utils.py
##########
@@ -600,7 +600,7 @@ def column_labels_level(column_labels: List[Tuple]) -> int:
return list(levels)[0]
-def name_like_string(name: Optional[Union[str, Tuple]]) -> str:
+def name_like_string(name: Optional[Union[Any, Tuple]]) -> str:
Review comment:
Good point!
Actually `str` here was a mistake because the `name` in pandas-on-Spark can be non-string values, like `int`, `float`, etc., and `mypy` reported an error in the `namespace.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #32871:
URL: https://github.com/apache/spark/pull/32871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #32871:
URL: https://github.com/apache/spark/pull/32871#issuecomment-859211559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32871:
URL: https://github.com/apache/spark/pull/32871#issuecomment-859176870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32871:
URL: https://github.com/apache/spark/pull/32871#issuecomment-859195281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32871:
URL: https://github.com/apache/spark/pull/32871#discussion_r649665382
##########
File path: python/pyspark/pandas/utils.py
##########
@@ -600,7 +600,7 @@ def column_labels_level(column_labels: List[Tuple]) -> int:
return list(levels)[0]
-def name_like_string(name: Optional[Union[str, Tuple]]) -> str:
+def name_like_string(name: Optional[Union[Any, Tuple]]) -> str:
Review comment:
Question: Is this change required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32871:
URL: https://github.com/apache/spark/pull/32871#issuecomment-859195281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32871:
URL: https://github.com/apache/spark/pull/32871#discussion_r649659322
##########
File path: python/pyspark/pandas/namespace.py
##########
@@ -2480,6 +2531,7 @@ def isna(obj):
isnull = isna
+@no_type_check
Review comment:
Out of curiosity, what methods should we mark as `no_type_check`? With only one parameter that is of Any type?
##########
File path: python/pyspark/pandas/namespace.py
##########
@@ -2480,6 +2531,7 @@ def isna(obj):
isnull = isna
+@no_type_check
Review comment:
According to its definition, `no_type_check` is `Decorator to indicate that annotations are not type hints.`.
Are there any annotations in this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32871:
URL: https://github.com/apache/spark/pull/32871#discussion_r651086027
##########
File path: python/pyspark/pandas/namespace.py
##########
@@ -2480,6 +2531,7 @@ def isna(obj):
isnull = isna
+@no_type_check
Review comment:
Makes sense! Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #32871: [SPARK-35475][PYTHON] Fix disallow_untyped_defs mypy checks.
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #32871:
URL: https://github.com/apache/spark/pull/32871#issuecomment-859231008
LGTM, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org