You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/11 05:24:07 UTC
[GitHub] [spark] Yikun opened a new pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Yikun opened a new pull request #32122:
URL: https://github.com/apache/spark/pull/32122
### What changes were proposed in this pull request?
Fix type hints mismatches in pyspark.sql.*
### Why are the changes needed?
There were some mismatches in pyspark.sql.*
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
dev/lint-python passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817279950
cc @zero323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817418931
@zero323 OK, if it should be suqashed into your patch, I'd like to close this PR, or we could just merge this first?
And looks like you have already started to fix these mismacthes for [Python type hints improvement](https://issues.apache.org/jira/browse/SPARK-34629), so I am going stop these work (fix these mismatches module by module), but if you need any help pls feel free to ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817252754
**[Test build #137177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137177/testReport)** for PR 32122 at commit [`d9d40af`](https://github.com/apache/spark/commit/d9d40afb383d1f5e63949520b7bf2a8785ea6bde).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817750241
Thanks for your detail and patient explanation for the background of type hints, learned a lot : ) and really much thanks for your previous effort on python stub works.
And I started to the annotions fix work because I saw [SPARK-34629](https://issues.apache.org/jira/browse/SPARK-34629) is `OPEN` and have a short discussion with @HyukjinKwon, and found there's some mismatches on type hints, so I thought maybe can help some on it. I believe maybe there are many other people also have will to help it.
> We should probably avoid fixing individual, low priority (not bug, not reported as an issue) annotations in separate PRs, to keep review work managable, but I don't think its a problem here.
Definately yes, I was going to fix these type annotations by module ranther than by file or smaller granularity (such as this patch, I looked in deep in every files under the pyspark.sql.*), I also considered the huge patch will hard to review, so I just submit it in the score of module. And also, I paste the origin py lines for every pyi.
> I am bit preoccupied lately, but if you plan to work more on this part and have any questions, please let me know.
I really know about the difficult of fix these (due to large amount of py files and needs to be very careful), but I'm still not sure it's (I mean fix and submit it by module) the good way to improve pyspark hints or not. If yes, I will continue this work when I have time; if not, maybe we also need to find a better way to let more people join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817257334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817418931
@zero323 OK, will fix the `between` change, if it should be suqashed into your patch, I'd like to close this PR, or we could just merge this first?
And looks like you have already started to fix these mismacthes for [Python type hints improvement](https://issues.apache.org/jira/browse/SPARK-34629), so I am going stop these work (fix these mismatches module by module), but if you need any help pls feel free to ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32122:
URL: https://github.com/apache/spark/pull/32122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817772044
**[Test build #137213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137213/testReport)** for PR 32122 at commit [`a45bdbb`](https://github.com/apache/spark/commit/a45bdbb9bf4137cc9028e973e7b042d5af40b792).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817251407
**[Test build #137175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137175/testReport)** for PR 32122 at commit [`999f925`](https://github.com/apache/spark/commit/999f925c443d0e6e669aedfb6d0627e39118a977).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-818382544
Merged to master and branch-3.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817803138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32122:
URL: https://github.com/apache/spark/pull/32122#discussion_r611133213
##########
File path: python/pyspark/sql/catalog.pyi
##########
@@ -53,7 +54,7 @@ class Catalog:
def dropGlobalTempView(self, viewName: str) -> None: ...
def registerFunction(
self, name: str, f: Callable[..., Any], returnType: DataType = ...
- ) -> None: ...
+ ) -> UserDefinedFunctionLike: ...
Review comment:
Note for reviewer:
https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/python/pyspark/sql/catalog.py#L257
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/udf.pyi#L46-L51
##########
File path: python/pyspark/sql/column.pyi
##########
@@ -109,7 +109,7 @@ class Column:
self,
lowerBound: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral],
upperBound: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral],
- ) -> Column: ...
+ ) -> bool: ...
Review comment:
Note for reviewer:
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/column.py#L801
##########
File path: python/pyspark/sql/context.pyi
##########
@@ -63,7 +64,7 @@ class SQLContext:
) -> DataFrame: ...
def registerFunction(
self, name: str, f: Callable[..., Any], returnType: DataType = ...
- ) -> None: ...
+ ) -> UserDefinedFunctionLike: ...
Review comment:
Note for reviewer:
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/context.py#L236
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/udf.pyi#L46-L51
##########
File path: python/pyspark/sql/functions.pyi
##########
@@ -57,7 +57,7 @@ def monotonically_increasing_id() -> Column: ...
def nanvl(col1: ColumnOrName, col2: ColumnOrName) -> Column: ...
def percentile_approx(
col: ColumnOrName,
- percentage: Union[Column, float, List[float]],
+ percentage: Union[Column, float, List[float], tuple[float]],
Review comment:
Note for reviewer:
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/functions.py#L1218
##########
File path: python/pyspark/sql/conf.pyi
##########
@@ -21,7 +21,7 @@ from py4j.java_gateway import JavaObject # type: ignore[import]
class RuntimeConfig:
def __init__(self, jconf: JavaObject) -> None: ...
- def set(self, key: str, value: str) -> str: ...
+ def set(self, key: str, value: str) -> None: ...
Review comment:
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/conf.py#L34-L36
##########
File path: python/pyspark/sql/dataframe.pyi
##########
@@ -85,7 +85,7 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
def checkpoint(self, eager: bool = ...) -> DataFrame: ...
def localCheckpoint(self, eager: bool = ...) -> DataFrame: ...
def withWatermark(
- self, eventTime: ColumnOrName, delayThreshold: str
+ self, eventTime: str, delayThreshold: str
Review comment:
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/dataframe.py#L608-L609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817754024
> I've done first sweep and, with exception of Column.between, it looks OK.
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817718455
Personally, I don't see a problemem with having this (with addressedw current and possibly future comments) added separately.
Annotations are pretty large and not fixed thing - current state is a result of a few years of maintenance and multiple passes and there still problems that haven't been noticed so far. Not to mention there are parts that are morenmy personal an opinion about style (i.e. list vs. tuple usage or balance between false positives and false negatives), which are probably disputable if you come from different background. There are also some decisions that we made during initial port (some tests where omitted, some stubs where removed, we decided to prefer ignores instead of adding new annotations in quite a few places).
We should probably avoid fixing individual, low priority (not bug, not reported as an issue) annotations in separate PRs, to keep review work.
managable, but I don't think its a problem here.
Also, there are cases where annotations require a lot of caution, like ML or UDFs.
I am bit preoccupied lately, but if you plan to work more on this part and have any questions, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817750241
Thanks for your detail and patient explanation for the background of type hints, learned a lot : ) and really much thanks for your previous effort on python stub works.
And I started to the annotions fix work because I saw [SPARK-34629](https://issues.apache.org/jira/browse/SPARK-34629) is `OPEN` and have a short discussion with @HyukjinKwon, and found there's some mismatches on type hints, so I thought maybe can help some on it. I believe maybe there are many other people also have will to help it.
> We should probably avoid fixing individual, low priority (not bug, not reported as an issue) annotations in separate PRs, to keep review work managable, but I don't think its a problem here.
Definately yes, I was going to fix these type annotations by module ranther than by file or smaller granularity (such as this patch, I looked in deep in every files under the pyspark.sql.*), I also considered the huge patch will hard to review, so I just submit it in the score of module. And also, I paste the origin py lines for every pyi.
> I am bit preoccupied lately, but if you plan to work more on this part and have any questions, please let me know.
I really know about the difficult of fix these (due to large amount of py files and needs to be very careful), but I'm still not sure it's the good way (I mean fix and submit it by module) to improve pyspark hints or not. If yes, I will continue this work when I have time; if not, maybe we also need to find a better way to let more people join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817812550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817257210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32122:
URL: https://github.com/apache/spark/pull/32122#discussion_r611567066
##########
File path: python/pyspark/sql/column.pyi
##########
@@ -109,7 +109,7 @@ class Column:
self,
lowerBound: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral],
upperBound: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral],
- ) -> Column: ...
+ ) -> bool: ...
Review comment:
Done, : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817330608
I've done first sweep and, with exception of `Column.between`, it looks OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817257210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817253704
**[Test build #137175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137175/testReport)** for PR 32122 at commit [`999f925`](https://github.com/apache/spark/commit/999f925c443d0e6e669aedfb6d0627e39118a977).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817252754
**[Test build #137177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137177/testReport)** for PR 32122 at commit [`d9d40af`](https://github.com/apache/spark/commit/d9d40afb383d1f5e63949520b7bf2a8785ea6bde).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817257342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41755/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817251407
**[Test build #137175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137175/testReport)** for PR 32122 at commit [`999f925`](https://github.com/apache/spark/commit/999f925c443d0e6e669aedfb6d0627e39118a977).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817772044
**[Test build #137213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137213/testReport)** for PR 32122 at commit [`a45bdbb`](https://github.com/apache/spark/commit/a45bdbb9bf4137cc9028e973e7b042d5af40b792).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32122:
URL: https://github.com/apache/spark/pull/32122#discussion_r611134480
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -588,7 +588,7 @@ def withWatermark(self, eventTime, delayThreshold):
Parameters
----------
- eventTime : str or :class:`Column`
+ eventTime : str
Review comment:
The scalar eventTime is also `str`
https://github.com/apache/spark/blob/61d038f26e1beeeb39638fdf9703d2e86d058342/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L753
and python implements is also validated as string
https://github.com/apache/spark/blob/ff1fc5ed4b685b4f5f83d5f600b24f089dd4522e/python/pyspark/sql/dataframe.py#L608-L609
but wrong doc on the param and typehints
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #32122:
URL: https://github.com/apache/spark/pull/32122#discussion_r611209494
##########
File path: python/pyspark/sql/column.pyi
##########
@@ -109,7 +109,7 @@ class Column:
self,
lowerBound: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral],
upperBound: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral],
- ) -> Column: ...
+ ) -> bool: ...
Review comment:
This doesn't look right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817254728
**[Test build #137177 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137177/testReport)** for PR 32122 at commit [`d9d40af`](https://github.com/apache/spark/commit/d9d40afb383d1f5e63949520b7bf2a8785ea6bde).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817788330
**[Test build #137213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137213/testReport)** for PR 32122 at commit [`a45bdbb`](https://github.com/apache/spark/commit/a45bdbb9bf4137cc9028e973e7b042d5af40b792).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817812550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817257342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41755/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32122: [SPARK-35019][PYTHON][SQL]Fix type hints mismatches in pyspark.sql.*
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32122:
URL: https://github.com/apache/spark/pull/32122#issuecomment-817255972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org