You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/31 06:17:28 UTC

[GitHub] [spark] zero323 opened a new pull request #29591: [SPARK-32681][PYTHON] Initial pyspark-stubs port.

zero323 opened a new pull request #29591:
URL: https://github.com/apache/spark/pull/29591


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683915728






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688423212


   **[Test build #128361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128361/testReport)** for PR 29591 at commit [`d7d876a`](https://github.com/apache/spark/commit/d7d876a1f070bd672baf9ca4f1d26e0f04632c13).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687918072






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon edited a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688561042


   cc @BryanCutler, @viirya, @ueshin, @Fokko, @holdenk can you review this when you guys fine some time?
   
   FYI:
   
   We'll probably try to expose the type hints for user-facing APIs only but it will be done in a separate JIRA. It was discussed at https://github.com/apache/spark/pull/29591#discussion_r479940442.
   
   We'll need to document when/where/how to do with type hinting in contribution guide as https://github.com/apache/spark/blob/master/python/docs/source/development/contributing.rst in a separate JIRA.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32681][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683585253






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688085102






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-684399679






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698132703


   > @zero323 mind working on the below ones?
   
   On it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698117279


   @zero323 mind working on:
   - writing the guidelines in the doc
   - removing non-API type hints
   ?
   I think these two are pretty important followups to be done soon ..


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697185517






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688506080






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r487129652



##########
File path: dev/tox.ini
##########
@@ -20,5 +20,5 @@ exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,wor
 
 [flake8]
 select = E901,E999,F821,F822,F823,F401,F405
-exclude = python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
+exclude = python/pyspark/cloudpickle/*.py,shared.py*,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*,python/out,python/pyspark/sql/pandas/functions.pyi,python/pyspark/sql/column.pyi,python/pyspark/worker.pyi,python/pyspark/java_gateway.pyi

Review comment:
       > Personally, I would require mypy to be added in this PR As well. Otherwise we run the risk that the annotations might go out of sync quickly.
   
   In general I see your point, however I wanted to avoid duplicating your work from https://github.com/apache/spark/pull/29180 and we cannot setup the environment anyway.
   
   
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690709554






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r484137854



##########
File path: examples/src/main/python/ml/estimator_transformer_param_example.py
##########
@@ -54,7 +56,7 @@
     print(model1.extractParamMap())
 
     # We may alternatively specify parameters using a Python dictionary as a paramMap
-    paramMap = {lr.maxIter: 20}
+    paramMap: Dict[Param, Any] = {lr.maxIter: 20}

Review comment:
       @zero323, is it because mypy complains? We can exclude the example files from the mypy check for the time being I guess rather than adding the types into the codes for now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693581392






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 edited a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 edited a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-686278472


   __Update__
   
   At the moment I'm working on re-syncing `pyspark-stubs` to reflect changes introduced by [SPARK-32719](https://issues.apache.org/jira/browse/SPARK-32719) and [SPARK-32319](https://issues.apache.org/jira/browse/SPARK-32319).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697147463






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489410868



##########
File path: python/pyspark/sql/column.pyi
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import overload
+from typing import Any, Union
+
+from pyspark.sql._typing import LiteralType, DecimalLiteral, DateTimeLiteral
+from pyspark.sql.types import (  # noqa: F401
+    DataType,
+    StructField,
+    StructType,
+    IntegerType,
+    StringType,
+)
+from pyspark.sql.window import WindowSpec
+
+from py4j.java_gateway import JavaObject  # type: ignore[import]
+
+class Column:
+    def __init__(self, JavaObject) -> None: ...
+    def __neg__(self) -> Column: ...
+    def __add__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __sub__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __mul__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __div__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __truediv__(
+        self, other: Union[Column, LiteralType, DecimalLiteral]
+    ) -> Column: ...
+    def __mod__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __radd__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rsub__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rmul__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rdiv__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rtruediv__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rmod__(self, other: Union[bool, int, float, DecimalLiteral]) -> Column: ...
+    def __pow__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __rpow__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __eq__(self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]) -> Column: ...  # type: ignore[override]
+    def __ne__(self, other: Any) -> Column: ...  # type: ignore[override]
+    def __lt__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def __le__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def __ge__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def __gt__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def eqNullSafe(
+        self, other: Union[Column, LiteralType, DecimalLiteral]
+    ) -> Column: ...
+    def __and__(self, other: Column) -> Column: ...
+    def __or__(self, other: Column) -> Column: ...
+    def __invert__(self) -> Column: ...
+    def __rand__(self, other: Column) -> Column: ...
+    def __ror__(self, other: Column) -> Column: ...
+    def __contains__(self, other: Any) -> Column: ...
+    def __getitem__(self, other: Any) -> Column: ...
+    def bitwiseOR(self, other: Union[Column, int]) -> Column: ...
+    def bitwiseAND(self, other: Union[Column, int]) -> Column: ...
+    def bitwiseXOR(self, other: Union[Column, int]) -> Column: ...
+    def getItem(self, key: Any) -> Column: ...
+    def getField(self, name: Any) -> Column: ...
+    def __getattr__(self, item: Any) -> Column: ...
+    def __iter__(self) -> None: ...
+    def rlike(self, item: str) -> Column: ...
+    def like(self, item: str) -> Column: ...
+    def startswith(self, item: Union[str, Column]) -> Column: ...
+    def endswith(self, item: Union[str, Column]) -> Column: ...
+    @overload
+    def substr(self, startPos: int, length: int) -> Column: ...
+    @overload
+    def substr(self, startPos: Column, length: Column) -> Column: ...
+    def __getslice__(self, startPos: int, length: int) -> Column: ...
+    def isin(self, *cols: Any) -> Column: ...
+    def asc(self) -> Column: ...
+    def asc_nulls_first(self) -> Column: ...
+    def asc_nulls_last(self) -> Column: ...
+    def desc(self) -> Column: ...
+    def desc_nulls_first(self) -> Column: ...
+    def desc_nulls_last(self) -> Column: ...
+    def isNull(self) -> Column: ...
+    def isNotNull(self) -> Column: ...
+    def alias(self, *alias: str, **kwargs: Any) -> Column: ...
+    def name(self, *alias: str) -> Column: ...
+    def cast(self, dataType: Union[DataType, str]) -> Column: ...
+    def astype(self, dataType: Union[DataType, str]) -> Column: ...
+    def between(self, lowerBound, upperBound) -> Column: ...
+    def when(self, condition: Column, value: Any) -> Column: ...
+    def otherwise(self, value: Any) -> Column: ...
+    def over(self, window: WindowSpec) -> Column: ...
+    def __nonzero__(self) -> None: ...
+    def __bool__(self) -> None: ...

Review comment:
       Nice one!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693673774


   **[Test build #128777 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128777/testReport)** for PR 29591 at commit [`9998d0a`](https://github.com/apache/spark/commit/9998d0adcaed46a48f795f078e32b5f2dbb9151c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693593841


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128775/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697052814


   @zero323, sorry I caused a bit of conflicts. Once it's resolved, let's just merge and go ahead.
   I think there's not much point of leaving it open and blocking the works.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690035346






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r486075570



##########
File path: python/pyspark/ml/common.pyi
##########
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Stubs for pyspark.ml.common (Python 3)
+
+from typing import Type
+
+long: Type[int]

Review comment:
       That's an artifact of Python 2 support. Let me fix it upstream and resync.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r479940442



##########
File path: python/pyspark/cloudpickle.pyi
##########
@@ -0,0 +1,86 @@
+#

Review comment:
       I believe this file can be removed. It's a copy of [cloudpickle](https://github.com/cloudpipe/cloudpickle), which should be separate from PySpark. I believe it makes less sense to type this file.
   
   BTW, should we maybe start it with typing only exposed APIs for now? I think having type hints in internal packages has less benefits (only to PySpark dev) and it's arguable to maintain it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687916040


   **[Test build #128327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128327/testReport)** for PR 29591 at commit [`1aede7c`](https://github.com/apache/spark/commit/1aede7c7616fa045dae2bd9483e24ed48f089a02).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697176725






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687901349






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688460425


   **[Test build #128363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128363/testReport)** for PR 29591 at commit [`d7d876a`](https://github.com/apache/spark/commit/d7d876a1f070bd672baf9ca4f1d26e0f04632c13).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688423801






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687901337


   **[Test build #128324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128324/testReport)** for PR 29591 at commit [`93fb711`](https://github.com/apache/spark/commit/93fb711e1621abcb944eb4b8b411d6f670cf7774).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692315021






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697303691


   I'm going to merge if there's no more comment tomorrow.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697390632






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697980912


   Jenkins, retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687929579


   **[Test build #128325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128325/testReport)** for PR 29591 at commit [`27409f2`](https://github.com/apache/spark/commit/27409f29fa2c5f9515a1af55085f0d2a12113d01).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692343926


   **[Test build #128667 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128667/testReport)** for PR 29591 at commit [`6272d9d`](https://github.com/apache/spark/commit/6272d9db38ba7d23ba8fc441f359ee1abce34e32).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r484184260



##########
File path: examples/src/main/python/ml/estimator_transformer_param_example.py
##########
@@ -54,7 +56,7 @@
     print(model1.extractParamMap())
 
     # We may alternatively specify parameters using a Python dictionary as a paramMap
-    paramMap = {lr.maxIter: 20}
+    paramMap: Dict[Param, Any] = {lr.maxIter: 20}

Review comment:
       It is. In such cases  Mypy infers the type of `paramMap` to be `Dict[Param, int]` and will fail on subsequent update:
   
   ```python
   paramMap.update({lr.regParam: 0.1, lr.threshold: 0.55}) 
   ```
   
   In general I'd prefer to keep tests against examples as these are the biggest chunks of "real-life" code (short of docstrings, but these create way more issues)  that we have and nicely highlight many possible problems.
   
   In `pyspark-stubs` I simply clone the repo and [patch examples](https://github.com/zero323/pyspark-stubs/blob/faad51cfd4b8971cda889f79c6857dd28ad62078/.travis.yml#L11-L12) before tests. 
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r486428984



##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,8 +32,8 @@
 
 
 def dataframe_with_arrow_example(spark):
-    import numpy as np
-    import pandas as pd
+    import numpy as np  # type: ignore[import]

Review comment:
       Because, AFAIK, released versions are not typed ‒ doesn't seem there were any released versions (nothing for 1.19.1 ‒https://github.com/numpy/numpy/tree/v1.19.1/numpy) since https://github.com/numpy/numpy/commit/11b95d15f10c2bc652ed19d5e27efa0384396cb8.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688058436


   **[Test build #128338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128338/testReport)** for PR 29591 at commit [`8f9ef95`](https://github.com/apache/spark/commit/8f9ef95f570dee0167ace375676d64b39829b3d1).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32681][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683585043


   **[Test build #128074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128074/testReport)** for PR 29591 at commit [`60b126e`](https://github.com/apache/spark/commit/60b126e9b25bdad0b44b6e484f6a3c053b254f3e).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r484560092



##########
File path: python/MANIFEST.in
##########
@@ -22,4 +22,5 @@ recursive-include deps/data *.data *.txt
 recursive-include deps/licenses *.txt
 recursive-include deps/examples *.py
 recursive-include lib *.zip
+recursive-include pyspark *.pyi py.typed

Review comment:
       __Note__:
   
   Is it possible to do it through `package_data`, i.e.
   
   ```python
           package_data={
               '': ['*.pyi', 'py.typed'],
               ...
   ```
   
   Doesn't seem to work here...




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683593190


   Let's use SPARK-32714 for this initial port, and use SPARK-32681 as an umbrella ticket to add other related tickets for followup works.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697051865






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489419177



##########
File path: python/pyspark/sql/pandas/_typing/protocols/series.pyi
##########
@@ -0,0 +1,253 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This Protocol resuses core Pandas annotation.
+# Overall pipeline looks as follows
+# - Stubgen pandas.core.series
+# - Add Protocol as a base class
+# - Replace imports with Any
+
+import numpy as np  # type: ignore[import]
+from typing import Any, Callable, Hashable, IO, Optional
+from typing_extensions import Protocol
+
+groupby_generic = Any
+
+class SeriesLike(Protocol):

Review comment:
       Oh... That's a tricky question. Let me provide some context...
   
   In general this protocols exist not so much to annotate Pandas interfaces, but rather to make our side more robust. Initially, I just type ignored Pandas and was pretty satisfied about it.
   
   However, I missed that on the way, it resulted in a situation, where ignore swallowed all the errors. Since Pandas-related are quite complex and it is easy to make a mistake I introduced this thingy.
   
   So what can be done here:
   
   - Removing protocols and going back to `type: ignore`. To be honest, I am not very fond of that idea, but I see why you're concerned.
   - Creating a transient package that will fill the hole we have here.
   - Trying to convince good Pandas folks to take a leap [here](https://github.com/pandas-dev/pandas/issues/28142) :)
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693218508


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697266761






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-684398372


   **[Test build #128137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128137/testReport)** for PR 29591 at commit [`df12013`](https://github.com/apache/spark/commit/df12013595906747ca39d3a59b99def246348663).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693148274


   **[Test build #128740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128740/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697657470






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489509397



##########
File path: python/pyspark/sql/pandas/_typing/protocols/series.pyi
##########
@@ -0,0 +1,253 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This Protocol resuses core Pandas annotation.
+# Overall pipeline looks as follows
+# - Stubgen pandas.core.series
+# - Add Protocol as a base class
+# - Replace imports with Any
+
+import numpy as np  # type: ignore[import]
+from typing import Any, Callable, Hashable, IO, Optional
+from typing_extensions import Protocol
+
+groupby_generic = Any
+
+class SeriesLike(Protocol):

Review comment:
       One argument for keeping this as-is, is that it is primarily an internal type checking feature. It is not exposed in a way that can be used by the final user (though I thought about it in the past). 
   
   Thought it might be an argument against keeping it in the first place.
   
   One more option is to extract both Pandas annotations from source just for the use in CI pipeline, until typed version is released. This will ensure consistency on our side.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693218508






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687956805


   @zero323, I usually prefer to don't block something by the env issue in Jenkins so such issue can be handled with enough time - @shaneknapp is sort of busy at this moment IIRC. We could work around for now, and file a separate JIRA for him about the dependency upgade.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687918072






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692229669






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683907533


   **[Test build #128109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128109/testReport)** for PR 29591 at commit [`a936e03`](https://github.com/apache/spark/commit/a936e033bc640ce5862e84a3952b62570f81cac1).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687906242


   **[Test build #128326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128326/testReport)** for PR 29591 at commit [`b8d4876`](https://github.com/apache/spark/commit/b8d4876ecce95e9e62f6069871e0e835216cad65).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692512473






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688084983






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697390632






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690025077


   **[Test build #128491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128491/testReport)** for PR 29591 at commit [`8608a12`](https://github.com/apache/spark/commit/8608a1251e372e9bf097efe084fe49b769f1599b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687916361






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690930268


   Will take a look too @zero323 within few days. Thanks for doing this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon edited a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698117279


   @zero323 mind working on the below ones?
   
   - writing the guidelines in the doc
   - removing non-API type hints
   
   I think these two are pretty important followups to be done soon ..


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489350335



##########
File path: python/pyspark/ml/evaluation.pyi
##########
@@ -0,0 +1,283 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import abc
+from typing import Optional, Type
+from pyspark.ml._typing import (
+    ParamMap,
+    BinaryClassificationEvaluatorMetricType,
+    ClusteringEvaluatorMetricType,
+    MulticlassClassificationEvaluatorMetricType,
+    MultilabelClassificationEvaluatorMetricType,
+    RankingEvaluatorMetricType,
+    RegressionEvaluatorMetricType,
+)
+
+from pyspark.ml.wrapper import JavaParams
+from pyspark.ml.param import Param, Params
+from pyspark.ml.param.shared import (
+    HasFeaturesCol,
+    HasLabelCol,
+    HasPredictionCol,
+    HasProbabilityCol,
+    HasRawPredictionCol,
+    HasWeightCol,
+)
+from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+
+class Evaluator(Params):
+    __metaclass__: Type[abc.ABCMeta]
+    def evaluate(self, dataset, params: Optional[ParamMap] = ...) -> float: ...
+    def isLargerBetter(self) -> bool: ...
+
+class JavaEvaluator(JavaParams, Evaluator):
+    __metaclass__: Type[abc.ABCMeta]
+    def isLargerBetter(self) -> bool: ...
+
+class BinaryClassificationEvaluator(
+    JavaEvaluator,
+    HasLabelCol,
+    HasRawPredictionCol,
+    HasWeightCol,
+    JavaMLReadable[BinaryClassificationEvaluator],
+    JavaMLWritable,
+):
+    metricName: Param[BinaryClassificationEvaluatorMetricType]
+    numBins: Param[int]
+    def __init__(
+        self,
+        *,

Review comment:
       We should actually change in the main codes too .. and remove `keyword_only`. Of course in a separate PR.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489419328



##########
File path: python/pyspark/sql/types.pyi
##########
@@ -0,0 +1,220 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import overload
+from typing import Any, Callable, Dict, Iterator, List, Optional, Union, Tuple, TypeVar
+import datetime
+
+T = TypeVar("T")
+U = TypeVar("U")
+
+class DataType:
+    def __hash__(self) -> int: ...
+    def __eq__(self, other: Any) -> bool: ...
+    def __ne__(self, other: Any) -> bool: ...
+    @classmethod
+    def typeName(cls) -> str: ...
+    def simpleString(self) -> str: ...
+    def jsonValue(self) -> Union[str, Dict[str, Any]]: ...
+    def json(self) -> str: ...
+    def needConversion(self) -> bool: ...
+    def toInternal(self, obj: Any) -> Any: ...
+    def fromInternal(self, obj: Any) -> Any: ...
+
+class DataTypeSingleton(type):
+    def __call__(cls): ...
+
+class NullType(DataType):
+    __metaclass__: type

Review comment:
       Thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688460425


   **[Test build #128363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128363/testReport)** for PR 29591 at commit [`d7d876a`](https://github.com/apache/spark/commit/d7d876a1f070bd672baf9ca4f1d26e0f04632c13).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687917811


   **[Test build #128328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128328/testReport)** for PR 29591 at commit [`601a577`](https://github.com/apache/spark/commit/601a57787e5e0224ee000ccef0a817c2ee493b65).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690035155


   **[Test build #128491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128491/testReport)** for PR 29591 at commit [`8608a12`](https://github.com/apache/spark/commit/8608a1251e372e9bf097efe084fe49b769f1599b).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687916361






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692270432






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690489890


   **[Test build #128529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128529/testReport)** for PR 29591 at commit [`91cd14b`](https://github.com/apache/spark/commit/91cd14bac570513c5acf870dfa8bc3aea19aaa6e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688059124






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687918887


   __Update__: 
   
   At the moment:
   
   -  both MyPy and Flake8 tests pass with 
     - `flake8==3.8.3`
     - `flake8-pyi==20.5.0`
   
     and F401 (unused import) excludes on a few `pyi` files.
   
   - With `flake8==3.7.*` all tests pass with excludes as [added to tox fiile](https://github.com/apache/spark/pull/29591/files#diff-3d0e2fd61fb61f36804ed2ea3f8ff95f). Additionally to F401 violations, flake8 doesn't seem to understand specific type ignores.
   
   - With `flake8==3.5.0` there are no file specific ignores, so we get multiple failures. This could be addressed, for the time being, by either excluding problematic files from mypy or flake8 tests, and adjusting inline ignores. However, it is rather brute-force solution, and would require some discussion about the priorities.
   
   From the perspective of this PR, an ideal solution would be an update of test dependencies, but I am not sure if that's realistic at the moment (hate to ask, but do you have any thoughts about it @shaneknapp?).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687906433


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128326/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688630650


   __Note__ 
   
   In `pyspark-stubs` we use data driven test cases ([scenarios](https://github.com/zero323/pyspark-stubs/tree/master/test-data/unit) and [runner](https://github.com/zero323/pyspark-stubs/blob/master/tests/unit/testpyspark.py)), which are not included here.
   
   This depends on internal MyPy testing framework and is somewhat hard to maintain, but it is extremely useful, especially to control for negative cases. I think it might be worth portioning in the future.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690707762


   **[Test build #128529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128529/testReport)** for PR 29591 at commit [`91cd14b`](https://github.com/apache/spark/commit/91cd14bac570513c5acf870dfa8bc3aea19aaa6e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687929845


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697399954


   **[Test build #129033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129033/testReport)** for PR 29591 at commit [`fab00f1`](https://github.com/apache/spark/commit/fab00f1107beb301d093618d5c797799df768253).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692269822


   **[Test build #128667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128667/testReport)** for PR 29591 at commit [`6272d9d`](https://github.com/apache/spark/commit/6272d9db38ba7d23ba8fc441f359ee1abce34e32).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688498902


   **[Test build #128364 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128364/testReport)** for PR 29591 at commit [`d91b199`](https://github.com/apache/spark/commit/d91b1995edece9c5e65bd7e7d0c5b5d0a9a9349a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697176313


   **[Test build #129005 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129005/testReport)** for PR 29591 at commit [`b9ac4f8`](https://github.com/apache/spark/commit/b9ac4f89a95c4c86d1b43757cf9481f7c879a495).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689142218






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687901167


   **[Test build #128324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128324/testReport)** for PR 29591 at commit [`93fb711`](https://github.com/apache/spark/commit/93fb711e1621abcb944eb4b8b411d6f670cf7774).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687904553


   **[Test build #128325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128325/testReport)** for PR 29591 at commit [`27409f2`](https://github.com/apache/spark/commit/27409f29fa2c5f9515a1af55085f0d2a12113d01).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698136949


   Thank you @zero323 for leading type hint support in PySpark.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683782291


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128098/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690328592






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697147040


   **[Test build #129005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129005/testReport)** for PR 29591 at commit [`b9ac4f8`](https://github.com/apache/spark/commit/b9ac4f89a95c4c86d1b43757cf9481f7c879a495).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693148600






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690035351


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128491/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697184876


   **[Test build #129017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129017/testReport)** for PR 29591 at commit [`b9ac4f8`](https://github.com/apache/spark/commit/b9ac4f89a95c4c86d1b43757cf9481f7c879a495).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683916050






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692506530


   **[Test build #128698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128698/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688532441


   **[Test build #128364 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128364/testReport)** for PR 29591 at commit [`d91b199`](https://github.com/apache/spark/commit/d91b1995edece9c5e65bd7e7d0c5b5d0a9a9349a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r484138235



##########
File path: dev/tox.ini
##########
@@ -20,5 +20,16 @@ exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,wor
 
 [flake8]
 select = E901,E999,F821,F822,F823,F401,F405
-exclude = python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
+exclude = python/pyspark/cloudpickle/*.py,shared.py*,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*,python/out
 max-line-length = 100
+per-file-ignores =
+    python/pyspark/sql/tests/test_arrow.py: F405
+    python/pyspark/sql/tests/test_dataframe.py: F405
+    python/pyspark/sql/tests/test_pandas_udf_scalar.py: F405
+    python/pyspark/sql/tests/test_udf.py: F405
+    python/pyspark/testing/streamingutils.py: F821
+    python/pyspark/testing/mlutils.py: F821

Review comment:
       Looks like these are mostly because flake8 doesn't understand `# type: ignore[<code>]` - finds `<code>` is a proper instance to check if it's being imported or not(?). If that's the case, let's just use `# type: ignore`. I think this is the easiest way.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688498902


   **[Test build #128364 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128364/testReport)** for PR 29591 at commit [`d91b199`](https://github.com/apache/spark/commit/d91b1995edece9c5e65bd7e7d0c5b5d0a9a9349a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690025077


   **[Test build #128491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128491/testReport)** for PR 29591 at commit [`8608a12`](https://github.com/apache/spark/commit/8608a1251e372e9bf097efe084fe49b769f1599b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688505434


   **[Test build #128363 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128363/testReport)** for PR 29591 at commit [`d7d876a`](https://github.com/apache/spark/commit/d7d876a1f070bd672baf9ca4f1d26e0f04632c13).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688459710


   Jenkins, retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-684399710






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32681][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683585056






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r486075857



##########
File path: python/pyspark/ml/tests/test_algorithms.py
##########
@@ -333,7 +333,7 @@ def test_linear_regression_with_huber_loss(self):
     from pyspark.ml.tests.test_algorithms import *  # noqa: F401
 
     try:
-        import xmlrunner
+        import xmlrunner  # type: ignore[import]

Review comment:
       I think we should. Next to examples, these are the biggest chunks of code that show (most of the time) intended usage. This is a great sanity check.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692344825






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688085277


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128338/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693675033






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692228949


   **[Test build #128665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128665/testReport)** for PR 29591 at commit [`6272d9d`](https://github.com/apache/spark/commit/6272d9db38ba7d23ba8fc441f359ee1abce34e32).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r486428984



##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,8 +32,8 @@
 
 
 def dataframe_with_arrow_example(spark):
-    import numpy as np
-    import pandas as pd
+    import numpy as np  # type: ignore[import]

Review comment:
       Because, AFAIK, released versions are not typed ‒ doesn't seem there were any released versions since https://github.com/numpy/numpy/commit/11b95d15f10c2bc652ed19d5e27efa0384396cb8.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697147040


   **[Test build #129005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129005/testReport)** for PR 29591 at commit [`b9ac4f8`](https://github.com/apache/spark/commit/b9ac4f89a95c4c86d1b43757cf9481f7c879a495).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697107694






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688561042


   cc @BryanCutler, @viirya, @ueshin, @Fokko, @holdenk can you review this when you guys fine some time?
   
   FYI:
   
   We'll probably try to expose the type hints for user-facing APIs only but it will be done as a followup. It was discussed at https://github.com/apache/spark/pull/29591#discussion_r479940442.
   
   We'll need to document when/where/how to do with type hinting in contribution guide as https://github.com/apache/spark/blob/master/python/docs/source/development/contributing.rst


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-684399618


   **[Test build #128137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128137/testReport)** for PR 29591 at commit [`df12013`](https://github.com/apache/spark/commit/df12013595906747ca39d3a59b99def246348663).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692269822


   **[Test build #128667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128667/testReport)** for PR 29591 at commit [`6272d9d`](https://github.com/apache/spark/commit/6272d9db38ba7d23ba8fc441f359ee1abce34e32).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697051865






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697053470


   **[Test build #129001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129001/testReport)** for PR 29591 at commit [`9998d0a`](https://github.com/apache/spark/commit/9998d0adcaed46a48f795f078e32b5f2dbb9151c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683908392


   **[Test build #128109 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128109/testReport)** for PR 29591 at commit [`a936e03`](https://github.com/apache/spark/commit/a936e033bc640ce5862e84a3952b62570f81cac1).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687929852


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128325/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683895309


   **[Test build #128106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128106/testReport)** for PR 29591 at commit [`4b9000e`](https://github.com/apache/spark/commit/4b9000e1a77f290ff1c934ac53c0fcd2d1918901).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `class MultilayerPerceptronClassificationSummary(_ClassificationSummary): ...`
     * `class MultilayerPerceptronClassificationTrainingSummary(`
     * `class PythonException(CapturedException): ...`
     * `class InheritableThread(threading.Thread):`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689122473






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688506080






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687906242


   **[Test build #128326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128326/testReport)** for PR 29591 at commit [`b8d4876`](https://github.com/apache/spark/commit/b8d4876ecce95e9e62f6069871e0e835216cad65).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697265923


   **[Test build #129017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129017/testReport)** for PR 29591 at commit [`b9ac4f8`](https://github.com/apache/spark/commit/b9ac4f89a95c4c86d1b43757cf9481f7c879a495).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689036781






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693580781


   **[Test build #128777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128777/testReport)** for PR 29591 at commit [`9998d0a`](https://github.com/apache/spark/commit/9998d0adcaed46a48f795f078e32b5f2dbb9151c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697653808


   **[Test build #129033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129033/testReport)** for PR 29591 at commit [`fab00f1`](https://github.com/apache/spark/commit/fab00f1107beb301d093618d5c797799df768253).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688085266






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489359322



##########
File path: python/pyspark/sql/column.pyi
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import overload
+from typing import Any, Union
+
+from pyspark.sql._typing import LiteralType, DecimalLiteral, DateTimeLiteral
+from pyspark.sql.types import (  # noqa: F401
+    DataType,
+    StructField,
+    StructType,
+    IntegerType,
+    StringType,
+)
+from pyspark.sql.window import WindowSpec
+
+from py4j.java_gateway import JavaObject  # type: ignore[import]
+
+class Column:
+    def __init__(self, JavaObject) -> None: ...
+    def __neg__(self) -> Column: ...
+    def __add__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __sub__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __mul__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __div__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __truediv__(
+        self, other: Union[Column, LiteralType, DecimalLiteral]
+    ) -> Column: ...
+    def __mod__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __radd__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rsub__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rmul__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rdiv__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rtruediv__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __rmod__(self, other: Union[bool, int, float, DecimalLiteral]) -> Column: ...
+    def __pow__(self, other: Union[Column, LiteralType, DecimalLiteral]) -> Column: ...
+    def __rpow__(self, other: Union[LiteralType, DecimalLiteral]) -> Column: ...
+    def __eq__(self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]) -> Column: ...  # type: ignore[override]
+    def __ne__(self, other: Any) -> Column: ...  # type: ignore[override]
+    def __lt__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def __le__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def __ge__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def __gt__(
+        self, other: Union[Column, LiteralType, DateTimeLiteral, DecimalLiteral]
+    ) -> Column: ...
+    def eqNullSafe(
+        self, other: Union[Column, LiteralType, DecimalLiteral]
+    ) -> Column: ...
+    def __and__(self, other: Column) -> Column: ...
+    def __or__(self, other: Column) -> Column: ...
+    def __invert__(self) -> Column: ...
+    def __rand__(self, other: Column) -> Column: ...
+    def __ror__(self, other: Column) -> Column: ...
+    def __contains__(self, other: Any) -> Column: ...
+    def __getitem__(self, other: Any) -> Column: ...
+    def bitwiseOR(self, other: Union[Column, int]) -> Column: ...
+    def bitwiseAND(self, other: Union[Column, int]) -> Column: ...
+    def bitwiseXOR(self, other: Union[Column, int]) -> Column: ...
+    def getItem(self, key: Any) -> Column: ...
+    def getField(self, name: Any) -> Column: ...
+    def __getattr__(self, item: Any) -> Column: ...
+    def __iter__(self) -> None: ...
+    def rlike(self, item: str) -> Column: ...
+    def like(self, item: str) -> Column: ...
+    def startswith(self, item: Union[str, Column]) -> Column: ...
+    def endswith(self, item: Union[str, Column]) -> Column: ...
+    @overload
+    def substr(self, startPos: int, length: int) -> Column: ...
+    @overload
+    def substr(self, startPos: Column, length: Column) -> Column: ...
+    def __getslice__(self, startPos: int, length: int) -> Column: ...
+    def isin(self, *cols: Any) -> Column: ...
+    def asc(self) -> Column: ...
+    def asc_nulls_first(self) -> Column: ...
+    def asc_nulls_last(self) -> Column: ...
+    def desc(self) -> Column: ...
+    def desc_nulls_first(self) -> Column: ...
+    def desc_nulls_last(self) -> Column: ...
+    def isNull(self) -> Column: ...
+    def isNotNull(self) -> Column: ...
+    def alias(self, *alias: str, **kwargs: Any) -> Column: ...
+    def name(self, *alias: str) -> Column: ...
+    def cast(self, dataType: Union[DataType, str]) -> Column: ...
+    def astype(self, dataType: Union[DataType, str]) -> Column: ...
+    def between(self, lowerBound, upperBound) -> Column: ...
+    def when(self, condition: Column, value: Any) -> Column: ...
+    def otherwise(self, value: Any) -> Column: ...
+    def over(self, window: WindowSpec) -> Column: ...
+    def __nonzero__(self) -> None: ...
+    def __bool__(self) -> None: ...

Review comment:
       Sorry I merged one more PR https://github.com/apache/spark/pull/29699. It'll probably have to be updated.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698117104


   Merged to master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692506530


   **[Test build #128698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128698/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489360010



##########
File path: python/pyspark/sql/pandas/_typing/protocols/series.pyi
##########
@@ -0,0 +1,253 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This Protocol resuses core Pandas annotation.
+# Overall pipeline looks as follows
+# - Stubgen pandas.core.series
+# - Add Protocol as a base class
+# - Replace imports with Any
+
+import numpy as np  # type: ignore[import]
+from typing import Any, Callable, Hashable, IO, Optional
+from typing_extensions import Protocol
+
+groupby_generic = Any
+
+class SeriesLike(Protocol):

Review comment:
       This is kind of sad that we have to do pandas' series in PySpark. Can we avoid?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697983153


   **[Test build #129047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129047/testReport)** for PR 29591 at commit [`fab00f1`](https://github.com/apache/spark/commit/fab00f1107beb301d093618d5c797799df768253).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692315021






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693378666


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128761/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r484200753



##########
File path: dev/tox.ini
##########
@@ -20,5 +20,16 @@ exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,wor
 
 [flake8]
 select = E901,E999,F821,F822,F823,F401,F405
-exclude = python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
+exclude = python/pyspark/cloudpickle/*.py,shared.py*,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*,python/out
 max-line-length = 100
+per-file-ignores =
+    python/pyspark/sql/tests/test_arrow.py: F405
+    python/pyspark/sql/tests/test_dataframe.py: F405
+    python/pyspark/sql/tests/test_pandas_udf_scalar.py: F405
+    python/pyspark/sql/tests/test_udf.py: F405
+    python/pyspark/testing/streamingutils.py: F821
+    python/pyspark/testing/mlutils.py: F821

Review comment:
       Sounds reasonable, though I'd prefer to use mixed strategy:
   
   - Less specific `type: ignore` for non-pyi files.
   - Ignore selected `pyi`  files in flake8.
   
   This way we can get the most coverage.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689142218






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r487129652



##########
File path: dev/tox.ini
##########
@@ -20,5 +20,5 @@ exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,wor
 
 [flake8]
 select = E901,E999,F821,F822,F823,F401,F405
-exclude = python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
+exclude = python/pyspark/cloudpickle/*.py,shared.py*,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*,python/out,python/pyspark/sql/pandas/functions.pyi,python/pyspark/sql/column.pyi,python/pyspark/worker.pyi,python/pyspark/java_gateway.pyi

Review comment:
       > Personally, I would require mypy to be added in this PR As well. Otherwise we run the risk that the annotations might go out of sync quickly.
   
   In general I see your point, however I wanted to avoid duplicating your work from https://github.com/apache/spark/pull/29180 and we cannot setup the environment anyway.
   
   
   
   

##########
File path: dev/tox.ini
##########
@@ -20,5 +20,5 @@ exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,wor
 
 [flake8]
 select = E901,E999,F821,F822,F823,F401,F405
-exclude = python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
+exclude = python/pyspark/cloudpickle/*.py,shared.py*,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*,python/out,python/pyspark/sql/pandas/functions.pyi,python/pyspark/sql/column.pyi,python/pyspark/worker.pyi,python/pyspark/java_gateway.pyi

Review comment:
       > Personally, I would require mypy to be added in this PR As well. Otherwise we run the risk that the annotations might go out of sync quickly.
   
   In general I see your point, however I wanted to avoid duplicating your work from https://github.com/apache/spark/pull/29180 and we cannot setup the environment anyway.
   
   
   
   

##########
File path: dev/tox.ini
##########
@@ -20,5 +20,5 @@ exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,wor
 
 [flake8]
 select = E901,E999,F821,F822,F823,F401,F405
-exclude = python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
+exclude = python/pyspark/cloudpickle/*.py,shared.py*,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*,python/out,python/pyspark/sql/pandas/functions.pyi,python/pyspark/sql/column.pyi,python/pyspark/worker.pyi,python/pyspark/java_gateway.pyi

Review comment:
       > Personally, I would require mypy to be added in this PR As well. Otherwise we run the risk that the annotations might go out of sync quickly.
   
   In general I see your point, however I wanted to avoid duplicating your work from https://github.com/apache/spark/pull/29180 and we cannot setup the environment anyway.
   
   
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688481670






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687906423


   **[Test build #128326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128326/testReport)** for PR 29591 at commit [`b8d4876`](https://github.com/apache/spark/commit/b8d4876ecce95e9e62f6069871e0e835216cad65).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697051865






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693516144


   **[Test build #128775 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128775/testReport)** for PR 29591 at commit [`baa9c36`](https://github.com/apache/spark/commit/baa9c36df0fa2e333fce293ccf9d8ea2554efd89).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon edited a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688561042


   cc @BryanCutler, @viirya, @ueshin, @Fokko, @holdenk can you review this when you guys fine some time? I think it's important to have you guys' review here.
   
   FYI:
   
   We'll probably try to expose the type hints for user-facing APIs only but it will be done in a separate JIRA. It was discussed at https://github.com/apache/spark/pull/29591#discussion_r479940442.
   
   We'll need to document when/where/how to do with type hinting in contribution guide as https://github.com/apache/spark/blob/master/python/docs/source/development/contributing.rst in a separate JIRA.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683915709


   **[Test build #128111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128111/testReport)** for PR 29591 at commit [`0bc7183`](https://github.com/apache/spark/commit/0bc7183a26d8098c3df10096143946e7fd4fe5d4).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688085102






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692228949


   **[Test build #128665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128665/testReport)** for PR 29591 at commit [`6272d9d`](https://github.com/apache/spark/commit/6272d9db38ba7d23ba8fc441f359ee1abce34e32).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692270432






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r484188746



##########
File path: examples/src/main/python/ml/estimator_transformer_param_example.py
##########
@@ -54,7 +56,7 @@
     print(model1.extractParamMap())
 
     # We may alternatively specify parameters using a Python dictionary as a paramMap
-    paramMap = {lr.maxIter: 20}
+    paramMap: Dict[Param, Any] = {lr.maxIter: 20}

Review comment:
       I don't object to having the type hints in the examples but I guess we should type everything in examples if we want. We could discuss and do separately - we'd likely have to file multiple followup JIRAs under SPARK-32681.
   
   I was thinking it's good to port the PySpark type hints alone first with minimized diff to other codes in Spark in this JIRA/PR.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689200399






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688423801






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690328592






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683895320






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688532731






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690035346


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689040772






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697053470


   **[Test build #129001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129001/testReport)** for PR 29591 at commit [`9998d0a`](https://github.com/apache/spark/commit/9998d0adcaed46a48f795f078e32b5f2dbb9151c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697107706


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129001/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693517008






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683915728


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688071729






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698033089


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-684399679


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687901167


   **[Test build #128324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128324/testReport)** for PR 29591 at commit [`93fb711`](https://github.com/apache/spark/commit/93fb711e1621abcb944eb4b8b411d6f670cf7774).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698117104






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683895727






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697051612






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683894896


   **[Test build #128106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128106/testReport)** for PR 29591 at commit [`4b9000e`](https://github.com/apache/spark/commit/4b9000e1a77f290ff1c934ac53c0fcd2d1918901).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697053470






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697183521


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689200399






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683781786


   **[Test build #128098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128098/testReport)** for PR 29591 at commit [`e59b562`](https://github.com/apache/spark/commit/e59b5627503bdcf1ddca646369c1a7656ae08997).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683908419


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128109/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687901351






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687941936


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693216732


   **[Test build #128740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128740/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693593830






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692344825






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698134142


   Thanks everyone!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689040190


   **[Test build #128420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128420/testReport)** for PR 29591 at commit [`8e8c3b7`](https://github.com/apache/spark/commit/8e8c3b784a667f37956c18d6eb1bdf8e6bc4e17a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689040772






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683907533


   **[Test build #128109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128109/testReport)** for PR 29591 at commit [`a936e03`](https://github.com/apache/spark/commit/a936e033bc640ce5862e84a3952b62570f81cac1).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693147207


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693517008






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683782276






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693306467






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489356481



##########
File path: python/pyspark/py.typed
##########
@@ -0,0 +1 @@
+partial

Review comment:
       quick question: what is this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690489890


   **[Test build #128529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128529/testReport)** for PR 29591 at commit [`91cd14b`](https://github.com/apache/spark/commit/91cd14bac570513c5acf870dfa8bc3aea19aaa6e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693592665


   **[Test build #128775 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128775/testReport)** for PR 29591 at commit [`baa9c36`](https://github.com/apache/spark/commit/baa9c36df0fa2e333fce293ccf9d8ea2554efd89).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693378651






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693305678


   **[Test build #128761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128761/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693148274


   **[Test build #128740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128740/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683895334


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128106/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687906429






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697147463






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692494840






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688499254






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29591:
URL: https://github.com/apache/spark/pull/29591


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687941944


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128327/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683894896


   **[Test build #128106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128106/testReport)** for PR 29591 at commit [`4b9000e`](https://github.com/apache/spark/commit/4b9000e1a77f290ff1c934ac53c0fcd2d1918901).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489360897



##########
File path: python/pyspark/sql/types.pyi
##########
@@ -0,0 +1,220 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import overload
+from typing import Any, Callable, Dict, Iterator, List, Optional, Union, Tuple, TypeVar
+import datetime
+
+T = TypeVar("T")
+U = TypeVar("U")
+
+class DataType:
+    def __hash__(self) -> int: ...
+    def __eq__(self, other: Any) -> bool: ...
+    def __ne__(self, other: Any) -> bool: ...
+    @classmethod
+    def typeName(cls) -> str: ...
+    def simpleString(self) -> str: ...
+    def jsonValue(self) -> Union[str, Dict[str, Any]]: ...
+    def json(self) -> str: ...
+    def needConversion(self) -> bool: ...
+    def toInternal(self, obj: Any) -> Any: ...
+    def fromInternal(self, obj: Any) -> Any: ...
+
+class DataTypeSingleton(type):
+    def __call__(cls): ...
+
+class NullType(DataType):
+    __metaclass__: type

Review comment:
       I merged https://github.com/apache/spark/pull/29664 too




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689141458


   **[Test build #128425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128425/testReport)** for PR 29591 at commit [`8e8c3b7`](https://github.com/apache/spark/commit/8e8c3b784a667f37956c18d6eb1bdf8e6bc4e17a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698033089






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688532731






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698033096


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129047/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687941936


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] holdenk commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697823681


   LGTM pending passing both GHA and Jenkins.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688459710


   Jenkins, retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687904837






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688423212


   **[Test build #128361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128361/testReport)** for PR 29591 at commit [`d7d876a`](https://github.com/apache/spark/commit/d7d876a1f070bd672baf9ca4f1d26e0f04632c13).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687901440






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683908410






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683915738






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693148600






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693378157


   **[Test build #128761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128761/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489348192



##########
File path: python/pyspark/__init__.pyi
##########
@@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Callable, Optional, TypeVar
+
+from pyspark.accumulators import (  # noqa: F401
+    Accumulator as Accumulator,
+    AccumulatorParam as AccumulatorParam,

Review comment:
       quick question, why should we do `as` to the same import?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688071729






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688060012


   > @zero323, I usually prefer to don't block something by the env issue in Jenkins so such issue can be handled with enough time - @shaneknapp is sort of busy at this moment IIRC. We could work around for now, and file a separate JIRA for him about the dependency upgade.
   
   Agreed. I thought it is worth raising the question, as it seems like [we'll need some changes](https://github.com/apache/spark/pull/29180/files/eeabe659c2c06ccd82bc5474295fdc45993468f2..2af1250e02b6debdf31409f2a356a1aa3dbee7d9) to the environment anyway.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683782254


   **[Test build #128098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128098/testReport)** for PR 29591 at commit [`e59b562`](https://github.com/apache/spark/commit/e59b5627503bdcf1ddca646369c1a7656ae08997).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687904837






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683782427






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698032745


   **[Test build #129047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129047/testReport)** for PR 29591 at commit [`fab00f1`](https://github.com/apache/spark/commit/fab00f1107beb301d093618d5c797799df768253).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689122485


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128420/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690025530






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697657470






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693581392






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon edited a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-698117279


   @zero323 mind working on the below ones?
   
   - writing the guidelines in the doc
   - removing non-API type hints
   
   I think these two are pretty important followups to be done soon ..


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r486399441



##########
File path: python/pyspark/context.pyi
##########
@@ -0,0 +1,179 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Stubs for pyspark.context (Python 3.5)

Review comment:
       These can be safely removed (have been automatically added by stubgen  and in practice don't really reflect much more beyond the interpreter used at the moment).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693516144


   **[Test build #128775 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128775/testReport)** for PR 29591 at commit [`baa9c36`](https://github.com/apache/spark/commit/baa9c36df0fa2e333fce293ccf9d8ea2554efd89).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697185517






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489406537



##########
File path: python/pyspark/__init__.pyi
##########
@@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Callable, Optional, TypeVar
+
+from pyspark.accumulators import (  # noqa: F401
+    Accumulator as Accumulator,
+    AccumulatorParam as AccumulatorParam,

Review comment:
       In case of stub packages this allows us to skip annotations for certain modules ‒ if stub file is not found, type checker will go and inspect the module itself.
   
   In theory, if stub files reside within the same directory structure as the source, it shouldn't be necessary.  But since things are not always smooth, I'd prefer to keep this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687941944


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128327/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687916040


   **[Test build #128327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128327/testReport)** for PR 29591 at commit [`1aede7c`](https://github.com/apache/spark/commit/1aede7c7616fa045dae2bd9483e24ed48f089a02).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697053470


   **[Test build #129001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129001/testReport)** for PR 29591 at commit [`9998d0a`](https://github.com/apache/spark/commit/9998d0adcaed46a48f795f078e32b5f2dbb9151c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688481107


   **[Test build #128361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128361/testReport)** for PR 29591 at commit [`d7d876a`](https://github.com/apache/spark/commit/d7d876a1f070bd672baf9ca4f1d26e0f04632c13).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r489360010



##########
File path: python/pyspark/sql/pandas/_typing/protocols/series.pyi
##########
@@ -0,0 +1,253 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This Protocol resuses core Pandas annotation.
+# Overall pipeline looks as follows
+# - Stubgen pandas.core.series
+# - Add Protocol as a base class
+# - Replace imports with Any
+
+import numpy as np  # type: ignore[import]
+from typing import Any, Callable, Hashable, IO, Optional
+from typing_extensions import Protocol
+
+groupby_generic = Any
+
+class SeriesLike(Protocol):

Review comment:
       This is kind of said that we have to do pandas' series in PySpark. Can we avoid?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690025530






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697176734


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129005/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687904553


   **[Test build #128325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128325/testReport)** for PR 29591 at commit [`27409f2`](https://github.com/apache/spark/commit/27409f29fa2c5f9515a1af55085f0d2a12113d01).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693593830


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon edited a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-688561042






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683895320






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-693378651


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689121636


   **[Test build #128420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128420/testReport)** for PR 29591 at commit [`8e8c3b7`](https://github.com/apache/spark/commit/8e8c3b784a667f37956c18d6eb1bdf8e6bc4e17a).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692512299


   **[Test build #128698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128698/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-689140799


   Jenkins, retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-697176725


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687929845






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-692267888


   Jenkins, retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29591:
URL: https://github.com/apache/spark/pull/29591#discussion_r480566498



##########
File path: python/pyspark/cloudpickle.pyi
##########
@@ -0,0 +1,86 @@
+#

Review comment:
       I remember mypy just ignores and doesn't require `type: ignore` when the type hints don't exist. That's what I experienced in another project that uses type hints but maybe I am missing some cases here.
   
   In any event, let's port first in whichever way best to you, and then we can try to remove non-API type hints separately as a followup. Please go ahead in whichever way easiest to you as the initial port!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32681][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-683585056


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687901349


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-687906429






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29591:
URL: https://github.com/apache/spark/pull/29591#issuecomment-690709554






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org