You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/13 10:54:00 UTC

[GitHub] [spark] zero323 opened a new pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

zero323 opened a new pull request #34273:
URL: https://github.com/apache/spark/pull/34273


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'core/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   This PR:
   
   - Adds `mypy_examples_test` and `mypy_annotation_test` and refactors `mypy_test` in `def/lint-python` to enable testing PySpark examples with mypy.
   - Adjusts examples for `ml`, `sql`, `streaming` and core to address detected type checking issues.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   The goal of this PR is to improve test coverage of type hints.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   In general, no. 
   
   The only change, directly visible to the end user, are small adjustments to the example scripts.
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   Existing tests with additions listed above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-953816059


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49185/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-953873940


   **[Test build #144715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144715/testReport)** for PR 34273 at commit [`60af634`](https://github.com/apache/spark/commit/60af634e7538c14a56b3ef10cf1222922f13acca).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `public class JavaModuleOptions `
     * `class SparkUpgradeException(CapturedException):`
     * `public class ExpressionImplUtils `
     * `public class NumericHistogram `
     * `   * The Coord class defines a histogram bin, which is just an (x,y) pair.`
     * `  public static class Coord implements Comparable `
     * `trait FunctionExpressionBuilder `
     * `case class HistogramNumeric(`
     * `trait ExtractValue extends Expression with NullIntolerant `
     * `case class AesEncrypt(input: Expression, key: Expression, child: Expression)`
     * `case class AesDecrypt(input: Expression, key: Expression, child: Expression)`
     * `case class CreateView(`
     * `case class CreateIndex(`
     * `case class ShowCatalogsCommand(pattern: Option[String]) extends LeafRunnableCommand `
     * `case class CreateIndexExec(`
     * `class SparkUDFExpressionBuilder extends FunctionExpressionBuilder `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954876749


   **[Test build #144763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144763/testReport)** for PR 34273 at commit [`43df3a1`](https://github.com/apache/spark/commit/43df3a1d2aede3eb1b9a294a6fe80b17a3ac54d0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r745562516



##########
File path: examples/src/main/python/avro_inputformat.py
##########
@@ -75,7 +75,7 @@
         schema_rdd = sc.textFile(sys.argv[2], 1).collect()
         conf = {"avro.schema.input.key": reduce(lambda x, y: x + y, schema_rdd)}
 
-    avro_rdd = sc.newAPIHadoopFile(
+    avro_rdd = sc.newAPIHadoopFile(  # type: ignore[var-annotated]

Review comment:
       Seems OK to me




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963794142


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145013/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964622325


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49511/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974324692


   **[Test build #145461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145461/testReport)** for PR 34273 at commit [`dc2742d`](https://github.com/apache/spark/commit/dc2742d62fcac176cf5bd09ff8f76511844cc5d1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r727943307



##########
File path: examples/src/main/python/ml/__init__,py
##########
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#

Review comment:
       `__init__.py` files where added to examples directories, so mypy treats them as valid Python packages. Otherwise, mypy would collect module names alone and fail due duplicate names.
   
   Alternatively, we can run tests in loop for each directory, but this seems cleaner.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954902684


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49232/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974490666


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145461/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963735054


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49486/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963794142


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145013/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963844452


   nit: `def/lint-python` -> `dev/lint-python` in PR description


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964092969


   **[Test build #145025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145025/testReport)** for PR 34273 at commit [`660af11`](https://github.com/apache/spark/commit/660af110a12739e0b2c9e1d7ce1b67b14a021d25).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964231350


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145025/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964299108


   **[Test build #145028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145028/testReport)** for PR 34273 at commit [`a498229`](https://github.com/apache/spark/commit/a498229904c2601ba1c6c16a2d71faf707b72673).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964347253


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49505/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964399958


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49505/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964540334


   **[Test build #145040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145040/testReport)** for PR 34273 at commit [`7d5b512`](https://github.com/apache/spark/commit/7d5b512df75858263f740eaacc6a188670d35c9b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942349605


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144202/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942293007


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48681/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r727940837



##########
File path: dev/lint-python
##########
@@ -124,39 +124,67 @@ function pycodestyle_test {
     fi
 }
 
-function mypy_test {
+
+function mypy_annotation_test {
     local MYPY_REPORT=
     local MYPY_STATUS=
 
-    if ! hash "$MYPY_BUILD" 2> /dev/null; then
-        echo "The $MYPY_BUILD command was not found. Skipping for now."
-        return
+    echo "starting mypy annotations test..."
+    MYPY_REPORT=$( ($MYPY_BUILD --config-file python/mypy.ini python/pyspark) 2>&1)
+    MYPY_STATUS=$?
+
+    if [ "$MYPY_STATUS" -ne 0 ]; then
+        echo "annotations failed mypy checks:"
+        echo "$MYPY_REPORT"
+        echo "$MYPY_STATUS"
+        exit "$MYPY_STATUS"
+    else
+        echo "annotations passed mypy checks."
+        echo
     fi
+}
 
-    _MYPY_VERSION=($($MYPY_BUILD --version))
-    MYPY_VERSION="${_MYPY_VERSION[1]}"
-    EXPECTED_MYPY="$(satisfies_min_version $MYPY_VERSION $MINIMUM_MYPY)"
 
-    if [[ "$EXPECTED_MYPY" == "False" ]]; then
-        echo "The minimum mypy version needs to be $MINIMUM_MYPY. Your current version is $MYPY_VERSION. Skipping for now."
-        return
-    fi
+function mypy_examples_test {
+    local MYPY_REPORT=
+    local MYPY_STATUS=
+
+    echo "starting mypy examples test..."
+
+    MYPY_REPORT=$( (MYPYPATH=python $MYPY_BUILD \
+      --allow-untyped-defs \
+      --config-file python/mypy.ini \
+      --exclude "mllib/*" \

Review comment:
       At the moment, `mllib` examples yield a number of errors, which require adjustments of the actual annotations. I'll resolve this separately drop exclude later.

##########
File path: examples/src/main/python/als.py
##########
@@ -87,15 +87,15 @@ def update(i, mat, ratings):
     for i in range(ITERATIONS):
         ms = sc.parallelize(range(M), partitions) \
                .map(lambda x: update(x, usb.value, Rb.value)) \
-               .collect()
+               .collect()  # type: ignore[assignment]
         # collect() returns a list, so array ends up being
         # a 3-d array, we take the first 2 dims for the matrix
-        ms = matrix(np.array(ms)[:, :, 0])
+        ms = matrix(np.array(ms)[:, :, 0])  # type: ignore[assignment]
         msb = sc.broadcast(ms)
 
         us = sc.parallelize(range(U), partitions) \
                .map(lambda x: update(x, msb.value, Rb.value.T)) \
-               .collect()
+               .collect()  # type: ignore[assignment]

Review comment:
       These ignores are required, because we keep overwriting the same names.

##########
File path: examples/src/main/python/ml/__init__,py
##########
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#

Review comment:
       `__init__.py` files where added to examples directories, so mypy treats them as valid Python packages. Otherwise, mypy would collect module names alone and fail due duplicate names.
   
   Alternatively, we can run tests in loop for each directory, but this seems cleaner.

##########
File path: examples/src/main/python/ml/chi_square_test_example.py
##########
@@ -42,9 +42,9 @@
     df = spark.createDataFrame(data, ["label", "features"])
 
     r = ChiSquareTest.test(df, "features", "label").head()
-    print("pValues: " + str(r.pValues))
-    print("degreesOfFreedom: " + str(r.degreesOfFreedom))
-    print("statistics: " + str(r.statistics))
+    print("pValues: " + str(r.pValues))  # type: ignore[union-attr]
+    print("degreesOfFreedom: " + str(r.degreesOfFreedom))  # type: ignore[union-attr]
+    print("statistics: " + str(r.statistics))   # type: ignore[union-attr]

Review comment:
       This is required, because `DataFrame.head` has rather unfriendly signature.

##########
File path: examples/src/main/python/sql/streaming/structured_network_wordcount_windowed.py
##########
@@ -87,7 +87,7 @@
     windowedCounts = words.groupBy(
         window(words.timestamp, windowDuration, slideDuration),
         words.word
-    ).count().orderBy('window')
+    ).count().orderBy('window')  # type: ignore[arg-type]

Review comment:
       During tests with `mypy 0.920+dev.5f7eb8161bdeefcc526705a48b03115aa1e83a14` I experienced non-deterministic  problems resulting in
   
   ```
   examples/src/main/python/sql/streaming/structured_network_wordcount_windowed.py:90: error: Argument 1 has incompatible type "str"; expected "DataFrame"  [arg-type]
   Found 1 error in 1 file (checked 8 source files)
   ```
   
   It seems like it happens because `orderBy` is created as an alias for `sort`
   
   
   https://github.com/apache/spark/blob/9290924054f8e4e24afe3c6796465c4cf60e3a2b/python/pyspark/sql/dataframe.py#L1649
   
   and in certain conditions (?) mypy treats it as static or classmethod (could be wrong).
   
   I couldn't reproduce it on synthetic examples, though.  

##########
File path: examples/src/main/python/streaming/network_wordjoinsentiments.py
##########
@@ -50,10 +51,17 @@ def print_happiest_words(rdd):
     sc = SparkContext(appName="PythonStreamingNetworkWordJoinSentiments")
     ssc = StreamingContext(sc, 5)
 
+    def line_to_tuple(line: str) -> Tuple[str, str]:
+        try:
+            k, v = line.split(" ")
+            return k, v
+        except ValueError:
+            return "", ""
+
     # Read in the word-sentiment list and create a static RDD from it
     word_sentiments_file_path = "data/streaming/AFINN-111.txt"
     word_sentiments = ssc.sparkContext.textFile(word_sentiments_file_path) \
-        .map(lambda line: tuple(line.split("\t")))
+        .map(line_to_tuple)

Review comment:
       I am really not sure how to handle this.
   
   ```python
   .map(lambda line: tuple(line.split("\t")))
   ```
   returns `RDD[Tuple[str, ...]]`, which will cause cascade of errors later.
   
   We could exclude this file from type check, add casts, rewrite code to:
   
   ```python
   .map(lambda line: line.split("\t"))
   .map(lambda xs: (xs[0], xs[1]))
   ````
   
   but this felt like the least intrusive and the cleanest approach.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964232479


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49499/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964092969


   **[Test build #145025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145025/testReport)** for PR 34273 at commit [`660af11`](https://github.com/apache/spark/commit/660af110a12739e0b2c9e1d7ce1b67b14a021d25).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r745237660



##########
File path: examples/src/main/python/avro_inputformat.py
##########
@@ -75,7 +75,7 @@
         schema_rdd = sc.textFile(sys.argv[2], 1).collect()
         conf = {"avro.schema.input.key": reduce(lambda x, y: x + y, schema_rdd)}
 
-    avro_rdd = sc.newAPIHadoopFile(
+    avro_rdd = sc.newAPIHadoopFile(  # type: ignore[var-annotated]

Review comment:
       How about just not running checks on examples if they're to be treated differently? Then there aren't even ignore comments in them




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-950673641


   Are you OK with getting is merged @HyukjinKwon?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974490666


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145461/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964267115


   **[Test build #145027 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145027/testReport)** for PR 34273 at commit [`b80f2df`](https://github.com/apache/spark/commit/b80f2dfa3a6dd1a40f3b651e74cb95360f60db76).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964300937


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145028/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964184050


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49497/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964176181


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49499/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r745223322



##########
File path: examples/src/main/python/avro_inputformat.py
##########
@@ -75,7 +75,7 @@
         schema_rdd = sc.textFile(sys.argv[2], 1).collect()
         conf = {"avro.schema.input.key": reduce(lambda x, y: x + y, schema_rdd)}
 
-    avro_rdd = sc.newAPIHadoopFile(
+    avro_rdd = sc.newAPIHadoopFile(  # type: ignore[var-annotated]

Review comment:
       It's good that mypy tests can pass on examples but I wonder if we can remove such `ignore`. Examples are supposed to be verbose and close to the standard. cc @srowen @holdenk FYI in case you have some opinions on this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963716366


   **[Test build #145013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145013/testReport)** for PR 34273 at commit [`735b83e`](https://github.com/apache/spark/commit/735b83eba04708bc3f701f7ee614ea253abc5c4f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964300937


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145028/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942193082


   **[Test build #144202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144202/testReport)** for PR 34273 at commit [`846cdc0`](https://github.com/apache/spark/commit/846cdc0e200d9ab8cc009b0d3c39122116f9d6fd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] codecov-commenter edited a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942196328


   # [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#34273](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9290924) into [master](https://codecov.io/gh/apache/spark/commit/1020c6d8e51d3dc135cc012645950e9a8eea8376?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1020c6d) will **decrease** coverage by `0.04%`.
   > The diff coverage is `88.03%`.
   
   > :exclamation: Current head 9290924 differs from pull request most recent head 846cdc0. Consider uploading reports for the commit 846cdc0 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/spark/pull/34273/graphs/tree.svg?width=650&height=150&src=pr&token=R9pHLWgWi8&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #34273      +/-   ##
   ==========================================
   - Coverage   90.19%   90.14%   -0.05%     
   ==========================================
     Files         288      288              
     Lines       61391    61650     +259     
     Branches     9021     9032      +11     
   ==========================================
   + Hits        55372    55576     +204     
   - Misses       4670     4720      +50     
   - Partials     1349     1354       +5     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `90.12% <88.03%> (-0.05%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [python/pyspark/pandas/indexing.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4aW5nLnB5) | `88.88% <50.00%> (ø)` | |
   | [python/pyspark/sql/observation.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL29ic2VydmF0aW9uLnB5) | `91.30% <50.00%> (-4.16%)` | :arrow_down: |
   | [python/pyspark/pandas/tests/test\_series.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL3Rlc3RzL3Rlc3Rfc2VyaWVzLnB5) | `92.53% <81.81%> (-0.20%)` | :arrow_down: |
   | [python/pyspark/sql/dataframe.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2RhdGFmcmFtZS5weQ==) | `81.50% <82.79%> (-2.87%)` | :arrow_down: |
   | [python/pyspark/pandas/namespace.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL25hbWVzcGFjZS5weQ==) | `77.20% <83.33%> (+0.03%)` | :arrow_up: |
   | [python/pyspark/sql/utils.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL3V0aWxzLnB5) | `85.96% <85.00%> (-1.14%)` | :arrow_down: |
   | [python/pyspark/pandas/indexes/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4ZXMvYmFzZS5weQ==) | `96.64% <85.71%> (+0.01%)` | :arrow_up: |
   | [python/pyspark/sql/column.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2NvbHVtbi5weQ==) | `90.07% <92.00%> (-1.56%)` | :arrow_down: |
   | [python/pyspark/pandas/accessors.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2FjY2Vzc29ycy5weQ==) | `92.01% <100.00%> (ø)` | |
   | [python/pyspark/pandas/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2Jhc2UucHk=) | `94.11% <100.00%> (+0.03%)` | :arrow_up: |
   | ... and [25 more](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f678c75...846cdc0](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942193082






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] codecov-commenter edited a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942196328


   # [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#34273](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9290924) into [master](https://codecov.io/gh/apache/spark/commit/1020c6d8e51d3dc135cc012645950e9a8eea8376?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1020c6d) will **decrease** coverage by `0.04%`.
   > The diff coverage is `88.03%`.
   
   > :exclamation: Current head 9290924 differs from pull request most recent head 846cdc0. Consider uploading reports for the commit 846cdc0 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/spark/pull/34273/graphs/tree.svg?width=650&height=150&src=pr&token=R9pHLWgWi8&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #34273      +/-   ##
   ==========================================
   - Coverage   90.19%   90.14%   -0.05%     
   ==========================================
     Files         288      288              
     Lines       61391    61650     +259     
     Branches     9021     9032      +11     
   ==========================================
   + Hits        55372    55576     +204     
   - Misses       4670     4720      +50     
   - Partials     1349     1354       +5     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `90.12% <88.03%> (-0.05%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [python/pyspark/pandas/indexing.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4aW5nLnB5) | `88.88% <50.00%> (ø)` | |
   | [python/pyspark/sql/observation.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL29ic2VydmF0aW9uLnB5) | `91.30% <50.00%> (-4.16%)` | :arrow_down: |
   | [python/pyspark/pandas/tests/test\_series.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL3Rlc3RzL3Rlc3Rfc2VyaWVzLnB5) | `92.53% <81.81%> (-0.20%)` | :arrow_down: |
   | [python/pyspark/sql/dataframe.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2RhdGFmcmFtZS5weQ==) | `81.50% <82.79%> (-2.87%)` | :arrow_down: |
   | [python/pyspark/pandas/namespace.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL25hbWVzcGFjZS5weQ==) | `77.20% <83.33%> (+0.03%)` | :arrow_up: |
   | [python/pyspark/sql/utils.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL3V0aWxzLnB5) | `85.96% <85.00%> (-1.14%)` | :arrow_down: |
   | [python/pyspark/pandas/indexes/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4ZXMvYmFzZS5weQ==) | `96.64% <85.71%> (+0.01%)` | :arrow_up: |
   | [python/pyspark/sql/column.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2NvbHVtbi5weQ==) | `90.07% <92.00%> (-1.56%)` | :arrow_down: |
   | [python/pyspark/pandas/accessors.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2FjY2Vzc29ycy5weQ==) | `92.01% <100.00%> (ø)` | |
   | [python/pyspark/pandas/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2Jhc2UucHk=) | `94.11% <100.00%> (+0.03%)` | :arrow_up: |
   | ... and [25 more](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f678c75...846cdc0](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-953852812


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49185/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974324692


   **[Test build #145461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145461/testReport)** for PR 34273 at commit [`dc2742d`](https://github.com/apache/spark/commit/dc2742d62fcac176cf5bd09ff8f76511844cc5d1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974361746


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49933/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974429510


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49933/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954942168


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49232/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974448048


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49933/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964131124


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49497/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964289235


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145027/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964614892


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49511/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963768047


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49486/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942293007






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964184050






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964414738


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145033/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964576121


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49511/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974488204


   **[Test build #145461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145461/testReport)** for PR 34273 at commit [`dc2742d`](https://github.com/apache/spark/commit/dc2742d62fcac176cf5bd09ff8f76511844cc5d1).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `class ProbabilisticClassifier(Classifier, _ProbabilisticClassifierParams, metaclass=ABCMeta):`
     * `class ProbabilisticClassificationModel(`
     * `class _JavaProbabilisticClassifier(ProbabilisticClassifier, _JavaClassifier, metaclass=ABCMeta):`
     * `class _JavaProbabilisticClassificationModel(`
     * `class _LinearSVCParams(`
     * `class LinearSVCModel(`
     * `class _LogisticRegressionParams(`
     * `class LogisticRegression(`
     * `class LogisticRegressionModel(`
     * `class BinaryLogisticRegressionSummary(_BinaryClassificationSummary, LogisticRegressionSummary):`
     * `class BinaryLogisticRegressionTrainingSummary(`
     * `class DecisionTreeClassifier(`
     * `class DecisionTreeClassificationModel(`
     * `class RandomForestClassifier(`
     * `class RandomForestClassificationModel(`
     * `class RandomForestClassificationTrainingSummary(`
     * `class BinaryRandomForestClassificationTrainingSummary(`
     * `class GBTClassifier(`
     * `class GBTClassificationModel(`
     * `class NaiveBayes(`
     * `class NaiveBayesModel(`
     * `class _MultilayerPerceptronParams(`
     * `class MultilayerPerceptronClassifier(`
     * `class MultilayerPerceptronClassificationModel(`
     * `class MultilayerPerceptronClassificationTrainingSummary(`
     * `class FMClassifier(`
     * `class FMClassificationModel(`
     * `class _GaussianMixtureParams(`
     * `class GaussianMixtureModel(`
     * `class _KMeansParams(`
     * `class KMeansModel(`
     * `class _BisectingKMeansParams(`
     * `class BisectingKMeansModel(`
     * `class PowerIterationClustering(`
     * `class BinaryClassificationEvaluator(`
     * `class RegressionEvaluator(`
     * `class MulticlassClassificationEvaluator(`
     * `class MultilabelClassificationEvaluator(`
     * `class ClusteringEvaluator(`
     * `class RankingEvaluator(`
     * `class Binarizer(`
     * `class BucketedRandomProjectionLSH(`
     * `class BucketedRandomProjectionLSHModel(`
     * `class Bucketizer(`
     * `class ElementwiseProduct(`
     * `class FeatureHasher(`
     * `class HashingTF(`
     * `class _OneHotEncoderParams(`
     * `class PolynomialExpansion(`
     * `class QuantileDiscretizer(`
     * `class _StringIndexerParams(`
     * `class StopWordsRemover(`
     * `class VectorAssembler(`
     * `class VectorSizeHint(`
     * `class VarianceThresholdSelector(`
     * `class VarianceThresholdSelectorModel(`
     * `class UnivariateFeatureSelector(`
     * `class UnivariateFeatureSelectorModel(`
     * `class _LinearRegressionParams(`
     * `class LinearRegressionModel(`
     * `class IsotonicRegression(`
     * `class IsotonicRegressionModel(JavaModel, _IsotonicRegressionParams, JavaMLWritable, JavaMLReadable):`
     * `class DecisionTreeRegressor(`
     * `class RandomForestRegressor(`
     * `class _AFTSurvivalRegressionParams(`
     * `class AFTSurvivalRegression(`
     * `class AFTSurvivalRegressionModel(`
     * `class _GeneralizedLinearRegressionParams(`
     * `class GeneralizedLinearRegression(`
     * `class GeneralizedLinearRegressionModel(`
     * `class _FactorizationMachinesParams(`
     * `class FMRegressionModel(`
     * `class CrossValidator(`
     * `class TrainValidationSplit(`
     * `                + \"class name `
     * `class MultivariateGaussian(NamedTuple):`
     * `class PandasAPIOnSparkAdviceWarning(Warning):`
     * `class ArrowStreamUDFSerializer(ArrowStreamSerializer):`
     * `class DayTimeIntervalType(AtomicType):`
     * `class DayTimeIntervalTypeConverter(object):`
     * `case class ExpressionStats(expr: Expression)(var useCount: Int)`
     * `case class PythonMapInArrow(`
     * `case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements)`
     * `trait MapInBatchExec extends UnaryExecNode `
     * `case class PythonMapInArrowExec(`
     * `  // When this is enabled, this class does additional lookup on write operations (put/delete) to`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963754665


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49486/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963768047


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49486/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942187822


   cc @HyukjinKwon @itholic @ueshin 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942238495


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48681/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942187822


   cc @HyukjinKwon @itholic @ueshin 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r727943790



##########
File path: examples/src/main/python/ml/chi_square_test_example.py
##########
@@ -42,9 +42,9 @@
     df = spark.createDataFrame(data, ["label", "features"])
 
     r = ChiSquareTest.test(df, "features", "label").head()
-    print("pValues: " + str(r.pValues))
-    print("degreesOfFreedom: " + str(r.degreesOfFreedom))
-    print("statistics: " + str(r.statistics))
+    print("pValues: " + str(r.pValues))  # type: ignore[union-attr]
+    print("degreesOfFreedom: " + str(r.degreesOfFreedom))  # type: ignore[union-attr]
+    print("statistics: " + str(r.statistics))   # type: ignore[union-attr]

Review comment:
       This is required, because `DataFrame.head` has rather unfriendly signature.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942275914


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48681/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964399990


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49505/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964540334


   **[Test build #145040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145040/testReport)** for PR 34273 at commit [`7d5b512`](https://github.com/apache/spark/commit/7d5b512df75858263f740eaacc6a188670d35c9b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964653060


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145040/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964184000


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49497/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964232419


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49499/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-953887400






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-953887400






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974448048


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49933/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r728550246



##########
File path: examples/src/main/python/als.py
##########
@@ -87,15 +87,15 @@ def update(i, mat, ratings):
     for i in range(ITERATIONS):
         ms = sc.parallelize(range(M), partitions) \
                .map(lambda x: update(x, usb.value, Rb.value)) \
-               .collect()
+               .collect()  # type: ignore[assignment]
         # collect() returns a list, so array ends up being
         # a 3-d array, we take the first 2 dims for the matrix
-        ms = matrix(np.array(ms)[:, :, 0])
+        ms = matrix(np.array(ms)[:, :, 0])  # type: ignore[assignment]
         msb = sc.broadcast(ms)
 
         us = sc.parallelize(range(U), partitions) \
                .map(lambda x: update(x, msb.value, Rb.value.T)) \
-               .collect()
+               .collect()  # type: ignore[assignment]

Review comment:
       hmmm .. I wonder if we can avoid these `ignore` in the examples .. because the examples should better have a clean standard stuff .. do you plan to type these examples after this PR? then I am good w/ this temporarily.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r727946428



##########
File path: examples/src/main/python/sql/streaming/structured_network_wordcount_windowed.py
##########
@@ -87,7 +87,7 @@
     windowedCounts = words.groupBy(
         window(words.timestamp, windowDuration, slideDuration),
         words.word
-    ).count().orderBy('window')
+    ).count().orderBy('window')  # type: ignore[arg-type]

Review comment:
       During tests with `mypy 0.920+dev.5f7eb8161bdeefcc526705a48b03115aa1e83a14` I experienced non-deterministic  problems resulting in
   
   ```
   examples/src/main/python/sql/streaming/structured_network_wordcount_windowed.py:90: error: Argument 1 has incompatible type "str"; expected "DataFrame"  [arg-type]
   Found 1 error in 1 file (checked 8 source files)
   ```
   
   It seems like it happens because `orderBy` is created as an alias for `sort`
   
   
   https://github.com/apache/spark/blob/9290924054f8e4e24afe3c6796465c4cf60e3a2b/python/pyspark/sql/dataframe.py#L1649
   
   and in certain conditions (?) mypy treats it as static or classmethod (could be wrong).
   
   I couldn't reproduce it on synthetic examples, though.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] codecov-commenter commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942196328


   # [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#34273](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9290924) into [master](https://codecov.io/gh/apache/spark/commit/1020c6d8e51d3dc135cc012645950e9a8eea8376?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1020c6d) will **decrease** coverage by `0.04%`.
   > The diff coverage is `88.03%`.
   
   > :exclamation: Current head 9290924 differs from pull request most recent head 846cdc0. Consider uploading reports for the commit 846cdc0 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/spark/pull/34273/graphs/tree.svg?width=650&height=150&src=pr&token=R9pHLWgWi8&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #34273      +/-   ##
   ==========================================
   - Coverage   90.19%   90.14%   -0.05%     
   ==========================================
     Files         288      288              
     Lines       61391    61650     +259     
     Branches     9021     9032      +11     
   ==========================================
   + Hits        55372    55576     +204     
   - Misses       4670     4720      +50     
   - Partials     1349     1354       +5     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `90.12% <88.03%> (-0.05%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [python/pyspark/pandas/indexing.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4aW5nLnB5) | `88.88% <50.00%> (ø)` | |
   | [python/pyspark/sql/observation.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL29ic2VydmF0aW9uLnB5) | `91.30% <50.00%> (-4.16%)` | :arrow_down: |
   | [python/pyspark/pandas/tests/test\_series.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL3Rlc3RzL3Rlc3Rfc2VyaWVzLnB5) | `92.53% <81.81%> (-0.20%)` | :arrow_down: |
   | [python/pyspark/sql/dataframe.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2RhdGFmcmFtZS5weQ==) | `81.50% <82.79%> (-2.87%)` | :arrow_down: |
   | [python/pyspark/pandas/namespace.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL25hbWVzcGFjZS5weQ==) | `77.20% <83.33%> (+0.03%)` | :arrow_up: |
   | [python/pyspark/sql/utils.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL3V0aWxzLnB5) | `85.96% <85.00%> (-1.14%)` | :arrow_down: |
   | [python/pyspark/pandas/indexes/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4ZXMvYmFzZS5weQ==) | `96.64% <85.71%> (+0.01%)` | :arrow_up: |
   | [python/pyspark/sql/column.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2NvbHVtbi5weQ==) | `90.07% <92.00%> (-1.56%)` | :arrow_down: |
   | [python/pyspark/pandas/accessors.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2FjY2Vzc29ycy5weQ==) | `92.01% <100.00%> (ø)` | |
   | [python/pyspark/pandas/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2Jhc2UucHk=) | `94.11% <100.00%> (+0.03%)` | :arrow_up: |
   | ... and [25 more](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f678c75...846cdc0](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954979154


   **[Test build #144763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144763/testReport)** for PR 34273 at commit [`43df3a1`](https://github.com/apache/spark/commit/43df3a1d2aede3eb1b9a294a6fe80b17a3ac54d0).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942293007






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942293007


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48681/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r728758011



##########
File path: examples/src/main/python/als.py
##########
@@ -87,15 +87,15 @@ def update(i, mat, ratings):
     for i in range(ITERATIONS):
         ms = sc.parallelize(range(M), partitions) \
                .map(lambda x: update(x, usb.value, Rb.value)) \
-               .collect()
+               .collect()  # type: ignore[assignment]
         # collect() returns a list, so array ends up being
         # a 3-d array, we take the first 2 dims for the matrix
-        ms = matrix(np.array(ms)[:, :, 0])
+        ms = matrix(np.array(ms)[:, :, 0])  # type: ignore[assignment]
         msb = sc.broadcast(ms)
 
         us = sc.parallelize(range(U), partitions) \
                .map(lambda x: update(x, msb.value, Rb.value.T)) \
-               .collect()
+               .collect()  # type: ignore[assignment]

Review comment:
       Yes, we can (or at least we can reduce the number `ignores` to minimum). The question is, how to do it:
   
   - We can rewrite examples [as I've done in other place](https://github.com/apache/spark/pull/34273/files/846cdc0e200d9ab8cc009b0d3c39122116f9d6fd#diff-da9c07a08d3b20939b5456cd79104717d94ad6909074213f56109a6d92467a2b)
   - We can use less strict checker default (not fond of that)
   
   If we're going to rewrite (here it is primarily a matter of using different names I guess), how far should we go? Type everything? Type only when types are necessary to make mypy happy? If the latter one, we'd have to leak typing import into rendered examples. WDYT?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-953746263


   **[Test build #144715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144715/testReport)** for PR 34273 at commit [`60af634`](https://github.com/apache/spark/commit/60af634e7538c14a56b3ef10cf1222922f13acca).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-953746263


   **[Test build #144715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144715/testReport)** for PR 34273 at commit [`60af634`](https://github.com/apache/spark/commit/60af634e7538c14a56b3ef10cf1222922f13acca).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954990037


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144763/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963716366


   **[Test build #145013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145013/testReport)** for PR 34273 at commit [`735b83e`](https://github.com/apache/spark/commit/735b83eba04708bc3f701f7ee614ea253abc5c4f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964291006


   **[Test build #145033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145033/testReport)** for PR 34273 at commit [`7d5b512`](https://github.com/apache/spark/commit/7d5b512df75858263f740eaacc6a188670d35c9b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964133583


   **[Test build #145028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145028/testReport)** for PR 34273 at commit [`a498229`](https://github.com/apache/spark/commit/a498229904c2601ba1c6c16a2d71faf707b72673).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964653060


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145040/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964399990


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49505/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r745223322



##########
File path: examples/src/main/python/avro_inputformat.py
##########
@@ -75,7 +75,7 @@
         schema_rdd = sc.textFile(sys.argv[2], 1).collect()
         conf = {"avro.schema.input.key": reduce(lambda x, y: x + y, schema_rdd)}
 
-    avro_rdd = sc.newAPIHadoopFile(
+    avro_rdd = sc.newAPIHadoopFile(  # type: ignore[var-annotated]

Review comment:
       It's good that mypy tests can pass on examples but I wonder if we can remove such `ignore`. Examples are supposed to be verbose and close to the standard. cc @holdenk FYI in case you have some opinions on this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-974492424


   Merged into master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r745236503



##########
File path: examples/src/main/python/avro_inputformat.py
##########
@@ -75,7 +75,7 @@
         schema_rdd = sc.textFile(sys.argv[2], 1).collect()
         conf = {"avro.schema.input.key": reduce(lambda x, y: x + y, schema_rdd)}
 
-    avro_rdd = sc.newAPIHadoopFile(
+    avro_rdd = sc.newAPIHadoopFile(  # type: ignore[var-annotated]

Review comment:
       The code is generic, in a sense that we don't know what `avro_rdd` actually is, other than being `RDD[Tuple[Any, Any]]`. Since we depend on passing strings and reading schema from file, there is really no way to fill `Any` here, which makes the whole annotations useless in practice. But 
   
   
   ```python
   from typing import Any, Tuple
   ...
   from pyspark.rdd import RDD
   ...
   avro_rdd: RDD[Tuple[Any, Any]] = ...
   ```
   should silence this one without ignore.
   
   The real question is probably what we mean by 
   
   > verbose and close to the standard.
   
   and what style of coding we want to promote. 
   
   There is a lot of value in annotating library code. However, when it comes to simple scripts like our examples, I wouldn't really expect users to add annotations, maybe with exception to annotation functions.
   
   Also, there is another consideration here ‒ at least some of these are included in the examples. If we decide to prefer annotated code instead of ignores (or annotate things in general), all required type imports will have to go into rendered examples, to make things copy-pasteable. It is not only quite verbose and adds a lot of noise to otherwise simple snippets, but also is related to the problems that we discussed offline.
   
   Finally, some ignores are probably unavoidable, unless we want to ban certain methods (talking about you, `DataFrame.head`) from examples whatsoever,  because of the API design.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964289235


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145027/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964644316


   **[Test build #145040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145040/testReport)** for PR 34273 at commit [`7d5b512`](https://github.com/apache/spark/commit/7d5b512df75858263f740eaacc6a188670d35c9b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r727948460



##########
File path: examples/src/main/python/streaming/network_wordjoinsentiments.py
##########
@@ -50,10 +51,17 @@ def print_happiest_words(rdd):
     sc = SparkContext(appName="PythonStreamingNetworkWordJoinSentiments")
     ssc = StreamingContext(sc, 5)
 
+    def line_to_tuple(line: str) -> Tuple[str, str]:
+        try:
+            k, v = line.split(" ")
+            return k, v
+        except ValueError:
+            return "", ""
+
     # Read in the word-sentiment list and create a static RDD from it
     word_sentiments_file_path = "data/streaming/AFINN-111.txt"
     word_sentiments = ssc.sparkContext.textFile(word_sentiments_file_path) \
-        .map(lambda line: tuple(line.split("\t")))
+        .map(line_to_tuple)

Review comment:
       I am really not sure how to handle this.
   
   ```python
   .map(lambda line: tuple(line.split("\t")))
   ```
   returns `RDD[Tuple[str, ...]]`, which will cause cascade of errors later.
   
   We could exclude this file from type check, add casts, rewrite code to:
   
   ```python
   .map(lambda line: line.split("\t"))
   .map(lambda xs: (xs[0], xs[1]))
   ````
   
   but this felt like the least intrusive and the cleanest approach.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942193082


   **[Test build #144202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144202/testReport)** for PR 34273 at commit [`846cdc0`](https://github.com/apache/spark/commit/846cdc0e200d9ab8cc009b0d3c39122116f9d6fd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964130484


   **[Test build #145027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145027/testReport)** for PR 34273 at commit [`b80f2df`](https://github.com/apache/spark/commit/b80f2dfa3a6dd1a40f3b651e74cb95360f60db76).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964261688


   FYI everyone ‒ the few commits added since review, show how tested examples could be modify to drop ignores. Only the one that is related to possible mypy bug is kept, to avoid non-deterministic test failures.
   
   If you believe this looks better than previous variant, I am fine with, if not, I'll revert the changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964414738


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145033/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964505257


   Retest this please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964622325


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49511/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r745226073



##########
File path: examples/src/main/python/avro_inputformat.py
##########
@@ -75,7 +75,7 @@
         schema_rdd = sc.textFile(sys.argv[2], 1).collect()
         conf = {"avro.schema.input.key": reduce(lambda x, y: x + y, schema_rdd)}
 
-    avro_rdd = sc.newAPIHadoopFile(
+    avro_rdd = sc.newAPIHadoopFile(  # type: ignore[var-annotated]

Review comment:
       I don't know how the python annotations related to underlying types - would love to learn! can we change the code to make this not a warning otherwise?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-963787156


   **[Test build #145013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145013/testReport)** for PR 34273 at commit [`735b83e`](https://github.com/apache/spark/commit/735b83eba04708bc3f701f7ee614ea253abc5c4f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `class SparkConf(object):`
     * `class ResultIterable(Iterable[T]):`
     * `class SparkJobInfo(NamedTuple):`
     * `class SparkStageInfo(NamedTuple):`
     * `trait ExpressionBuilder `
     * `trait PadExpressionBuilderBase extends ExpressionBuilder `
     * `case class StringLPad(str: Expression, len: Expression, pad: Expression)`
     * `case class BinaryLPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
     * `case class BinaryRPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
     * `case class DropIndex(`
     * `public class ColumnIOUtil `
     * `public class OrcColumnStatistics `
     * `public class OrcFooterReader `
     * `case class ParquetColumn(`
     * `case class DropIndexExec(`
     * `case class PushedDownOperators(`
     * `case class TableSampleInfo(`
     * `class RatePerMicroBatchProvider extends SimpleTableProvider with DataSourceRegister `
     * `class RatePerMicroBatchTable(`
     * `class RatePerMicroBatchStream(`
     * `case class RatePerMicroBatchStreamOffset(offset: Long, timestamp: Long) extends Offset `
     * `case class RatePerMicroBatchStreamInputPartition(`
     * `class RatePerMicroBatchStreamPartitionReader(`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954876749


   **[Test build #144763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144763/testReport)** for PR 34273 at commit [`43df3a1`](https://github.com/apache/spark/commit/43df3a1d2aede3eb1b9a294a6fe80b17a3ac54d0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954950425


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49232/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954950425


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49232/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-954990037


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144763/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964133583


   **[Test build #145028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145028/testReport)** for PR 34273 at commit [`a498229`](https://github.com/apache/spark/commit/a498229904c2601ba1c6c16a2d71faf707b72673).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r745561614



##########
File path: examples/src/main/python/avro_inputformat.py
##########
@@ -75,7 +75,7 @@
         schema_rdd = sc.textFile(sys.argv[2], 1).collect()
         conf = {"avro.schema.input.key": reduce(lambda x, y: x + y, schema_rdd)}
 
-    avro_rdd = sc.newAPIHadoopFile(
+    avro_rdd = sc.newAPIHadoopFile(  # type: ignore[var-annotated]

Review comment:
       > How about just not running checks on examples if they're to be treated differently?
   
   That's an option, but one I'd personally like to avoid. That's because examples that are most likely to cause problems, are the ones corresponding to the "old" API's, which are the trickiest annotate (these depend heavily on generics, including generics `self`, interacting with user code) and have the least comprehensive coverage.
   
   Just to stress this point ‒ my main motivation to add this tests is not to validate `examples` (this are stable, highly exposed and majority of them has been around for long enough to assert that they're OK), but ensure that annotations are consistent with common usage patterns. 
   
   I definitely won't fight over that, but I am really not convinced that targeted ignores are a problem, especially when we have tons of these (and a huge bag of `casts`) within pyspark code.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964194481


   **[Test build #145025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145025/testReport)** for PR 34273 at commit [`660af11`](https://github.com/apache/spark/commit/660af110a12739e0b2c9e1d7ce1b67b14a021d25).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-964130484


   **[Test build #145027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145027/testReport)** for PR 34273 at commit [`b80f2df`](https://github.com/apache/spark/commit/b80f2dfa3a6dd1a40f3b651e74cb95360f60db76).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 closed pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 closed pull request #34273:
URL: https://github.com/apache/spark/pull/34273


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34273:
URL: https://github.com/apache/spark/pull/34273#discussion_r727940837



##########
File path: dev/lint-python
##########
@@ -124,39 +124,67 @@ function pycodestyle_test {
     fi
 }
 
-function mypy_test {
+
+function mypy_annotation_test {
     local MYPY_REPORT=
     local MYPY_STATUS=
 
-    if ! hash "$MYPY_BUILD" 2> /dev/null; then
-        echo "The $MYPY_BUILD command was not found. Skipping for now."
-        return
+    echo "starting mypy annotations test..."
+    MYPY_REPORT=$( ($MYPY_BUILD --config-file python/mypy.ini python/pyspark) 2>&1)
+    MYPY_STATUS=$?
+
+    if [ "$MYPY_STATUS" -ne 0 ]; then
+        echo "annotations failed mypy checks:"
+        echo "$MYPY_REPORT"
+        echo "$MYPY_STATUS"
+        exit "$MYPY_STATUS"
+    else
+        echo "annotations passed mypy checks."
+        echo
     fi
+}
 
-    _MYPY_VERSION=($($MYPY_BUILD --version))
-    MYPY_VERSION="${_MYPY_VERSION[1]}"
-    EXPECTED_MYPY="$(satisfies_min_version $MYPY_VERSION $MINIMUM_MYPY)"
 
-    if [[ "$EXPECTED_MYPY" == "False" ]]; then
-        echo "The minimum mypy version needs to be $MINIMUM_MYPY. Your current version is $MYPY_VERSION. Skipping for now."
-        return
-    fi
+function mypy_examples_test {
+    local MYPY_REPORT=
+    local MYPY_STATUS=
+
+    echo "starting mypy examples test..."
+
+    MYPY_REPORT=$( (MYPYPATH=python $MYPY_BUILD \
+      --allow-untyped-defs \
+      --config-file python/mypy.ini \
+      --exclude "mllib/*" \

Review comment:
       At the moment, `mllib` examples yield a number of errors, which require adjustments of the actual annotations. I'll resolve this separately drop exclude later.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942193082


   **[Test build #144202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144202/testReport)** for PR 34273 at commit [`846cdc0`](https://github.com/apache/spark/commit/846cdc0e200d9ab8cc009b0d3c39122116f9d6fd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] codecov-commenter commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942196328


   # [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#34273](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9290924) into [master](https://codecov.io/gh/apache/spark/commit/1020c6d8e51d3dc135cc012645950e9a8eea8376?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1020c6d) will **decrease** coverage by `0.04%`.
   > The diff coverage is `88.03%`.
   
   > :exclamation: Current head 9290924 differs from pull request most recent head 846cdc0. Consider uploading reports for the commit 846cdc0 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/spark/pull/34273/graphs/tree.svg?width=650&height=150&src=pr&token=R9pHLWgWi8&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #34273      +/-   ##
   ==========================================
   - Coverage   90.19%   90.14%   -0.05%     
   ==========================================
     Files         288      288              
     Lines       61391    61650     +259     
     Branches     9021     9032      +11     
   ==========================================
   + Hits        55372    55576     +204     
   - Misses       4670     4720      +50     
   - Partials     1349     1354       +5     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `90.12% <88.03%> (-0.05%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [python/pyspark/pandas/indexing.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4aW5nLnB5) | `88.88% <50.00%> (ø)` | |
   | [python/pyspark/sql/observation.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL29ic2VydmF0aW9uLnB5) | `91.30% <50.00%> (-4.16%)` | :arrow_down: |
   | [python/pyspark/pandas/tests/test\_series.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL3Rlc3RzL3Rlc3Rfc2VyaWVzLnB5) | `92.53% <81.81%> (-0.20%)` | :arrow_down: |
   | [python/pyspark/sql/dataframe.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2RhdGFmcmFtZS5weQ==) | `81.50% <82.79%> (-2.87%)` | :arrow_down: |
   | [python/pyspark/pandas/namespace.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL25hbWVzcGFjZS5weQ==) | `77.20% <83.33%> (+0.03%)` | :arrow_up: |
   | [python/pyspark/sql/utils.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL3V0aWxzLnB5) | `85.96% <85.00%> (-1.14%)` | :arrow_down: |
   | [python/pyspark/pandas/indexes/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2luZGV4ZXMvYmFzZS5weQ==) | `96.64% <85.71%> (+0.01%)` | :arrow_up: |
   | [python/pyspark/sql/column.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL2NvbHVtbi5weQ==) | `90.07% <92.00%> (-1.56%)` | :arrow_down: |
   | [python/pyspark/pandas/accessors.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2FjY2Vzc29ycy5weQ==) | `92.01% <100.00%> (ø)` | |
   | [python/pyspark/pandas/base.py](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2Jhc2UucHk=) | `94.11% <100.00%> (+0.03%)` | :arrow_up: |
   | ... and [25 more](https://codecov.io/gh/apache/spark/pull/34273/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f678c75...846cdc0](https://codecov.io/gh/apache/spark/pull/34273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34273: [SPARK-36997][PYTHON][TESTS] Run mypy tests against ml, sql, streaming and core examples

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34273:
URL: https://github.com/apache/spark/pull/34273#issuecomment-942349605


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144202/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org