You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/02 21:29:18 UTC
[GitHub] [spark] zero323 opened a new pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
zero323 opened a new pull request #33900:
URL: https://github.com/apache/spark/pull/33900
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
This PR removes `type: ignore[import]` annotations from numpy imports.
Additionally, minimum version of numpy required for the mypy tests is explicitly stated in the GitHub workflow files.
### Why are the changes needed?
Since version 1.20 numpy is PEP 561 compatible so these ignores are no longer necessary.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Type checker only: NumPy types should be validated now, instead of interpreted as `Any`.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727715248
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912429923
**[Test build #142978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142978/testReport)** for PR 33900 at commit [`46362c0`](https://github.com/apache/spark/commit/46362c01457be24b3e50c0bd37deafc3aa274b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912073619
**[Test build #142941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142941/testReport)** for PR 33900 at commit [`5250191`](https://github.com/apache/spark/commit/5250191e12e4ce99fc1f416bc0fe08d1e6ea87ee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727651776
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
Just removing the extra comments should not affect the end users.
For developers, actually seems like `mypy` fails without numpy even now.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -197,7 +197,7 @@ jobs:
- name: Install Python packages (Python 3.8)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
run: |
- python3.8 -m pip install numpy 'pyarrow<5.0.0' pandas scipy xmlrunner
+ python3.8 -m pip install 'numpy>=1.20.0' 'pyarrow<5.0.0' pandas scipy xmlrunner
Review comment:
I'm wondering why we need this change?
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -460,7 +460,7 @@ jobs:
# See also https://github.com/sphinx-doc/sphinx/issues/7551.
# Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
- python3.9 -m pip install 'sphinx<3.1.0' mkdocs numpy pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
+ python3.9 -m pip install 'sphinx<3.1.0' mkdocs 'numpy>=1.20.0' pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
python3.9 -m pip install sphinx_plotly_directive 'pyarrow<5.0.0' pandas 'plotly>=4.8'
Review comment:
I guess we should install `numpy` at the next line, at the same time as pyarrow and pandas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942100582
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727617939
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
WDYT @ueshin ?
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
one valid point on numpy is that ml/mllib already require to have Numpy by default (e.g., we can't even run the tests without numpy) .. so I think it might be fine to do it for NumPy for now ....
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727618797
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
one valid point on numpy is that ml/mllib already require to have Numpy by default (e.g., we can't even run the tests without numpy) .. so I think it might be fine to do it for NumPy for now ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940944111
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912093261
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47442/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r702889707
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
Thank you for your feedback @HyukjinKwon!
I am aware that this is an optional dependency, but I don't think it really affects this PR ‒ I will update the description within the next few days to describe different scenarios and possible impact of this change to make that clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-949056859
Thanks @HyukjinKwon, @ueshin
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912484978
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47479/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942059768
**[Test build #144193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144193/testReport)** for PR 33900 at commit [`dc06c69`](https://github.com/apache/spark/commit/dc06c69ae8f2f30929a92c59f5b0718623097f0f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942083365
**[Test build #144193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144193/testReport)** for PR 33900 at commit [`dc06c69`](https://github.com/apache/spark/commit/dc06c69ae8f2f30929a92c59f5b0718623097f0f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727617939
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
WDYT @ueshin ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912089050
**[Test build #142941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142941/testReport)** for PR 33900 at commit [`5250191`](https://github.com/apache/spark/commit/5250191e12e4ce99fc1f416bc0fe08d1e6ea87ee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912459708
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727651776
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
Just removing the extra comments should not affect the end users.
For developers, actually seems like `mypy` fails without numpy even now.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -197,7 +197,7 @@ jobs:
- name: Install Python packages (Python 3.8)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
run: |
- python3.8 -m pip install numpy 'pyarrow<5.0.0' pandas scipy xmlrunner
+ python3.8 -m pip install 'numpy>=1.20.0' 'pyarrow<5.0.0' pandas scipy xmlrunner
Review comment:
I'm wondering why we need this change?
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -460,7 +460,7 @@ jobs:
# See also https://github.com/sphinx-doc/sphinx/issues/7551.
# Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
- python3.9 -m pip install 'sphinx<3.1.0' mkdocs numpy pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
+ python3.9 -m pip install 'sphinx<3.1.0' mkdocs 'numpy>=1.20.0' pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
python3.9 -m pip install sphinx_plotly_directive 'pyarrow<5.0.0' pandas 'plotly>=4.8'
Review comment:
I guess we should install `numpy` at the next line, at the same time as pyarrow and pandas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942095132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912459708
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r701528321
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
I know this kind of dependency stuff is not very clear in Apache Spark .. but It think we should better keep the ignore for now .. by right NumPy is one of extra dependencies in ML yet .. https://github.com/apache/spark/blob/master/python/setup.py#L262-L274.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940901177
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727815534
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -460,7 +460,7 @@ jobs:
# See also https://github.com/sphinx-doc/sphinx/issues/7551.
# Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
- python3.9 -m pip install 'sphinx<3.1.0' mkdocs numpy pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
+ python3.9 -m pip install 'sphinx<3.1.0' mkdocs 'numpy>=1.20.0' pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
python3.9 -m pip install sphinx_plotly_directive 'pyarrow<5.0.0' pandas 'plotly>=4.8'
Review comment:
Good point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727815420
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -197,7 +197,7 @@ jobs:
- name: Install Python packages (Python 3.8)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
run: |
- python3.8 -m pip install numpy 'pyarrow<5.0.0' pandas scipy xmlrunner
+ python3.8 -m pip install 'numpy>=1.20.0' 'pyarrow<5.0.0' pandas scipy xmlrunner
Review comment:
Strictly speaking, we don't, or we could use package version bound (>=1.7).
But it is useful to have things consistent (of course, unless we intentionally test against different versions), for anyone who, like me, builds dev environments using tests as a guideline. If that's something that doesn't look right, I am happy to revert it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-944893691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r734037532
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -197,7 +197,7 @@ jobs:
- name: Install Python packages (Python 3.8)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
run: |
- python3.8 -m pip install numpy 'pyarrow<5.0.0' pandas scipy xmlrunner
+ python3.8 -m pip install 'numpy>=1.20.0' 'pyarrow<5.0.0' pandas scipy xmlrunner
Review comment:
Are you OK with merging this @ueshin?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912429923
**[Test build #142978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142978/testReport)** for PR 33900 at commit [`46362c0`](https://github.com/apache/spark/commit/46362c01457be24b3e50c0bd37deafc3aa274b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940951701
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727412156
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
Sorry for a delay @HyukjinKwon. It too longer longer than I expected, to test this properly (I had to isolate things, because mypy cache + some undefined quirks, caused unpredictable behavior locally, even if tested within conda / venv environments) .
Long story short ‒ this shouldn't really impact end users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940951701
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942095132
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144193/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912491859
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47479/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-944893691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912445962
**[Test build #142978 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142978/testReport)** for PR 33900 at commit [`46362c0`](https://github.com/apache/spark/commit/46362c01457be24b3e50c0bd37deafc3aa274b3d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940847106
**[Test build #144142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144142/testReport)** for PR 33900 at commit [`b5833a3`](https://github.com/apache/spark/commit/b5833a35ec33435ea0ee520e7ec0564cfd28277b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940864098
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144142/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942059768
**[Test build #144193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144193/testReport)** for PR 33900 at commit [`dc06c69`](https://github.com/apache/spark/commit/dc06c69ae8f2f30929a92c59f5b0718623097f0f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942095132
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144193/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942059768
**[Test build #144193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144193/testReport)** for PR 33900 at commit [`dc06c69`](https://github.com/apache/spark/commit/dc06c69ae8f2f30929a92c59f5b0718623097f0f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942059768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912453810
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47479/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912073619
**[Test build #142941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142941/testReport)** for PR 33900 at commit [`5250191`](https://github.com/apache/spark/commit/5250191e12e4ce99fc1f416bc0fe08d1e6ea87ee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940864098
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144142/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940825099
**[Test build #144142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144142/testReport)** for PR 33900 at commit [`b5833a3`](https://github.com/apache/spark/commit/b5833a35ec33435ea0ee520e7ec0564cfd28277b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942143232
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-944888142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33900:
URL: https://github.com/apache/spark/pull/33900#discussion_r727412156
##########
File path: examples/src/main/python/sql/arrow.py
##########
@@ -32,7 +32,7 @@
def dataframe_with_arrow_example(spark):
- import numpy as np # type: ignore[import]
+ import numpy as np
Review comment:
Sorry for a delay @HyukjinKwon. It too longer longer than I expected, to test this properly (I had to isolate things, because mypy cache + some undefined quirks, caused unpredictable behavior locally, even if tested within conda / venv environments) .
Long story short ‒ this shouldn't really impact end users.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -197,7 +197,7 @@ jobs:
- name: Install Python packages (Python 3.8)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
run: |
- python3.8 -m pip install numpy 'pyarrow<5.0.0' pandas scipy xmlrunner
+ python3.8 -m pip install 'numpy>=1.20.0' 'pyarrow<5.0.0' pandas scipy xmlrunner
Review comment:
Strictly speaking, we don't, or we could use package version bound (>=1.7).
But it is useful to have things consistent (of course, unless we intentionally test against different versions), for anyone who, like me, builds dev environments using tests as a guideline. If that's something that doesn't look right, I am happy to revert it.
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -460,7 +460,7 @@ jobs:
# See also https://github.com/sphinx-doc/sphinx/issues/7551.
# Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
- python3.9 -m pip install 'sphinx<3.1.0' mkdocs numpy pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
+ python3.9 -m pip install 'sphinx<3.1.0' mkdocs 'numpy>=1.20.0' pydata_sphinx_theme ipython nbsphinx numpydoc 'jinja2<3.0.0'
python3.9 -m pip install sphinx_plotly_directive 'pyarrow<5.0.0' pandas 'plotly>=4.8'
Review comment:
Good point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912095752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942143232
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-940825099
**[Test build #144142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144142/testReport)** for PR 33900 at commit [`b5833a3`](https://github.com/apache/spark/commit/b5833a35ec33435ea0ee520e7ec0564cfd28277b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942095132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-942134326
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912491859
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47479/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-912095752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33900:
URL: https://github.com/apache/spark/pull/33900#issuecomment-944888142
**[Test build #144322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144322/testReport)** for PR 33900 at commit [`ca11801`](https://github.com/apache/spark/commit/ca118010dcac5c7ec4c03dfcba8fbfb4bb251d43).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 closed pull request #33900: [SPARK-36654][PYTHON] Drop type ignores from numpy imports
Posted by GitBox <gi...@apache.org>.
zero323 closed pull request #33900:
URL: https://github.com/apache/spark/pull/33900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org