You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/21 09:29:34 UTC
[GitHub] [spark] zero323 opened a new pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
zero323 opened a new pull request #34354:
URL: https://github.com/apache/spark/pull/34354
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR adds overloads to the following `pyspark.sql.functions`:
- `array`
- `struct`
- `create_map`
- `map_concat`
to support calls with a single `list` or `tuple` argument, i.e.
```python
array(["foo", "bar"])
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
These calls are supported by the current implementation, but don't type check.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Type checker only, as described above.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Existing tests and manual tests (to be added in SPARK-36989)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977765650
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50043/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977790001
**[Test build #145577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145577/testReport)** for PR 34354 at commit [`2693310`](https://github.com/apache/spark/commit/26933101d1bdad67112dc8df81a14631ec53880e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734823494
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
> How about using more general type, like `Sequence` or `Iterable`?
Yeah, this is something that has bothering me for a couple of days now. More general type would be great (assuming we'd modify the code, what wouldn't be a bad idea anyway), if it wasn't for the fact, that `str` is recursively `Sequence[str] ` / `Iterable[str]`.
```python
from typing import Sequence, Iterable
x: Sequence[str] = "abc"
y: Iterable[str] = "abc"
```
I know it backfires in some contexts, but maybe not here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r756322349
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3514,7 +3538,19 @@ def map_from_arrays(col1: "ColumnOrName", col2: "ColumnOrName") -> Column:
return Column(sc._jvm.functions.map_from_arrays(_to_java_column(col1), _to_java_column(col2)))
+@overload
def array(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def array(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
+ ...
+
+
+def array(
+ *cols: Union["ColumnOrName", Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]]
Review comment:
Out of curiosity would flattening unions of unions explicitly be better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948478104
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48972/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734744105
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
How about using more general type, like `Sequence` or `Iterable`?
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
+ ...
+
+
+def struct(
+ *cols: Union["ColumnOrName", Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]]
Review comment:
nit: style, 4-space indent?
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
Actually `List["ColumnOrName"]` is a bit tricky because `List` is not covariant and it doesn't accept `List[str]` or `List[Column]`.
e.g.,
```
% cat python/test.py
from pyspark.sql import functions as F
columns = ['a', 'b', 'c']
reveal_type(columns)
F.struct(columns)
% mypy --config-file python/mypy.ini python/test.py
python/test.py:4: note: Revealed type is "builtins.list[builtins.str*]"
python/test.py:6: error: Argument 1 to "struct" has incompatible type "List[str]"; expected "Union[List[Union[Column, str]], Tuple[Union[Column, str], ...]]" [arg-type]
python/test.py:6: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
python/test.py:6: note: Consider using "Sequence" instead, which is covariant
```
whereas with `def struct(__cols: Sequence["ColumnOrName"]) -> Column:`:
```
% mypy --config-file python/mypy.ini python/test.py
python/test.py:4: note: Revealed type is "builtins.list[builtins.str*]"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1002614239
> Could you rebase this branch to master ??
>
> mypy annotation tests seems not to be passed with current status
@itholic Done. It seems to pass everything in CI. Did you have any particular issues in mind (I've seen some weird local issues, until I cleared mypy cache after the upgrade).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r778528081
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1670,7 +1671,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+ ...
Review comment:
Maybe seems like `List["ColumnOrName"]` just works fine.
Could you briefly explain why we need to use `Union` the `List` and `Tuple` ??
And could I ask if what is the `...` mean in `["ColumnOrName_", ...]` ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948520742
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48972/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-949864223
> I think we might have to redefine `ColumnOrName` to fully support these
What's you idea like?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993090413
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997344156
**[Test build #146374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146374/testReport)** for PR 34354 at commit [`7a646b1`](https://github.com/apache/spark/commit/7a646b16cc1bb3b4a878d8d28ef260ad054c92ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977801622
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50050/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r757162250
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3514,7 +3538,19 @@ def map_from_arrays(col1: "ColumnOrName", col2: "ColumnOrName") -> Column:
return Column(sc._jvm.functions.map_from_arrays(_to_java_column(col1), _to_java_column(col2)))
+@overload
def array(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def array(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
+ ...
+
+
+def array(
+ *cols: Union["ColumnOrName", Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]]
Review comment:
Makes sense. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977854801
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50050/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978180840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977765686
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50043/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948485092
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144500/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734136206
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3455,7 +3469,19 @@ def translate(srcCol: "ColumnOrName", matching: str, replace: str) -> Column:
# ---------------------- Collection functions ------------------------------
+@overload
def create_map(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def create_map(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
qq: what does `__cols` mean here (with two underscores)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997356920
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50848/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997357674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997350446
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146374/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1002410501
Could you rebase this branch to master ??
mypy annotation tests seems not to be passed with current status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r778528081
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1670,7 +1671,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+ ...
Review comment:
Maybe seems like `List["ColumnOrName_"]` just works fine.
Could you briefly explain why we need to use `Union` the `List` and `Tuple` ??
And could I ask if what is the `...` mean in `["ColumnOrName_", ...]` ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1008188701
> t seems to be the best for now, until mypy itself supports looser typing conventions.
As @ueshin pointed out, using `Sequence` might provide more general approach (and allow us to forget about variance), but there are two problems:
- It is high level interface that potentially covers more than lists and tuples. So this would require change of the logic, and to provide consistent UX, we should probably to the same in other places where sequence-ish input is accepted.
- There is concern of ambiguity (`str` is a `Sequence[str]`), which causes my irrational feat that it can break in some hard to contain ways (most likely, I am overthinking it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r779096230
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1670,7 +1671,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+ ...
Review comment:
We still want to support (it was common user request in the past) calls like
```python
struct(("foo", "bar"))
```
which shouldn't be accepted without `Tuple` (or some supertype).
If you try
```python
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 006d10c9fc..caf17a84b3 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -1677,13 +1677,11 @@ def struct(*cols: "ColumnOrName") -> Column:
@overload
-def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+def struct(__cols: Union[List["ColumnOrName_"]]) -> Column:
...
-def struct(
- *cols: Union["ColumnOrName", Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]]
-) -> Column:
+def struct(*cols: Union["ColumnOrName", Union[List["ColumnOrName_"]]]) -> Column:
"""Creates a new struct column.
.. versionadded:: 1.4.0
diff --git a/python/pyspark/sql/tests/typing/test_functions.yml b/python/pyspark/sql/tests/typing/test_functions.yml
index efb3293472..f5f2f13c4f 100644
--- a/python/pyspark/sql/tests/typing/test_functions.yml
+++ b/python/pyspark/sql/tests/typing/test_functions.yml
@@ -66,6 +66,8 @@
create_map(col_objs)
map_concat(col_objs)
+ struct(("foo", "bar"))
+
out: |
main:29: error: No overload variant of "array" matches argument types "List[Column]", "List[Column]" [call-overload]
main:29: note: Possible overload variant:
```
you should see error in data tests
```
___________________________ varargFunctionsOverloads ___________________________
/path/to/spark/python/pyspark/sql/tests/typing/test_functions.yml:19:
E pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output:
E Actual:
E main:50: error: No overload variant of "struct" matches argument type "Tuple[str, str]" [call-overload] (diff)
E main:50: note: Possible overload variants: (diff)
E main:50: note: def struct(*cols: Union[Column, str]) -> Column (diff)
E main:50: note: def [ColumnOrName_] struct(List[ColumnOrName_]) -> Column (diff)
E Expected:
E (empty)
=========================== short test summary info ============================
````
> And could I ask if what is the `...` mean in `["ColumnOrName_", ...]` ??
`Tuples` are typed like product types, so `Tuple[ColumnOrName_]` matches tuple with exactly one column or str element. In contrast `Tuple["ColumnOrName_", ...]` matches tuples of arbitrary size, as long as all elements are columns or strings (there is [mypy doc section](https://mypy.readthedocs.io/en/stable/kinds_of_types.html?highlight=namedtuple#tuple-types) that discusses this syntax further).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997344156
**[Test build #146374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146374/testReport)** for PR 34354 at commit [`7a646b1`](https://github.com/apache/spark/commit/7a646b16cc1bb3b4a878d8d28ef260ad054c92ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948463850
**[Test build #144500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144500/testReport)** for PR 34354 at commit [`cdc1965`](https://github.com/apache/spark/commit/cdc19652a203e6fdc59da3317ad0f719b5366c9c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978180840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977834146
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50050/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977767329
**[Test build #145577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145577/testReport)** for PR 34354 at commit [`2693310`](https://github.com/apache/spark/commit/26933101d1bdad67112dc8df81a14631ec53880e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r779093510
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1670,7 +1671,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+ ...
Review comment:
We still want to support calls like
```python
struct(("foo", "bar"))
```
which shouldn't be accepted without `Tuple` (or some supertype).
If you try
```python
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 006d10c9fc..caf17a84b3 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -1677,13 +1677,11 @@ def struct(*cols: "ColumnOrName") -> Column:
@overload
-def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+def struct(__cols: Union[List["ColumnOrName_"]]) -> Column:
...
-def struct(
- *cols: Union["ColumnOrName", Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]]
-) -> Column:
+def struct(*cols: Union["ColumnOrName", Union[List["ColumnOrName_"]]]) -> Column:
"""Creates a new struct column.
.. versionadded:: 1.4.0
diff --git a/python/pyspark/sql/tests/typing/test_functions.yml b/python/pyspark/sql/tests/typing/test_functions.yml
index efb3293472..fdf2303890 100644
--- a/python/pyspark/sql/tests/typing/test_functions.yml
+++ b/python/pyspark/sql/tests/typing/test_functions.yml
@@ -44,6 +44,7 @@
struct([col("foo"), col("bar")])
struct("foo", "bar")
struct(["foo", "bar"])
+ struct(("foo", "bar"))
array([col("foo")], [col("bar")])
create_map([col("foo")], [col("bar")])
```
you should see error in data tests.
> And could I ask if what is the `...` mean in `["ColumnOrName_", ...]` ??
`Tuples` are typed like product types, so `Tuple[ColumnOrName_]` matches tuple with exactly one column or str element. In contrast `Tuple["ColumnOrName_", ...]` matches tuples of arbitrary size, as long as all elements are columns or strings (there is [mypy doc section](https://mypy.readthedocs.io/en/stable/kinds_of_types.html?highlight=namedtuple#tuple-types) that discusses this further).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 closed pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 closed pull request #34354:
URL: https://github.com/apache/spark/pull/34354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin edited a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
ueshin edited a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-949864223
> I think we might have to redefine `ColumnOrName` to fully support these
What's your idea like?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734825579
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
In general, I think we have a bigger problem with current aliases outliving their usefulness, but that's a topic for a longer discussion and maybe formal design document. Sigh....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977809654
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993090413
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993077400
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948485092
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144500/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948529534
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48972/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993063071
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146150/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993063071
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146150/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734824096
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
In general, I think we have a bigger problem with current aliases outliving their usability, but that's a topic for a longer discussion and maybe formal design document. Sigh....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977765686
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50043/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977716284
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50043/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978125692
**[Test build #145589 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145589/testReport)** for PR 34354 at commit [`7fe1908`](https://github.com/apache/spark/commit/7fe190876b80098e20279ee35fa1009aec1a5973).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978135461
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1008186488
Merged into master.
Thanks everyone!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997356080
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997349280
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1005354124
Oh, `python/pyspark/pandas/indexes/base.py:48: error: Module "pandas._libs" has no attribute "lib" [attr-defined]` is started to fail. Do you have same issue on your local ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1006021438
> @zero323 hmm.. I didn't see anything particularly strange except for a few issues caused by different mypy version. It seems to be working fine now.
I don't think I've seen this one. In my local env I primarily see some errors caused by output mismatch in data tests, dynamically defined methods and stuff covered by https://github.com/apache/spark/pull/34946/, but all of these are non-deterministic and rare, so I don't even have good place to start debugging and asking questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734483147
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3455,7 +3469,19 @@ def translate(srcCol: "ColumnOrName", matching: str, replace: str) -> Column:
# ---------------------- Collection functions ------------------------------
+@overload
def create_map(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def create_map(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
This indicates that argument should be treated as positional-only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948441267
**[Test build #144500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144500/testReport)** for PR 34354 at commit [`cdc1965`](https://github.com/apache/spark/commit/cdc19652a203e6fdc59da3317ad0f719b5366c9c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978144799
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145589/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977809654
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145577/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978108911
**[Test build #145589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145589/testReport)** for PR 34354 at commit [`7fe1908`](https://github.com/apache/spark/commit/7fe190876b80098e20279ee35fa1009aec1a5973).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r756336353
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3514,7 +3538,19 @@ def map_from_arrays(col1: "ColumnOrName", col2: "ColumnOrName") -> Column:
return Column(sc._jvm.functions.map_from_arrays(_to_java_column(col1), _to_java_column(col2)))
+@overload
def array(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def array(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
+ ...
+
+
+def array(
+ *cols: Union["ColumnOrName", Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]]
Review comment:
In general, nested `Unions` are flattened automatically:
```python
>>> Union["ColumnOrName", Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]]
typing.Union[ForwardRef('ColumnOrName'), typing.List[ForwardRef('ColumnOrName')], typing.Tuple[ForwardRef('ColumnOrName'), ...
```
Personally, I find structure that clearly groups similar categories useful (for similar problem see https://github.com/apache/spark/pull/34671#discussion_r754532969), but I am not very attached to this idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978161286
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977767329
**[Test build #145577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145577/testReport)** for PR 34354 at commit [`2693310`](https://github.com/apache/spark/commit/26933101d1bdad67112dc8df81a14631ec53880e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993045933
**[Test build #146150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146150/testReport)** for PR 34354 at commit [`635b230`](https://github.com/apache/spark/commit/635b2309c6198088c6e0f1afb45a9b3853db7daa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997350446
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146374/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997348694
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50848/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997357674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734824096
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
In general, I think we have a bigger problem with current aliases outliving their usefulness, but that's a topic for a longer discussion and maybe formal design document. Sigh....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948430499
New annotations are already implemented, but I think we might have to redefine `ColumnOrName` to fully support these, so I'll keep this as a draft for now.
FYI @ueshin, @HyukjinKwon, @xinrong-databricks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948441267
**[Test build #144500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144500/testReport)** for PR 34354 at commit [`cdc1965`](https://github.com/apache/spark/commit/cdc19652a203e6fdc59da3317ad0f719b5366c9c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1002347455
@itholic can you take a look please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-1005337074
@zero323 hmm.. I didn't see anything particularly strange except for a few issues caused by different mypy version. It seems to be working fine now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993029221
**[Test build #146150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146150/testReport)** for PR 34354 at commit [`635b230`](https://github.com/apache/spark/commit/635b2309c6198088c6e0f1afb45a9b3853db7daa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993056827
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50623/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-993029221
**[Test build #146150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146150/testReport)** for PR 34354 at commit [`635b230`](https://github.com/apache/spark/commit/635b2309c6198088c6e0f1afb45a9b3853db7daa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-948529534
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48972/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r778528187
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1670,7 +1671,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+ ...
+
+
+def struct(
+ *cols: Union["ColumnOrName", Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]]
+) -> Column:
Review comment:
ditto ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r779216109
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1670,7 +1671,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName_"], Tuple["ColumnOrName_", ...]]) -> Column:
+ ...
Review comment:
Got it. Thanks for the comment!! 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-997346413
**[Test build #146374 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146374/testReport)** for PR 34354 at commit [`7a646b1`](https://github.com/apache/spark/commit/7a646b16cc1bb3b4a878d8d28ef260ad054c92ac).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-977854801
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50050/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978108911
**[Test build #145589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145589/testReport)** for PR 34354 at commit [`7fe1908`](https://github.com/apache/spark/commit/7fe190876b80098e20279ee35fa1009aec1a5973).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-978144799
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145589/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-949961121
> > I think we might have to redefine `ColumnOrName` to fully support these
>
> What's your idea like?
Long story short, I've been looking into different scenarios for using aliases and types. Adding inline hints, definitely introduced uses cases that we didn't have before, most notably `casts` (which further separate into cases where we have generics, bound from function signature and none of the above). And there are variances, which pop up here in there.
I suspect, that some of the cases where invariant generics hit us, might be addressed with bounded type vars:
```python
from typing import overload, List, TypeVar, Union
from pyspark.sql import Column
from pyspark.sql.functions import col
ColumnOrName = Union[str, Column]
ColumnOrName_ = TypeVar("ColumnOrName_", bound=ColumnOrName)
def array(__cols: List[ColumnOrName_]): ...
column_names = ["a", "b", "c"]
array(column_names)
columns = [col(x) for x in column_names]
array(columns)
```
but these are not universal and there might be some caveats that I don't see at the moment.
I hope there will be an opportunity to discuss this stuff in a more interactive manner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 edited a comment on pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 edited a comment on pull request #34354:
URL: https://github.com/apache/spark/pull/34354#issuecomment-949961121
> > I think we might have to redefine `ColumnOrName` to fully support these
>
> What's your idea like?
Long story short, I've been looking into different scenarios for using aliases and types. Adding inline hints, definitely introduced uses cases that we didn't have before, most notably `casts` (which further separate into cases where we have generics, bound from function signature and none of the above). And there are variances, which pop up here in there.
I suspect, that some of the cases where invariant generics hit us, might be addressed with bounded type vars:
```python
from typing import overload, List, TypeVar, Union
from pyspark.sql import Column
from pyspark.sql.functions import col
ColumnOrName = Union[str, Column]
ColumnOrName_ = TypeVar("ColumnOrName_", bound=ColumnOrName)
def array(__cols: List[ColumnOrName_]): ...
column_names = ["a", "b", "c"]
array(column_names)
columns = [col(x) for x in column_names]
array(columns)
```
but these are not universal and there might be some caveats that I don't see at the moment.
I hope there will be an opportunity to discuss this stuff in a more interactive manner.
(_Note_: `ColumnOrName` is still needed for `casts` and other annotations in contexts where `ColumnOrName_` would be unbound, like functions without `ColumnOrName _` in arguments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34354: [WIP][SPARK-37085][PYTHON][SQL] Add list/tuple overloads to array, struct, create_map, map_concat
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34354:
URL: https://github.com/apache/spark/pull/34354#discussion_r734849802
##########
File path: python/pyspark/sql/functions.py
##########
@@ -1652,7 +1652,19 @@ def expr(str: str) -> Column:
return Column(sc._jvm.functions.expr(str))
+@overload
def struct(*cols: "ColumnOrName") -> Column:
+ ...
+
+
+@overload
+def struct(__cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
Review comment:
> I know it backfires in some contexts, but maybe not here.
But we'd need explicit checks for strings, like
```python
if len(cols) == 1 and isinstance(cols[0], Sequence) and not isinstance(cols[0], str):
cols = cols[0]
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org