You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/15 12:54:10 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #34007: [SPARK-36710][PYTHON] Support new syntax in function apply APIs in pandas API on Spark
HyukjinKwon opened a new pull request #34007:
URL: https://github.com/apache/spark/pull/34007
### What changes were proposed in this pull request?
This PR proposes the new syntax introduced in https://github.com/apache/spark/pull/33954. Namely, users now can specify the index type and name as below:
```python
import pandas as pd
import pyspark.pandas as ps
def transform(pdf) -> pd.DataFrame[int, [int, int]]:
pdf['A'] = pdf.id + 1
return pdf
ps.range(5).koalas.apply_batch(transform)
```
```
c0 c1
0 0 1
1 1 2
2 2 3
3 3 4
4 4 5
```
```python
import pandas as pd
import pyspark.pandas as ps
def transform(pdf) -> pd.DataFrame[("index", int), [("a", int), ("b", int)]]:
pdf['A'] = pdf.id * pdf.id
return pdf
ps.range(5).koalas.apply_batch(transform)
```
```
a b
index
0 0 0
1 1 1
2 2 4
3 3 9
4 4 16
```
### Why are the changes needed?
The rationale is described in https://github.com/apache/spark/pull/33954. In order to avoid unnecessary computation for default index or schema inference.
### Does this PR introduce _any_ user-facing change?
Yes, this PR affects the following APIs:
- `DataFrame.apply(..., axis=1)`
- `DataFrame.groupby.apply(...)`
- `DataFrame.pandas_on_spark.transform_batch(...)`
- `DataFrame.pandas_on_spark.apply_batch(...)`
Now they can specify the index type with the new syntax below:
```
DataFrame[index_type, [type, ...]]
DataFrame[(index_name, index_type), [(name, type), ...]]
DataFrame[dtype instance, dtypes instance]
DataFrame[(index_name, index_type), zip(names, types)]
```
### How was this patch tested?
Manually tested, and unittests were added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920037455
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47804/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920030188
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47804/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920405444
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143323/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920033639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920370855
**[Test build #143323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143323/testReport)** for PR 34007 at commit [`682a95a`](https://github.com/apache/spark/commit/682a95aeb9622d7b65aa67e8a76754cc9f1e216d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920088390
**[Test build #143304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143304/testReport)** for PR 34007 at commit [`0b436b8`](https://github.com/apache/spark/commit/0b436b8e5780c59e5f49a56ff9b485a287b14f54).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920139316
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-919994382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920041202
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47805/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920034970
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47805/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920406178
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47826/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920135624
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920144733
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920405444
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143323/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920015321
**[Test build #143301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143301/testReport)** for PR 34007 at commit [`7eef305`](https://github.com/apache/spark/commit/7eef3056f37968c42cc12f7923422907039cb0b9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920041269
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47805/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920033640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34007: [SPARK-36710][PYTHON] Support new syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-919993159
cc @ueshin @xinrong-databricks @itholic FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920406107
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47826/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920139316
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920144688
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920041269
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47805/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920402021
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47826/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920118009
**[Test build #143304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143304/testReport)** for PR 34007 at commit [`0b436b8`](https://github.com/apache/spark/commit/0b436b8e5780c59e5f49a56ff9b485a287b14f54).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920406178
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47826/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920370855
**[Test build #143323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143323/testReport)** for PR 34007 at commit [`682a95a`](https://github.com/apache/spark/commit/682a95aeb9622d7b65aa67e8a76754cc9f1e216d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34007: [SPARK-36710][PYTHON] Support new syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34007:
URL: https://github.com/apache/spark/pull/34007#discussion_r709156586
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -4428,6 +4428,28 @@ def f2(_) -> ps.Series[int]:
pdf.apply(lambda x: len(x), axis=1).sort_index(),
)
+ def test_apply_with_type(self):
+ pdf = self.pdf
+ psdf = ps.from_pandas(pdf)
+
+ def identify1(x) -> ps.DataFrame[int, int]:
+ return x
+
+ # Type hints set the default column names, and we use default index for
+ # pandas API on Spark. Here we ignore both diff.
+ actual = psdf.apply(identify1, axis=1)
+ expected = pdf.apply(identify1, axis=1)
+ self.assert_eq(sorted(actual["c0"].to_numpy()), sorted(expected["a"].to_numpy()))
+ self.assert_eq(sorted(actual["c1"].to_numpy()), sorted(expected["b"].to_numpy()))
+
+ def identify2(x) -> ps.DataFrame[slice("a", int), slice("b", int)]: # noqa: F405
Review comment:
I used slice because mypy fails. FYI, the new syntax doesn't fail w/ mypy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34007:
URL: https://github.com/apache/spark/pull/34007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-922588658
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920024917
**[Test build #143303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143303/testReport)** for PR 34007 at commit [`8f312b3`](https://github.com/apache/spark/commit/8f312b349702679d6260f1a3903f4a09bc273ad5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920003462
**[Test build #143303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143303/testReport)** for PR 34007 at commit [`8f312b3`](https://github.com/apache/spark/commit/8f312b349702679d6260f1a3903f4a09bc273ad5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920037455
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47804/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920020075
**[Test build #143302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143302/testReport)** for PR 34007 at commit [`4271ffc`](https://github.com/apache/spark/commit/4271ffc34dedf5e895380b7c59dda0ae43ebe864).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34007:
URL: https://github.com/apache/spark/pull/34007#discussion_r709564361
##########
File path: python/pyspark/pandas/accessors.py
##########
@@ -230,28 +231,24 @@ def apply_batch(
To avoid this, specify return type in ``func``, for instance, as below:
- >>> def plus_one(x) -> ps.DataFrame[float, float]:
+ >>> def plus_one(x) -> ps.DataFrame[int, [float, float]]:
... return x + 1
If the return type is specified, the output column names become
`c0, c1, c2 ... cn`. These names are positionally mapped to the returned
DataFrame in ``func``.
- To specify the column names, you can assign them in a pandas friendly style as below:
+ To specify the column names, you can assign them in a NumPy compound type style
+ as below:
- >>> def plus_one(x) -> ps.DataFrame["a": float, "b": float]:
+ >>> def plus_one(x) -> ps.DataFrame[("index", int), [("a", float), ("b", float)]:
Review comment:
```suggestion
>>> def plus_one(x) -> ps.DataFrame[("index", int), [("a", float), ("b", float)]]:
```
##########
File path: python/pyspark/pandas/accessors.py
##########
@@ -439,28 +447,24 @@ def transform_batch(
To avoid this, specify return type in ``func``, for instance, as below:
- >>> def plus_one(x) -> ps.DataFrame[float, float]:
+ >>> def plus_one(x) -> ps.DataFrame[int, [float, float]]:
... return x + 1
If the return type is specified, the output column names become
`c0, c1, c2 ... cn`. These names are positionally mapped to the returned
DataFrame in ``func``.
- To specify the column names, you can assign them in a pandas friendly style as below:
+ To specify the column names, you can assign them in a NumPy compound type style
+ as below:
- >>> def plus_one(x) -> ps.DataFrame['a': float, 'b': float]:
+ >>> def plus_one(x) -> ps.DataFrame[("index", int), [("a", float), ("b", float)]:
Review comment:
```suggestion
>>> def plus_one(x) -> ps.DataFrame[("index", int), [("a", float), ("b", float)]]:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-921362385
will merge in 3 days if there are no more comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-919998712
**[Test build #143302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143302/testReport)** for PR 34007 at commit [`4271ffc`](https://github.com/apache/spark/commit/4271ffc34dedf5e895380b7c59dda0ae43ebe864).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920144733
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920088390
**[Test build #143304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143304/testReport)** for PR 34007 at commit [`0b436b8`](https://github.com/apache/spark/commit/0b436b8e5780c59e5f49a56ff9b485a287b14f54).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920394342
**[Test build #143323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143323/testReport)** for PR 34007 at commit [`682a95a`](https://github.com/apache/spark/commit/682a95aeb9622d7b65aa67e8a76754cc9f1e216d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-919994382
**[Test build #143301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143301/testReport)** for PR 34007 at commit [`7eef305`](https://github.com/apache/spark/commit/7eef3056f37968c42cc12f7923422907039cb0b9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34007: [SPARK-36710][PYTHON] Support new typing syntax in function apply APIs in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34007:
URL: https://github.com/apache/spark/pull/34007#issuecomment-920037396
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47804/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org