You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "lucas-nelson-uiuc (via GitHub)" <gi...@apache.org> on 2024/11/24 21:46:23 UTC
[PR] [SPARK-50404][PySpark] PySpark DataFrame Pipe Method [spark]
lucas-nelson-uiuc opened a new pull request, #48947:
URL: https://github.com/apache/spark/pull/48947
Suggested implementation for a pipe method in the PySpark DataFrame. Similar to the transform method, this method can be called directly on a DataFrame to perform custom transformations functions. However, unlike the current transform method which requires one call per custom transformation, pipe can accept an ambiguous number of transformations and chain them together on the user's behalf.
Using the existing documentation for `DataFrame.transform`, the suggested pipe method would look like such:
```python
from pyspark.sql.functions import col
df = spark.createDataFrame([(1, 1.0), (2, 2.0)], ["int", "float"])
def cast_all_to_int(input_df):
return input_df.select([col(col_name).cast("int") for col_name in input_df.columns])
def sort_columns_asc(input_df):
return input_df.select(*sorted(input_df.columns))
# with transform method
df.transform(cast_all_to_int).transform(sort_columns_asc).show()
# with pipe method
df.pipe(cast_all_to_int, sort_columns_asc)
```
For functions that take parameters, users can pass closures or partially defined functions.
```python
from typing import Callable
import functools
def add_n(input_df, n):
return input_df.select([(col(col_name) + n).alias(col_name)
for col_name in input_df.columns])
# define a partial function
add_one = functools.partial(add_n, n=1)
# or, define a function that returns a closure
def add_n(n: int) -> Callable:
def closure(input_df: DataFrame) -> DataFrame:
return input_df.select([(col(col_name) + n).alias(col_name)
for col_name in input_df.columns])
return closure
# with transform method
df.transform(add_n, 1).transform(add_n, n=10).show()
# with pipe method
df.pipe(add_one, add_n(n=10)).show()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "lucas-nelson-uiuc (via GitHub)" <gi...@apache.org>.
lucas-nelson-uiuc commented on code in PR #48947:
URL: https://github.com/apache/spark/pull/48947#discussion_r1857165004
##########
python/pyspark/sql/classic/dataframe.py:
##########
@@ -1699,6 +1700,19 @@ def transform(
), "Func returned an instance of type [%s], " "should have been DataFrame." % type(result)
return result
+ def pipe(
+ self, *funcs: tuple[Callable[..., ParentDataFrame]]
+ ) -> ParentDataFrame:
+ result = functools.reduce(
+ lambda init, func: init.transform(func),
Review Comment:
@HyukjinKwon I feel like the same could be said about `DataFrame.transform` - the source code for that method is simply calling the passed function with optional arguments/keyword-arguments.
I agree that this isn't a difficult implementation - however, it does seem like a natural next step for applying custom transformations. As of right now, users can apply custom transformations through:
- Nested functions: `h(g(f(input_df)))`
- Transformation method: `input_df.transform(f).transform(g).transform(h)`
- Pipe method: `input_df.pipe(f, g, h)`
In my opinion, using the pipe method is much easier to read than nesting functions and more succinct than chaining transforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #48947:
URL: https://github.com/apache/spark/pull/48947#discussion_r1855611571
##########
python/pyspark/sql/classic/dataframe.py:
##########
@@ -1699,6 +1700,19 @@ def transform(
), "Func returned an instance of type [%s], " "should have been DataFrame." % type(result)
return result
+ def pipe(
+ self, *funcs: tuple[Callable[..., ParentDataFrame]]
+ ) -> ParentDataFrame:
+ result = functools.reduce(
+ lambda init, func: init.transform(func),
Review Comment:
I would't add this API - it can be easily done at user side as you're demonstrating here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #48947: [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method
URL: https://github.com/apache/spark/pull/48947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "lucas-nelson-uiuc (via GitHub)" <gi...@apache.org>.
lucas-nelson-uiuc commented on code in PR #48947:
URL: https://github.com/apache/spark/pull/48947#discussion_r1867891001
##########
python/pyspark/sql/classic/dataframe.py:
##########
@@ -1699,6 +1700,19 @@ def transform(
), "Func returned an instance of type [%s], " "should have been DataFrame." % type(result)
return result
+ def pipe(
+ self, *funcs: tuple[Callable[..., ParentDataFrame]]
+ ) -> ParentDataFrame:
+ result = functools.reduce(
+ lambda init, func: init.transform(func),
Review Comment:
@HyukjinKwon any thoughts on my comment above? Also noticed that the docstring for `DataFrame.transform` mentions it as a "concise syntax for chaining custom transformations". The proposed `DataFrame.pipe` method would be an even more concise syntax for performing the same task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "lucas-nelson-uiuc (via GitHub)" <gi...@apache.org>.
lucas-nelson-uiuc commented on code in PR #48947:
URL: https://github.com/apache/spark/pull/48947#discussion_r1857165004
##########
python/pyspark/sql/classic/dataframe.py:
##########
@@ -1699,6 +1700,19 @@ def transform(
), "Func returned an instance of type [%s], " "should have been DataFrame." % type(result)
return result
+ def pipe(
+ self, *funcs: tuple[Callable[..., ParentDataFrame]]
+ ) -> ParentDataFrame:
+ result = functools.reduce(
+ lambda init, func: init.transform(func),
Review Comment:
I feel like the same could be said about `DataFrame.transform`.
Before this was in the API, users would have to write `h(g(f(input_df)))`. With transform, users now write `input_df.transform(f).transform(g).transform(h)`. Implementing `DataFrame.pipe` would shorten this expression to `input_df.pipe(f, g, h)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48947:
URL: https://github.com/apache/spark/pull/48947#issuecomment-2722995058
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "lucas-nelson-uiuc (via GitHub)" <gi...@apache.org>.
lucas-nelson-uiuc commented on code in PR #48947:
URL: https://github.com/apache/spark/pull/48947#discussion_r1867891001
##########
python/pyspark/sql/classic/dataframe.py:
##########
@@ -1699,6 +1700,19 @@ def transform(
), "Func returned an instance of type [%s], " "should have been DataFrame." % type(result)
return result
+ def pipe(
+ self, *funcs: tuple[Callable[..., ParentDataFrame]]
+ ) -> ParentDataFrame:
+ result = functools.reduce(
+ lambda init, func: init.transform(func),
Review Comment:
@HyukjinKwon any thoughts on my comment above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50404][PYTHON] PySpark DataFrame Pipe Method [spark]
Posted by "lucas-nelson-uiuc (via GitHub)" <gi...@apache.org>.
lucas-nelson-uiuc commented on code in PR #48947:
URL: https://github.com/apache/spark/pull/48947#discussion_r1857165004
##########
python/pyspark/sql/classic/dataframe.py:
##########
@@ -1699,6 +1700,19 @@ def transform(
), "Func returned an instance of type [%s], " "should have been DataFrame." % type(result)
return result
+ def pipe(
+ self, *funcs: tuple[Callable[..., ParentDataFrame]]
+ ) -> ParentDataFrame:
+ result = functools.reduce(
+ lambda init, func: init.transform(func),
Review Comment:
@HyukjinKwon I feel like the same could be said about `DataFrame.transform` - the source code for that method is simply calling the passed function with optional arguments/keyword-arguments.
I agree that this isn't a difficult implementation - however, it does seem like a natural next step for applying custom transformations. As of right now, users can use:
- Nested functions: `h(g(f(input_df)))`
- Transformation method: `input_df.transform(f).transform(g).transform(h)`
- Pipe method: `input_df.pipe(f, g, h)`
In my opinion, using the pipe method is much easier to read than nesting functions and more succinct than chaining transforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org