You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/28 05:06:23 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
zhengruifeng opened a new pull request, #39254:
URL: https://github.com/apache/spark/pull/39254
### What changes were proposed in this pull request?
Implement `GroupedData.{min, max, avg, sum}`
### Why are the changes needed?
TLDR, `df.groupby().min` != `df.groupby().agg(min)`
1, if no columns are provided, `GroupedData.min` will check the schema and choose the columns with matched datatypes;
2, if a column with unacceptable datatype is given, than `GroupedData.min` will throw exception like
`"department" is not a numeric column. Aggregation function can only be applied on a numeric column.`
### Does this PR introduce _any_ user-facing change?
yes, make `GroupedData.{min, max, avg, sum}` consistent with PySpark
### How was this patch tested?
added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058089769
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) -> "DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) -> Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg", "sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
Wait, sorry my bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #39254:
URL: https://github.com/apache/spark/pull/39254#issuecomment-1367012699
thank you for the reviews @grundprinzip @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058066260
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1061,6 +1068,81 @@ class SparkConnectPlanner(session: SparkSession) {
}
}
+ private def transformNumericAggregate(agg: proto.Aggregate): LogicalPlan = {
+ if (!agg.hasInput) {
+ throw InvalidPlanInput("Aggregate needs a plan input")
+ }
+ if (agg.getAggregateExpressionsCount == 0) {
Review Comment:
wouldn't this be caught by the analyzer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058084926
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1061,6 +1068,81 @@ class SparkConnectPlanner(session: SparkSession) {
}
}
+ private def transformNumericAggregate(agg: proto.Aggregate): LogicalPlan = {
Review Comment:
I believe it's necessary to have a separate method for it, actually at first I attempted to implement it in existing `transformAggregate` then I found that the two paths are too different:
1, for the regular agg, we build the logical plan;
2, for the numeric agg, we have to use the dataframe API to avoid doing the analysis by myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39254:
URL: https://github.com/apache/spark/pull/39254#issuecomment-1367011628
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058085692
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1061,6 +1068,81 @@ class SparkConnectPlanner(session: SparkSession) {
}
}
+ private def transformNumericAggregate(agg: proto.Aggregate): LogicalPlan = {
Review Comment:
Can't you just let the analysis fail? It will fail anyways if you try to aggregate on non numeric columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
URL: https://github.com/apache/spark/pull/39254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058067328
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) -> "DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) -> Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg", "sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
what happens when you only do this change but leave the `is_mumeric` out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058092052
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) -> "DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) -> Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg", "sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
I see now why you do it that way, but I'm not convinced it's the best way.
The only delta I see that can't be handled reasonably is the case of empty agg expressions where we know the function. And here I'm wondering if we should just get the schema instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #39254:
URL: https://github.com/apache/spark/pull/39254#issuecomment-1366372126
cc @HyukjinKwon @grundprinzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #39254:
URL: https://github.com/apache/spark/pull/39254#issuecomment-1366999485
cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058066012
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1061,6 +1068,81 @@ class SparkConnectPlanner(session: SparkSession) {
}
}
+ private def transformNumericAggregate(agg: proto.Aggregate): LogicalPlan = {
Review Comment:
Is that duplication really necessary? What exactly is the difference on the server side for the numeric aggregation? Do we have this special handling in Catalyst for this special case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058085725
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1061,6 +1068,81 @@ class SparkConnectPlanner(session: SparkSession) {
}
}
+ private def transformNumericAggregate(agg: proto.Aggregate): LogicalPlan = {
+ if (!agg.hasInput) {
+ throw InvalidPlanInput("Aggregate needs a plan input")
+ }
+ if (agg.getAggregateExpressionsCount == 0) {
Review Comment:
this protocol `functionName: String, aggCols: Seq[String]` is just defined in this PR, we have to process it by ourself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058089392
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) -> "DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) -> Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg", "sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
Yeah, but why not convert the function to a scalar function like we do for the above agg method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058086316
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) -> "DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) -> Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg", "sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
the agg expressions are just literal strings, it will return something like:
```
In [6]: df.groupBy("age").agg(F.lit("max"), F.lit("col1")).show()
+---+---+----+
|age|max|col1|
+---+---+----+
| 2|max|col1|
| 5|max|col1|
+---+---+----+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058089550
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) -> "DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) -> Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg", "sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
You're already validating the input anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #39254: [SPARK-41333][SPARK-41737] Implement `GroupedData.{min, max, avg, sum}`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058094822
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) -> "DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) -> Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg", "sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
sound reasonable, we can analyze the schema in the client, then do not need to change the sever side. Let me take another look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org