You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "JacobZheng0927 (via GitHub)" <gi...@apache.org> on 2023/11/15 08:25:06 UTC
[PR] [SPARK-45898][SQL] Support groupingSets operation in dataframe api [spark]
JacobZheng0927 opened a new pull request, #43813:
URL: https://github.com/apache/spark/pull/43813
### What changes were proposed in this pull request?
Add groupingSets method in dataset api.
`select col1, col2, col3, sum(col4) FROM t GROUP col1, col2, col3 BY GROUPING SETS ((col1, col2), ())`
This SQL can be equivalently replaced with the following code:
`df.groupingSets(Seq(Seq("col1", "col2"), Seq()), "col1", "col2", "col3").sum("col4")`
### Why are the changes needed?
Currently grouping sets can only be used in spark sql. This feature is not available when developing with the dataset api.
### Does this PR introduce _any_ user-facing change?
Yes. This PR introduces the use of groupingSets in the dataset api.
### How was this patch tested?
Tests added in `DataFrameAggregateSuite.scala`.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43813:
URL: https://github.com/apache/spark/pull/43813#issuecomment-1818826704
I think you should rebase against the latest master branch to fix up the test failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398685261
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1825,6 +1825,33 @@ class Dataset[T] private[sql](
RelationalGroupedDataset(toDF(), cols.map(_.expr), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq($"department", $"group"),Seq()),$"department", $"group").avg()
Review Comment:
The SQL syntax is `GROUP BY ... GROUPING SETS (...)`, shall we put grouping cols as the first parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398802694
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1960,6 +1987,42 @@ class Dataset[T] private[sql](
toDF(), colNames.map(colName => resolve(colName)), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * This is a variant of groupingSets that can only group by existing columns using column names
+ * (i.e. cannot construct expressions).
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq("department", "group"),Seq()),"department", "group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq($"department"), Seq()).agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[String]],
Review Comment:
👍
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1960,6 +1987,42 @@ class Dataset[T] private[sql](
toDF(), colNames.map(colName => resolve(colName)), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * This is a variant of groupingSets that can only group by existing columns using column names
+ * (i.e. cannot construct expressions).
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq("department", "group"),Seq()),"department", "group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq($"department"), Seq()).agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[String]],
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398684974
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1960,6 +1987,42 @@ class Dataset[T] private[sql](
toDF(), colNames.map(colName => resolve(colName)), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * This is a variant of groupingSets that can only group by existing columns using column names
+ * (i.e. cannot construct expressions).
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq("department", "group"),Seq()),"department", "group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq($"department"), Seq()).agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[String]],
Review Comment:
Shall we stop adding string column overloads for new APIs? also cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43813: [SPARK-45929][SQL] Support groupingSets operation in dataframe api
URL: https://github.com/apache/spark/pull/43813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398827809
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1960,6 +1987,42 @@ class Dataset[T] private[sql](
toDF(), colNames.map(colName => resolve(colName)), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * This is a variant of groupingSets that can only group by existing columns using column names
+ * (i.e. cannot construct expressions).
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq("department", "group"),Seq()),"department", "group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq($"department"), Seq()).agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[String]],
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1395366613
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1825,6 +1825,33 @@ class Dataset[T] private[sql](
RelationalGroupedDataset(toDF(), cols.map(_.expr), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq($"department", $"group"),Seq()),$"department", $"group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq()),$"department", $"group").agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 3.5.0
Review Comment:
Should be 4.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398828300
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1825,6 +1825,33 @@ class Dataset[T] private[sql](
RelationalGroupedDataset(toDF(), cols.map(_.expr), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq($"department", $"group"),Seq()),$"department", $"group").avg()
Review Comment:
oh I see, then no problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "JacobZheng0927 (via GitHub)" <gi...@apache.org>.
JacobZheng0927 commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1395394359
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1825,6 +1825,33 @@ class Dataset[T] private[sql](
RelationalGroupedDataset(toDF(), cols.map(_.expr), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq($"department", $"group"),Seq()),$"department", $"group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq()),$"department", $"group").agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 3.5.0
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398628293
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1825,6 +1825,33 @@ class Dataset[T] private[sql](
RelationalGroupedDataset(toDF(), cols.map(_.expr), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq($"department", $"group"),Seq()),$"department", $"group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq()),$"department", $"group").agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[Column]],
+ cols: Column*): RelationalGroupedDataset = {
Review Comment:
```suggestion
groupingSets: Seq[Seq[Column]],
cols: Column*): RelationalGroupedDataset = {
```
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1960,6 +1987,42 @@ class Dataset[T] private[sql](
toDF(), colNames.map(colName => resolve(colName)), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * This is a variant of groupingSets that can only group by existing columns using column names
+ * (i.e. cannot construct expressions).
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq("department", "group"),Seq()),"department", "group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq($"department"), Seq()).agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[String]],
+ col1: String,
+ cols: String*): RelationalGroupedDataset = {
Review Comment:
```suggestion
groupingSets: Seq[Seq[String]],
col1: String,
cols: String*): RelationalGroupedDataset = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "JacobZheng0927 (via GitHub)" <gi...@apache.org>.
JacobZheng0927 commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398870581
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1960,6 +1987,42 @@ class Dataset[T] private[sql](
toDF(), colNames.map(colName => resolve(colName)), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * This is a variant of groupingSets that can only group by existing columns using column names
+ * (i.e. cannot construct expressions).
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq("department", "group"),Seq()),"department", "group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq($"department"), Seq()).agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[String]],
Review Comment:
Got it. This API has been removed.
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1825,6 +1825,33 @@ class Dataset[T] private[sql](
RelationalGroupedDataset(toDF(), cols.map(_.expr), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq($"department", $"group"),Seq()),$"department", $"group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq()),$"department", $"group").agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[Column]],
+ cols: Column*): RelationalGroupedDataset = {
Review Comment:
Fixed. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43813:
URL: https://github.com/apache/spark/pull/43813#issuecomment-1818148935
cc @zhengruifeng FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "JacobZheng0927 (via GitHub)" <gi...@apache.org>.
JacobZheng0927 commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398808757
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1960,6 +1987,42 @@ class Dataset[T] private[sql](
toDF(), colNames.map(colName => resolve(colName)), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * This is a variant of groupingSets that can only group by existing columns using column names
+ * (i.e. cannot construct expressions).
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq("department", "group"),Seq()),"department", "group").avg()
+ *
+ * // Compute the max age and average salary, group by specific grouping sets.
+ * ds.groupingSets(Seq($"department", $"gender"), Seq($"department"), Seq()).agg(Map(
+ * "salary" -> "avg",
+ * "age" -> "max"
+ * ))
+ * }}}
+ *
+ * @group untypedrel
+ * @since 4.0.0
+ */
+ @scala.annotation.varargs
+ def groupingSets(
+ groupingSets: Seq[Seq[String]],
Review Comment:
Are you saying that the methods using string columns are no longer needed, and that I only need to keep the methods using columns of type Column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "JacobZheng0927 (via GitHub)" <gi...@apache.org>.
JacobZheng0927 commented on code in PR #43813:
URL: https://github.com/apache/spark/pull/43813#discussion_r1398805099
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -1825,6 +1825,33 @@ class Dataset[T] private[sql](
RelationalGroupedDataset(toDF(), cols.map(_.expr), RelationalGroupedDataset.CubeType)
}
+ /**
+ * Create multi-dimensional aggregation for the current Dataset using the specified grouping sets,
+ * so we can run aggregation on them.
+ * See [[RelationalGroupedDataset]] for all the available aggregate functions.
+ *
+ * {{{
+ * // Compute the average for all numeric columns group by specific grouping sets.
+ * ds.groupingSets(Seq(Seq($"department", $"group"),Seq()),$"department", $"group").avg()
Review Comment:
This is because the * parameter must be placed at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45929][SQL] Support groupingSets operation in dataframe api [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43813:
URL: https://github.com/apache/spark/pull/43813#issuecomment-1820077852
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org