You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/03 04:07:26 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request, #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
HyukjinKwon opened a new pull request, #37390:
URL: https://github.com/apache/spark/pull/37390
### What changes were proposed in this pull request?
This PR proposes to apply the projection to respect the reordered columns in its child when group attributes are empty.
### Why are the changes needed?
To respect the column order in the child.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a bug as below:
```python
import pandas as pd
from pyspark.sql import functions as f
@f.pandas_udf("double")
def AVG(x: pd.Series) -> float:
return x.mean()
abc = spark.createDataFrame([(1.0, 5.0, 17.0)], schema=["a", "b", "c"])
abc.agg(AVG("a"), AVG("c")).show()
abc.select("c", "a").agg(AVG("a"), AVG("c")).show()
```
**Before**
```
+------+------+
|AVG(a)|AVG(c)|
+------+------+
| 17.0| 1.0|
+------+------+
```
**After**
```
+------+------+
|AVG(a)|AVG(c)|
+------+------+
| 1.0| 17.0|
+------+------+
```
### How was this patch tested?
Manually tested, and added an unittest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #37390:
URL: https://github.com/apache/spark/pull/37390#issuecomment-1204658808
@HyukjinKwon sorry, but it seems this backport breaks CI in bracnh-3.1 and branch-3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37390:
URL: https://github.com/apache/spark/pull/37390#issuecomment-1204663535
let me revert the tests only for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
URL: https://github.com/apache/spark/pull/37390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #37390:
URL: https://github.com/apache/spark/pull/37390#discussion_r936250722
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/AggregateInPandasExec.scala:
##########
@@ -131,12 +131,13 @@ case class AggregateInPandasExec(
val newIter: Iterator[InternalRow] = mayAppendUpdatingSessionIterator(iter)
val prunedProj = UnsafeProjection.create(allInputs.toSeq, child.output)
- val grouped = if (groupingExpressions.isEmpty) {
+ val groupedItr = if (groupingExpressions.isEmpty) {
// Use an empty unsafe row as a place holder for the grouping key
Iterator((new UnsafeRow(), newIter))
} else {
GroupedIterator(newIter, groupingExpressions, child.output)
- }.map { case (key, rows) =>
+ }
+ val grouped = groupedItr.map { case (key, rows) =>
(key, rows.map(prunedProj))
}
Review Comment:
This only applies on `GroupedIterator` previously? Looks like both...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37390:
URL: https://github.com/apache/spark/pull/37390#issuecomment-1203570075
Let me merge this. I'm 100% sure the test failure is irrelevant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
itholic commented on PR #37390:
URL: https://github.com/apache/spark/pull/37390#issuecomment-1203543002
Looks good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37390:
URL: https://github.com/apache/spark/pull/37390#issuecomment-1203573725
Merged to master, branch-3.3, branch-3.2, and branch-3.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37390: [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37390:
URL: https://github.com/apache/spark/pull/37390#issuecomment-1203494225
cc @viirya mind taking a quick look please? I think it's just a silly mistake from the very first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org