You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sinaiamonkar-sai (via GitHub)" <gi...@apache.org> on 2024/05/05 16:17:05 UTC
[PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
sinaiamonkar-sai opened a new pull request, #46391:
URL: https://github.com/apache/spark/pull/46391
### What changes were proposed in this pull request?
In a Scenario where we use GroupBy in PySpark API with relabeling of aggregate columns and using as_index = False,
the columns with which we group by are not returned in the DataFrame. The change proposes to fix this bug.
Example:
ps.DataFrame({"a": [0, 0], "b": [0, 1]}).groupby("a", as_index=False).agg(b_max=("b", "max"))
Result:
b_max
0 1
Required Result:
a b_max
0 0 1
### Why are the changes needed?
The relabeling part of the code only uses only the aggregate columns. In a scenario where as_index=True, it is not an issue as the columns with which we group by are included in the index. When as_index=False, we need to append the columns with which we grouped by to the relabeling code.
Please, check the commits/PR for the code changes
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Passed GA
- Passed Build tests
- Unit Tested including scenarios in addition to the one provided in the Jira ticket
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46391:
URL: https://github.com/apache/spark/pull/46391#issuecomment-2099529188
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "sinaiamonkar-sai (via GitHub)" <gi...@apache.org>.
sinaiamonkar-sai commented on PR #46391:
URL: https://github.com/apache/spark/pull/46391#issuecomment-2094885603
Hello, @holdenk ! This is my first Spark PR. Can you please review it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "sinaiamonkar-sai (via GitHub)" <gi...@apache.org>.
sinaiamonkar-sai commented on code in PR #46391:
URL: https://github.com/apache/spark/pull/46391#discussion_r1592517040
##########
python/pyspark/pandas/groupby.py:
##########
@@ -308,6 +308,7 @@ def aggregate(
)
if not self._as_index:
+ index_cols = list(psdf.columns)
Review Comment:
Hello, @itholic. Thank you for the inputs!
I have updated the code but used psdf._internal.column_labels instead as it gave the desired list of tuples. I hope that is fine. Kindly, check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46391:
URL: https://github.com/apache/spark/pull/46391#issuecomment-2095016743
cc @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "sinaiamonkar-sai (via GitHub)" <gi...@apache.org>.
sinaiamonkar-sai commented on code in PR #46391:
URL: https://github.com/apache/spark/pull/46391#discussion_r1592518229
##########
python/pyspark/pandas/groupby.py:
##########
@@ -322,8 +323,12 @@ def aggregate(
psdf = psdf.reset_index(level=should_drop_index, drop=drop)
if len(should_drop_index) < len(self._groupkeys):
psdf = psdf.reset_index()
+ index_cols = [c for c in list(psdf.columns) if c not in index_cols]
Review Comment:
Updated this is well with psdf._internal.column_labels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #46391: [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False
URL: https://github.com/apache/spark/pull/46391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #46391:
URL: https://github.com/apache/spark/pull/46391#discussion_r1593236433
##########
python/pyspark/pandas/groupby.py:
##########
@@ -308,6 +308,7 @@ def aggregate(
)
if not self._as_index:
+ index_cols = list(psdf.columns)
Review Comment:
Sounds good. Thanks for addressing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #46391:
URL: https://github.com/apache/spark/pull/46391#discussion_r1591930032
##########
python/pyspark/pandas/groupby.py:
##########
@@ -308,6 +308,7 @@ def aggregate(
)
if not self._as_index:
+ index_cols = list(psdf.columns)
Review Comment:
Recommend to use `psdf._internal.data_spark_column_names` here
##########
python/pyspark/pandas/groupby.py:
##########
@@ -322,8 +323,12 @@ def aggregate(
psdf = psdf.reset_index(level=should_drop_index, drop=drop)
if len(should_drop_index) < len(self._groupkeys):
psdf = psdf.reset_index()
+ index_cols = [c for c in list(psdf.columns) if c not in index_cols]
Review Comment:
ditto. `index_cols = [c for c in psdf._internal.data_spark_column_names if c not in index_cols]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False [spark]
Posted by "sinaiamonkar-sai (via GitHub)" <gi...@apache.org>.
sinaiamonkar-sai commented on PR #46391:
URL: https://github.com/apache/spark/pull/46391#issuecomment-2095018053
Thank you @dongjoon-hyun! Sure, let me add that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org