You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/04/21 22:23:36 UTC
[PR] [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF [spark]
Yicong-Huang opened a new pull request, #55467:
URL: https://github.com/apache/spark/pull/55467
### What changes were proposed in this pull request?
Add ASV microbenchmarks for `SQL_GROUPED_AGG_PANDAS_UDF` in `bench_eval_type.py`.
### Why are the changes needed?
Part of [SPARK-55724](https://issues.apache.org/jira/browse/SPARK-55724) (Micro-benchmark PySpark Eval Types). Establishes a performance baseline for `SQL_GROUPED_AGG_PANDAS_UDF` before refactoring.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`COLUMNS=120 asv run --python=same --bench "GroupedAggPandas" --attribute "repeat=(3,5,10.0)"`:
**GroupedAggPandasUDFTimeBench** (`SQL_GROUPED_AGG_PANDAS_UDF`):
```
================ ============ ================
-- udf
---------------- -----------------------------
scenario sum_udf mean_multi_udf
================ ============ ================
few_groups_sm 33.5+/-0.2ms 36.0+/-0.2ms
few_groups_lg 52.9+/-0.3ms 59.9+/-2ms
many_groups_sm 1.23+/-0.01s 1.32+/-0.01s
many_groups_lg 350+/-3ms 379+/-1ms
wide_cols 359+/-1ms 378+/-0.7ms
================ ============ ================
```
**GroupedAggPandasUDFPeakmemBench** (`SQL_GROUPED_AGG_PANDAS_UDF`):
```
================ ========= ================
-- udf
---------------- --------------------------
scenario sum_udf mean_multi_udf
================ ========= ================
few_groups_sm 472M 470M
few_groups_lg 535M 533M
many_groups_sm 501M 499M
many_groups_lg 606M 604M
wide_cols 587M 585M
================ ========= ================
```
Second run (stability check):
```
time: sum_udf few_groups_sm 33.5+/-0.5ms, mean_multi few_groups_sm 35.8+/-0.09ms, ...
peak: identical within 1-2M across all scenarios
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55467:
URL: https://github.com/apache/spark/pull/55467#issuecomment-4293306603
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #55467: [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF
URL: https://github.com/apache/spark/pull/55467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org