You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/04/29 07:02:09 UTC
[PR] [SPARK-56649][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_MAP_PANDAS_ITER_UDF [spark]
Yicong-Huang opened a new pull request, #55600:
URL: https://github.com/apache/spark/pull/55600
### What changes were proposed in this pull request?
Adds ASV microbenchmarks for `SQL_GROUPED_MAP_PANDAS_ITER_UDF` to `python/benchmarks/bench_eval_type.py`:
- `_GroupedMapPandasIterBenchMixin` reuses the parent `_GroupedMapPandasBenchMixin` scenario configs and overrides `_udfs` and `_write_scenario` to dispatch to `SQL_GROUPED_MAP_PANDAS_ITER_UDF`.
- UDFs cover the iterator pattern (`Iterator[pandas.DataFrame] -> Iterator[pandas.DataFrame]`):
- `identity_udf`: yield from input iterator
- `sort_udf`: sort each DataFrame by first column
- `key_identity_udf`: 2-arg variant `(key, pdfs)` yielding from input iterator
- Two bench classes: `GroupedMapPandasIterUDFTimeBench` and `GroupedMapPandasIterUDFPeakmemBench`.
### Why are the changes needed?
Establish a baseline before refactoring `SQL_GROUPED_MAP_PANDAS_ITER_UDF` (subtask of [SPARK-55724](https://issues.apache.org/jira/browse/SPARK-55724)).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Ran ASV with `COLUMNS=120 asv run --bench GroupedMapPandasIterUDF --quick --python=same`, twice, numbers stable.
\`\`\`text
[ 0.00%] ·· Benchmarking existing-py_home_yicong.huang_venv312_bin_python
[25.00%] ··· bench_eval_type.GroupedMapPandasIterUDFPeakmemBench.peakmem_worker ok
[25.00%] ··· ================= ============== ========== ==================
-- udf
----------------- --------------------------------------------
scenario identity_udf sort_udf key_identity_udf
================= ============== ========== ==================
sm_grp_few_col 477M 478M 472M
sm_grp_many_col 485M 485M 484M
lg_grp_few_col 635M 635M 551M
lg_grp_many_col 767M 768M 766M
mixed_types 467M 467M 467M
================= ============== ========== ==================
[50.00%] ··· bench_eval_type.GroupedMapPandasIterUDFTimeBench.time_worker ok
[50.00%] ··· ================= ============== ========== ==================
-- udf
----------------- --------------------------------------------
scenario identity_udf sort_udf key_identity_udf
================= ============== ========== ==================
sm_grp_few_col 460+/-0ms 519+/-0ms 425+/-0ms
sm_grp_many_col 419+/-0ms 429+/-0ms 410+/-0ms
lg_grp_few_col 817+/-0ms 1.08+/-0s 689+/-0ms
lg_grp_many_col 994+/-0ms 1.08+/-0s 1.01+/-0s
mixed_types 462+/-0ms 566+/-0ms 433+/-0ms
================= ============== ========== ==================
\`\`\`
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56649][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_MAP_PANDAS_ITER_UDF [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #55600: [SPARK-56649][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_MAP_PANDAS_ITER_UDF
URL: https://github.com/apache/spark/pull/55600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56649][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_MAP_PANDAS_ITER_UDF [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55600:
URL: https://github.com/apache/spark/pull/55600#issuecomment-4344148039
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org