You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/04/21 22:23:36 UTC

[PR] [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF [spark]

Yicong-Huang opened a new pull request, #55467:
URL: https://github.com/apache/spark/pull/55467

   ### What changes were proposed in this pull request?
   
   Add ASV microbenchmarks for `SQL_GROUPED_AGG_PANDAS_UDF` in `bench_eval_type.py`.
   
   ### Why are the changes needed?
   
   Part of [SPARK-55724](https://issues.apache.org/jira/browse/SPARK-55724) (Micro-benchmark PySpark Eval Types). Establishes a performance baseline for `SQL_GROUPED_AGG_PANDAS_UDF` before refactoring.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   `COLUMNS=120 asv run --python=same --bench "GroupedAggPandas" --attribute "repeat=(3,5,10.0)"`:
   
   **GroupedAggPandasUDFTimeBench** (`SQL_GROUPED_AGG_PANDAS_UDF`):
   ```
   ================ ============ ================
   --                            udf
   ---------------- -----------------------------
       scenario       sum_udf     mean_multi_udf
   ================ ============ ================
    few_groups_sm    33.5+/-0.2ms   36.0+/-0.2ms
    few_groups_lg    52.9+/-0.3ms   59.9+/-2ms
    many_groups_sm   1.23+/-0.01s   1.32+/-0.01s
    many_groups_lg    350+/-3ms     379+/-1ms
      wide_cols       359+/-1ms    378+/-0.7ms
   ================ ============ ================
   ```
   
   **GroupedAggPandasUDFPeakmemBench** (`SQL_GROUPED_AGG_PANDAS_UDF`):
   ```
   ================ ========= ================
   --                          udf
   ---------------- --------------------------
       scenario      sum_udf   mean_multi_udf
   ================ ========= ================
    few_groups_sm     472M          470M
    few_groups_lg     535M          533M
    many_groups_sm    501M          499M
    many_groups_lg    606M          604M
      wide_cols       587M          585M
   ================ ========= ================
   ```
   
   Second run (stability check):
   ```
   time:   sum_udf few_groups_sm 33.5+/-0.5ms, mean_multi few_groups_sm 35.8+/-0.09ms, ...
   peak:   identical within 1-2M across all scenarios
   ```
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55467:
URL: https://github.com/apache/spark/pull/55467#issuecomment-4293306603

   merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #55467: [SPARK-56562][PYTHON][TESTS] Add ASV microbenchmark for SQL_GROUPED_AGG_PANDAS_UDF
URL: https://github.com/apache/spark/pull/55467


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org