You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/07/09 07:24:27 UTC
[PR] [SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules [spark]
HyukjinKwon opened a new pull request, #57155:
URL: https://github.com/apache/spark/pull/57155
### What changes were proposed in this pull request?
This PR splits several large/slow PySpark test suites into smaller modules so
they run as separate test targets and parallelize better, reducing wall-clock
time. No test logic is changed; test methods are only relocated, following the
existing convention already used in the repo (e.g. `test_split_apply_*`,
`test_stat_*`).
The following suites are split, each with a matching Spark Connect parity module:
- `pyspark.sql.tests.pandas.test_pandas_cogrouped_map`: move the heavier,
self-contained tests (`test_with_window_function`, `test_with_local_data`,
`test_arrow_batch_slicing`, `test_cogroup_apply_in_pandas_with_logging`) into
a new `test_pandas_cogrouped_map_misc` module.
- `pyspark.sql.tests.arrow.test_arrow_cogrouped_map`: extract shared helpers
into `CogroupedMapInArrowTestsFuncMixin` and move the heavier tests into a new
`test_arrow_cogrouped_map_misc` module.
- `pyspark.pandas.tests.data_type_ops.test_num_ops`: move
`IntegralExtensionOpsTestsMixin` and `FractionalExtensionOpsTestsMixin` into
their own modules.
- `pyspark.pandas.tests.groupby.test_stat`: move `test_median` into a new
`test_stat_median` module.
- `pyspark.pandas.tests.groupby.test_split_apply`: move the `mean` case into a
new `test_split_apply_mean` module.
New modules are registered in `dev/sparktestsupport/modules.py`.
### Why are the changes needed?
These suites are slow in CI. Splitting them lets the test runner schedule the
pieces in parallel, lowering total wall-clock time without dropping coverage.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing tests, only relocated across modules. Verified that the total number
of test methods is unchanged for each split suite, that all new and modified
modules import and collect, and that `flake8` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57155:
URL: https://github.com/apache/spark/pull/57155#issuecomment-4924460892
Note on the one red job (`pyspark-sql, pyspark-core, ...`): the only failure is
`pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries`
(`Unexpected 3rd party package 'numpy' imported during 'import pyspark'`). This
is pre-existing on `master` and unrelated to this test-only change:
- `test_import_spark.py` is not part of this PR's diff.
- Reproduced on a clean checkout of the base commit (no changes applied): a bare
`import pyspark` pulls in `numpy` via
`pyspark.sql.streaming.stateful_processor_api_client`, which the test's
allowlist (stdlib + `py4j`) rejects.
All of the split/rebalanced modules in this PR pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57155:
URL: https://github.com/apache/spark/pull/57155#issuecomment-4925139868
### num_ops rebalance — measured before/after (CI durations, classic / Spark Connect)
The earlier revision left the extension modules heavier than the base (Connect: integral_ext 50s, fractional_ext 40s vs base 28s). The heavy work was `test_astype` (looped dtype conversions) plus the 6 comparison ops (`eq`/`ne`/`lt`/`le`/`gt`/`ge`, each using `ops_on_diff_frames`). These were split into dedicated `*_astype_cmp` modules:
| Module | before | after |
| --- | --- | --- |
| num_ops (base) | 27s / 28s | 26s / 25s |
| num_ops_integral_ext | 37s / 50s | 16s / 21s |
| num_ops_integral_ext_astype_cmp | — | 34s / 47s |
| num_ops_fractional_ext | 26s / 40s | 13s / 14s |
| num_ops_fractional_ext_astype_cmp | — | 26s / 36s |
Peak single-module time drops (Connect 50s→47s, classic 37s→34s) and the spread is flatter. The remaining ~47s in `integral_ext_astype_cmp` is dominated by `test_astype`'s dtype-conversion loop (a single test method), which is the practical floor without rewriting the test itself.
All split/rebalanced modules pass. (The unrelated `test_import_spark` numpy failure is pre-existing on master, as noted above.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57155:
URL: https://github.com/apache/spark/pull/57155#issuecomment-4926524369
I also agree that `pyspark.tests.test_import_spark` failure is irrelevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57155:
URL: https://github.com/apache/spark/pull/57155#issuecomment-4929841151
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/79f6dfa4b99865f581d9c073bddd352e666bcb47
- merged into branch-4.x https://github.com/apache/spark/commit/89b66fe831f1f3a82dd7c9b629cb0197297203cc
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57155:
URL: https://github.com/apache/spark/pull/57155#issuecomment-4923606823
### CI timing results (run on this PR)
All split modules pass. The one red job (`pyspark-sql, pyspark-core, ...`) failed only on `pyspark.tests.test_import_spark` (`Unexpected 3rd party package 'numpy' imported during 'import pyspark'`), which is unrelated to this test-only change and not part of this diff.
Measured per-module durations (classic / Spark Connect):
| Suite | base | new module(s) |
| --- | --- | --- |
| arrow cogrouped map | 22s | misc 23s |
| pandas cogrouped map | 41s | misc 25s |
| num_ops | 27s / 28s | integral_ext 37s / 50s, fractional_ext 26s / 40s |
| groupby stat | 27s / 36s | stat_median 12s / 15s |
| split_apply | 29s / 41s | split_apply_mean 29s / 41s |
Each split lowers the longest single-module wall-clock via parallelism. `arrow` and `split_apply` are well balanced; `num_ops`, `groupby stat`, and `pandas cogrouped map` are functional splits but not tightly balanced (documented here for transparency; can be rebalanced in a follow-up if desired).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #57155: [SPARK-58058][PYTHON][TESTS] Split slow PySpark test suites into smaller modules
URL: https://github.com/apache/spark/pull/57155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org