You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/09/12 17:36:01 UTC

[PR] test(spark): shared SparkContext for the SQL suites, opt-in [hudi]

voonhous opened a new pull request, #19926:
URL: https://github.com/apache/hudi/pull/19926

### Describe the issue this Pull Request addresses

Part of #19524, stacked on #19919 (the CI stack's commits are included until it merges). Supersedes the proof of concept #19921, which branched from master. The Java CI wall clock is set by the Scala SQL shards; scalatest has no fork count, so the lever is running suites concurrently in one JVM, which needs one shared SparkContext. This PR makes that shared lifecycle exist behind a property and runs it serially; #19923's successor removes the cross-suite state and the third PR turns the concurrency on.

### Summary and Changelog

- New pom property `hudi.spark.test.sharedSession` (default `false`), forwarded to the scalatest JVM. With it on, every suite on `HoodieSparkSqlTestBase` shares one never-stopped SparkContext and works in its own `SparkSession.newSession()` child: own SQL conf, temp views and HoodieCatalog; shared external catalog and warehouse.
- A suite's conf deltas (`extraConf`, `sparkConf()` overrides) are validated first, then applied to the child session; `spark.hadoop.*` keys go to the shared Hadoop conf and are restored in `afterAll`; any other context-level or static key is rejected before anything mutates.
- The child session is pinned as the active session on the suite thread, with a fail-fast check that the session's HoodieCatalog is bound to it. The per-test cleanup drops only the suite's own tables.
- Switched on for the three Java 17 Scala shards whose packages hold only `HoodieSparkSqlTestBase` suites: dml-1, dml-2, other-1. other-2 mixes in suites that create their own SparkContext and stays on the default.
- `TestHoodieDataUtils` no longer closes its JavaSparkContext in `afterAll` (it stopped the context).

With the property off, nothing changes.

<details>
<summary>Evidence from the proof of concept (#19921)</summary>

On master's layout the dml job ran in shared mode, serially, at master's own time: 51:56 and 53:08 ([run 34676800123](https://github.com/apache/hudi/actions/runs/34676800123), attempts 1 and 2) against 52 to 56 minutes for the same job on seven recent master runs (41:22 on one fast runner), with the same 326 tests, 4 canceled and 1 ignored. Two review rounds on #19921 are folded in: the conf deltas are validated before any mutation, static SQL confs are rejected, and generateTableName builds on the prefix the cleanup relies on.
</details>

### Impact

Test infrastructure only. No production code changes.

### Risk Level

low. Default behaviour is unchanged; the shared mode runs on three CI shards and was verified on #19921.

### Documentation Update

none

### Contributor's checklist

- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org