You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2026/10/28 05:13:07 UTC
[PR] test(spark): add Spark Connect SDP integration harness and CI coverage [hudi]
danny0405 opened a new pull request, #20453:
URL: https://github.com/apache/hudi/pull/20453
### Describe the issue this Pull Request addresses
Closes #20402. RFC-112 needs repeatable coverage of the real `spark-pipelines` CLI against a separately configured Spark Connect server, using the candidate Hudi bundle. Scala dataflow-graph tests do not exercise that remote boundary. The general Spark 4.1 entries in the current `bot.yml` matrices are disabled.
### Summary and Changelog
- Add a reusable Python harness that launches a foreground Spark 4.1.1 Connect server with the built Hudi bundle, extension, catalog, Kryo configuration, and isolated local storage. Pin the client environment to `pyspark[pipelines,connect]==4.1.1`.
- Run a deterministic JSON-to-Hudi streaming pipeline through the actual CLI, then verify exact rows through a separate Connect client using both catalog and Hudi path reads. Check Hudi metadata and an SDP-managed streaming checkpoint.
- Bound startup and client execution, detect server exits, retain client/server logs, and clean up temporary resources and owned process groups on success, failure, timeout, or interruption.
- Add a focused Spark 4.1 / Scala 2.13 / JDK 17 CI job in `bot.yml` that builds the candidate bundle, runs lifecycle regressions and the remote harness, and uploads diagnostics.
Validation:
- Spark 4.1 candidate bundle reactor build passed with JDK 17 (`mvn package -T 2 -Pjava17 -Dspark4.1 -Dscala-2.13 -DskipTests -Dgpg.skip -Djacoco.skip -am -pl packaging/hudi-spark-bundle`).
- Real CLI/Connect smoke test passed locally with that bundle and PySpark 4.1.1: the Hudi flow completed, both reads returned all three expected rows, and checkpoint/resource checks passed.
- All five standard-library lifecycle tests passed, covering client failure diagnostics, timeout isolation from an unrelated process, server exit, forced termination, and cleanup after startup failure.
- Python syntax, workflow YAML and shell-step syntax, `git diff --check`, and Apache RAT passed. The PR description was validated with `scripts/pr_compliance.py` before publication.
### Impact
Adds test infrastructure and one CI lane; no production APIs, storage formats, defaults, or writer semantics change. The smoke test establishes the remote execution baseline. Repeat materialized-view refresh, streaming option forwarding, restart/full-refresh, chaining, and keyed update/delete acceptance remain with the sibling tasks under #20401.
### Risk Level
Low. The new CI lane adds a Spark bundle build, a Python dependency installation, and a local Connect server. Process-group cleanup is scoped to harness-owned processes, clients have explicit time limits, and diagnostics are retained. Local end-to-end and lifecycle checks passed; the new Ubuntu CI lane has not yet completed on GitHub.
### Documentation Update
Add `hudi-integ-test/spark-pipelines/README.md` with local build/install/run commands, server versus session/flow configuration ownership, timeout and cleanup behavior, log locations, CI integration, and instructions for adding sibling scenarios. No website update is needed for this developer test harness; this PR does not declare a new supported runtime feature.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): add Spark Connect SDP integration harness and CI coverage [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #20453:
URL: https://github.com/apache/hudi/pull/20453#issuecomment-6701774199
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): add Spark Connect SDP integration harness and CI coverage [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #20453:
URL: https://github.com/apache/hudi/pull/20453#discussion_r4405763921
##########
.github/workflows/bot.yml:
##########
@@ -65,6 +65,55 @@ env:
SCALA_TEST_OTHERS_FILTER: -DwildcardSuites=org.apache.hudi,org.apache.spark.hudi,org.apache.spark.sql.avro,org.apache.spark.sql.execution,org.apache.spark.sql.hive,org.apache.spark.sql.hudi.analysis,org.apache.spark.sql.hudi.blob,org.apache.spark.sql.hudi.catalog,org.apache.spark.sql.hudi.command,org.apache.spark.sql.hudi.common,org.apache.spark.sql.hudi.ddl,org.apache.spark.sql.hudi.procedure,org.apache.spark.sql.hudi.feature
jobs:
+ # Spark 4.1 entries in the general test matrices below are disabled by CI-TRIM.
+ # Keep SDP's pinned 4.1.1 client/server and candidate bundle in one focused lane.
+ test-spark-connect-pipelines:
Review Comment:
This lane runs on every PR and push that changes any non-doc file, and each run builds the Spark 4.1 bundle with `-am` and installs PySpark (~400 MB) before the smoke test. That brings back a good part of the Spark 4.1 cost that CI-TRIM removed, including for PRs that cannot affect SDP (Flink, Trino, docs-adjacent changes). Could we scope it, e.g. move it to its own workflow with a `paths:` filter (`hudi-spark-datasource/**`, `hudi-client/**`, `hudi-common/**`, `packaging/hudi-spark-bundle/**`, `hudi-integ-test/spark-pipelines/**`, the workflow file), and/or run it on push to master plus a nightly schedule? Also worth checking that 40 minutes is enough for a cold-cache `-am` bundle build on `ubuntu-24.04`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): add Spark Connect SDP integration harness and CI coverage [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #20453:
URL: https://github.com/apache/hudi/pull/20453#issuecomment-6480221894
Following up on the CI question with a concrete proposal, since the harness itself looks good to me:
1. **Own workflow file** (e.g. `.github/workflows/spark_connect_pipelines.yml`) instead of a job in `bot.yml`, so its triggers, failures and ownership stay separate from the main Java CI.
2. **Run nightly** (`schedule`) plus `workflow_dispatch` for manual runs. Optionally, PRs touching SDP could opt in through a label, so authors of the #20401 sub-tasks can run it before merge.
3. **Not a required check.** A red nightly run files or updates an issue for the SDP owners rather than blocking merges.
4. **Revisit later:** once the #20401 sub-tasks have landed and the job has a stable track record, we can consider path-filtered per-PR runs.
This keeps the real CLI/Connect coverage RFC-112 asks for, without bringing back a per-PR Spark 4.1 bundle build for every change while the RFC is still under review.
@danny0405 does that work for you? @yihua would like your take as well, especially on adding a nightly Spark Connect / Python lane to the project's CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): add Spark Connect SDP integration harness and CI coverage [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20453:
URL: https://github.com/apache/hudi/pull/20453#issuecomment-6461452741
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20453?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.82%. Comparing base ([`82d724b`](https://app.codecov.io/gh/apache/hudi/commit/82d724b93b552b822a968ba4cbc7fdf3360c4ccb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2a0616a`](https://app.codecov.io/gh/apache/hudi/commit/2a0616a1c6f471a7eba10061bc8c24bdaef0cc27?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20453 +/- ##
============================================
- Coverage 80.82% 80.82% -0.01%
+ Complexity 36028 36027 -1
============================================
Files 2575 2575
Lines 147258 147258
Branches 18097 18097
============================================
- Hits 119020 119018 -2
- Misses 20035 20036 +1
- Partials 8203 8204 +1
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.18% <ø> (-0.01%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.03% <ø> (ø)` | |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.16% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.18% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.26% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.18% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.31% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.57% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.01% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20453/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.90% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20453/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20453/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.93% <ø> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1 file with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20453/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): add Spark Connect SDP integration harness and CI coverage [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #20453:
URL: https://github.com/apache/hudi/pull/20453#issuecomment-6485349284
Following up on the CI-placement discussion, I looked at how Apache Spark tests Connect and declarative pipelines, to see what we could borrow. Spark uses three layers:
1. **Python unit tests** (`python/pyspark/pipelines/tests`). These use `ReusedConnectTestCase`, which calls `.remote("local[4]")` so PySpark boots a local Connect server in-process. They call `create_dataflow_graph` / `start_run` directly, not the CLI.
2. **End-to-end CLI tests driven from ScalaTest** ([`EndToEndAPISuite`](https://github.com/apache/spark/blob/32dd961c36decf1d5f9799d0e87592d1dd94b939/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/EndToEndAPISuite.scala)). `SparkConnectServerTest` starts the Connect server inside the test JVM on a random port. Each test then:
- writes the spec and sources to a temp dir;
- runs the real `pyspark/pipelines/cli.py` as a subprocess against `sc://localhost:<port>`;
- waits with a timeout and fails with the CLI's stdout/stderr on a non-zero exit;
- skips via `assume(...)` when the Python deps are missing.
The scenarios live in a shared `APITest` trait so they can be reused with other runners.
3. **A separately launched server process.** On the JVM side this is `RemoteSparkSession`, which uses `spark-submit` via `ProcessBuilder`. On the Python side it is [`build_python_connect.yml`](https://github.com/apache/spark/blob/32dd961c36decf1d5f9799d0e87592d1dd94b939/.github/workflows/build_python_connect.yml), which runs `start-connect-server.sh` plus the client tests on a cron schedule, not per PR.
This harness is layer 3, and that is the layer Spark runs on a schedule. So my proposal is to split the coverage:
**Per PR: a small layer-2 ScalaTest in `hudi-spark` (Spark 4.1 profile), modelled on `EndToEndAPISuite`**
- Start a Connect server in the test JVM with Hudi already on the classpath (`SparkConnectService.start(spark.sparkContext)` on the shared session). Spark's `SparkConnectServerTest` is test-only, so we'd copy the ~30 lines of setup.
- Run the pipelines CLI from the pip-installed PySpark 4.1.1 (same `requirements.txt`) as a subprocess against `sc://localhost:<port>`, with a timeout, and fail with its output.
- `assume(...)` when PySpark is not installed, so local `mvn test` runs without Python skip instead of fail.
- Keep it to 2–3 scenarios that exercise the Hudi wiring through the real CLI:
- an MV plus a streaming table writing to Hudi;
- a per-flow `SET` reaching the sink (#20403 / #20455);
- a failing flow surfacing a clear error.
- It runs inside the existing Maven test jobs, so it needs no bundle build, no `spark-submit` and no new CI lane. The only extra CI step is a cached `pip install` in the Spark 4.1 job.
**Nightly: this harness (layer 3), as already proposed above**
- It keeps the cases only a separately launched server can show:
- static settings applied at server launch;
- the shared server surviving a failed flow (#20370 / #20452);
- resources released across runs (#20400);
- the real bundle jar on a real `spark-submit` classpath.
- Same blocking policy as in my earlier comment: a red nightly blocks merges until fixed, with the escape hatch for the fix PR.
With this split, regressions in the Hudi sink/SDP wiring are caught on the PR that introduces them at almost no extra CI cost. The heavier process-level checks still run every night. @danny0405 would you be open to adding the layer-2 suite, either in this PR or as a follow-up? @yihua thoughts welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): add Spark Connect SDP integration harness and CI coverage [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #20453: test(spark): add Spark Connect SDP integration harness and CI coverage
URL: https://github.com/apache/hudi/pull/20453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org