You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/10/08 09:39:56 UTC
[I] Add Spark Connect serving to hudi-lakehouse with ADBC client integrations [hudi]
cshuo opened a new issue, #20234:
URL: https://github.com/apache/hudi/issues/20234
## Summary
Provide an optional Helm-deployable Spark Connect endpoint for Hudi queries and writes, with ADBC access from hudi-agent-gateway, Superset, and Zeppelin.
This issue tracks the implementation discussed in [#19843](https://github.com/apache/hudi/discussions/19843). Local testing demonstrated Hudi reads and writes through Foundry ADBC with Superset and Zeppelin; the next step is to package the endpoint and integrations and validate them through Helm.
## Scope
- Provide a Hudi-enabled Spark Connect image and Helm chart, reusing the lakehouse catalog and object storage.
- Integrate hudi-agent-gateway through ADBC while preserving its existing tool interfaces and SELECT-only guardrails.
- Provide Superset and Zeppelin examples supporting direct queries and writes.
- Add repeatable end-to-end validation and a complete quickstart.
Direct-client write coverage includes table creation, INSERT, UPDATE, DELETE, and MERGE INTO. Agent gateway write tools are out of scope.
## Sub-issues
- [ ] #20231 — Add a Helm-deployable Spark Connect endpoint
- [ ] #20232 — Integrate hudi-agent-gateway and BI clients with Spark Connect
- [ ] #20233 — Add end-to-end validation and documentation for Spark Connect
## Completion Criteria
- A clean local Kubernetes environment can deploy the endpoint using the quickstart.
- Agent/MCP queries work through Gateway → ADBC → Spark Connect → Hudi, with read-only guardrails enforced.
- Superset and Zeppelin can query and write Hudi data, with result readback and rendering.
- Tests cover the scoped SQL operations, external-commit visibility, and persistence across endpoint restarts.
- Documentation records tested versions, setup, cleanup, and known limitations.
## Follow-up Work
Broader JDBC compatibility, gateway write tools, and production-scale security, concurrency, and isolation validation are separate follow-up work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Spark Connect serving to hudi-lakehouse with ADBC client integrations [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #20234:
URL: https://github.com/apache/hudi/issues/20234#issuecomment-6490037725
@nsivabalan Thanks, Siva. The four-phase split works for me. I’ve separated the issue scopes: #20232 for Agent/MCP, #20466 for Superset/Zeppelin, and #20233 for quickstart and validation.
For Phase 2, cancellation and timeout can stop Spark jobs under normal conditions. Cancellation should still be documented as best effort: for example, a network failure may prevent the interrupt request from reaching Spark, or cancellation may race with a job starting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Spark Connect serving to hudi-lakehouse with ADBC client integrations [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #20234:
URL: https://github.com/apache/hudi/issues/20234#issuecomment-6485859992
**Delivery plan for 2.0: four phases.** The target is Spark Connect serving for Hudi, shipped in 2.0 as *experimental* by the end of November.
Most of the code already exists as a stack: #20243 → #20329 → #20462. This plan reorganizes that stack by **what each phase delivers and who can use it**, not by PR order. Two changes from how the stack is set up today:
- **#20462 splits in two.** The Superset/Zeppelin adapter is a deliverable for BI users. The fault scripts, readiness probe and live CI answer a different question: is this safe to ship? Keeping them in one PR means BI support waits on review of ~1.3k lines of fault tests.
- **The gateway and the BI clients become parallel tracks.** Both depend only on the endpoint, so neither should wait on the other. The Superset adapter uses no gateway code and can be rebased directly onto #20243.
| Phase | Delivers | For | Status | Target |
|---|---|---|---|---|
| 1 | Spark Connect endpoint | Spark Connect / ADBC clients running Hudi SQL | #20243 approved | **Nov 3** |
| 2 | Read-only agent gateway on Spark Connect | Agent / MCP users | #20329 in review | **Nov 7** |
| 3 | BI clients: Superset, Zeppelin | BI users | In #20462; needs split | **Nov 14** |
| 4 | Validation and 2.0 release readiness | Release managers, operators | Rest of #20462; not started on 2.0 | **Nov 26** |
Phases 2 and 3 run in parallel after Phase 1.
### Phase 1: Spark Connect endpoint (#20231)
Scope: a Helm-deployed Spark Connect server with Hudi's SQL extensions and catalog, wired to the lakehouse HMS and object storage. Any Spark Connect client can run interactive Hudi SQL.
- ✅ M1.1 #20243: chart, image (Spark 4.1.3 + Hudi bundle), `hudi.defaults` writer defaults, local-dev quickstart integration, smoke / SQL-matrix / freshness / restart scripts, offline chart tests. Approved.
- ⬜ M1.2 Small README fix, before or after merge. With the Hive catalog, Spark SQL writes enable HMS sync automatically, so the per-table sync `TBLPROPERTIES` paragraph should be reworded.
**Acceptance criteria** (met on Hudi 1.2.0; Phase 4 reruns them on 2.0):
- CREATE, INSERT, UPDATE, DELETE, MERGE, CALL, time travel and incremental reads work through PySpark Connect and Foundry ADBC, with exact contents checked.
- Trino reads back CoW and partitioned tables written through the endpoint.
- Data persists across an endpoint restart. An open session sees external commits, and sees external schema changes after `REFRESH TABLE`.
- Kubernetes executor mode is opt-in and documented as unvalidated.
### Phase 2: read-only agent gateway on Spark Connect (#20232, gateway part)
Scope: `hudi-agent-gateway` can use Spark Connect instead of Trino for `list_tables`, `describe_table` and `query_lakehouse`. It stays read-only. Trino remains the default.
- ⬜ M2.1 #20329: Spark Connect backend, Spark-dialect guardrails, per-request sessions, bounded concurrency, timeout/cancel, separate readiness probe, `[spark-connect]` extra. Re-review pending on the table-function path fix, then rebase onto merged #20243.
- ⬜ M2.2 Result fidelity: keep timestamp timezones and nested struct field names in gateway JSON. Check decimal precision and truncation at the same time. Not in #20329 as far as I can see.
- ⬜ M2.3 Stop propagation: confirm that Stop in the chat UI cancels the Spark job and that the session stays usable. #20329 adds cancellation handling and `interruptAll`, so this may already work; it needs a test.
**Acceptance criteria:**
- Gateway table listing, descriptions and agent/MCP queries return correct results against the Helm-deployed endpoint.
- Writes, script transforms, reflection, file paths, Hudi table functions with a path argument, and `IDENTIFIER(...)` are rejected before reaching Spark.
- Stop and timeout cancel the Spark job, and later queries still work.
- Timestamps, decimals and structs round-trip correctly in tool results.
### Phase 3: BI clients (#20232, BI part)
Scope: Superset and Zeppelin can query and write Hudi tables through the endpoint over ADBC.
- ⬜ M3.1 New PR split from #20462, based on #20243: everything under `hudi-lakehouse/superset-spark-connect/`. That covers the SQLAlchemy dialect, the Superset engine spec, the Dockerfile, the README and screenshots, and the dialect, Superset and Zeppelin tests. About 820 lines.
- ⬜ M3.2 Link the BI section of the chart README to it.
**Acceptance criteria:**
- Superset SQL Lab and Zeppelin `%python` can query, run INSERT/UPDATE/DELETE/MERGE, and read back the changes, with exact contents asserted.
- A Superset dashboard refreshes after a write.
- Dependencies are pinned. No upstream driver binaries are modified. The adapter is labeled experimental, and upstreaming it is evaluated separately.
### Phase 4: validation and 2.0 release readiness (#20233)
Scope: everything needed to ship the above in 2.0 as experimental.
- ⬜ M4.1 The rest of #20462:
- fault scripts and regression tests: cancel, disconnect, lost response, restart, load, nullable array;
- SQL-aware readiness (`check-ready.py` plus the deployment probe), with separate HMS and storage checks;
- the Spark 3.5 writer image for cross-version external writes;
- the opt-in live workflow `hudi_spark_connect_live.yml`;
- validation docs.
- ⬜ M4.2 Remaining #20233 checks:
- `TIMESTAMP_NTZ` and `_hoodie_*` type mapping;
- the 1M-row `SELECT *` with the default 2g driver heap;
- triage of the NULL-array Parquet write failure. If it is a Hudi write bug, file it separately.
- ⬜ M4.3 Rebuild the image against the 2.0 RC bundle (the current tag is `4.1.3-hudi1.2.0`) and rerun the Phase 1–3 acceptance on it as part of RC validation.
- ⬜ M4.4 At least one green run of the live workflow on GitHub runners. If possible, share the Connect server setup with the RFC-112 SDP harness (#20402 / #20453).
- ⬜ M4.5 A second person runs the quickstart on a clean cluster through all three client paths.
- ⬜ M4.6 Release notes and a known-limitations page:
- experimental status;
- no authentication or TLS, so trusted networks only, including the driver's Kubernetes permissions in executor mode;
- the retry and cancel caveats;
- `REFRESH TABLE` after external schema changes;
- tested versions;
- verified versus untested behavior.
**Acceptance criteria:**
- The 2.0 RC image passes the full Phase 1–3 acceptance, with results posted here.
- Disconnect, lost-response and restart behavior can be reproduced from the scripts and is documented as observed behavior, not a guarantee.
- Readiness fails when HMS or object storage is unreachable.
- The live workflow has at least one green run.
- The quickstart works end to end for someone other than the author.
### Target dates
- **Phase 1: Nov 3.** Merge once the README fix is in or tracked.
- **Phase 2: Nov 7.** Gated on the #20329 re-review and M2.2.
- **Phase 3: Nov 14.** Starts once #20462 is split. It runs in parallel with Phase 2.
- **Phase 4: Nov 26.** M4.1 and M4.2 can start now. M4.3 depends on when the 2.0 RC is cut and moves with it.
Phase 3 and M4.5 are the easiest pieces to hand to a second contributor. If a phase is at risk, let's flag it here early rather than at the deadline.
### Out of scope for 2.0 (follow-up work)
- Validated Kubernetes executor mode and dynamic allocation
- Authentication, TLS, multi-tenant isolation, production-scale concurrency
- Spark UI, history server, metrics, and correlating client requests with Spark jobs and Hudi commits
- Cross-version external writers beyond the Spark 3.5 fixture (Flink)
- Broader JDBC compatibility, native Zeppelin `%jdbc`, upstreaming the Superset adapter
- Write tools in the gateway
@cshuo does this split work for you? In particular, can #20462 be split into the BI adapter (M3.1) and the validation work (M4.1)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Spark Connect serving to hudi-lakehouse with ADBC client integrations [hudi]
Posted by "cshuo (via GitHub) via commits" <co...@hudi.apache.org>.
cshuo closed issue #20234: Add Spark Connect serving to hudi-lakehouse with ADBC client integrations
URL: https://github.com/apache/hudi/issues/20234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org