You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vh...@apache.org on 2026/08/05 07:35:30 UTC
(hudi) 12/12: feat(trino): [RFC-105] Trino Hudi Connector — Shim/Bundle Refactor (#18782)
This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git
commit fbb5c1f1d90e7411a3d51e1e9355bbd79ed686fa
Author: Y Ethan Guo <et...@gmail.com>
AuthorDate: Tue May 26 20:12:18 2026 -0700
feat(trino): [RFC-105] Trino Hudi Connector — Shim/Bundle Refactor (#18782)
(cherry picked from commit e5ae1115458c99696399b7df62d48f6c00764426)
---
rfc/rfc-105/rfc-105.md | 225 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 225 insertions(+)
diff --git a/rfc/rfc-105/rfc-105.md b/rfc/rfc-105/rfc-105.md
new file mode 100644
index 000000000000..51c7aef2fb90
--- /dev/null
+++ b/rfc/rfc-105/rfc-105.md
@@ -0,0 +1,225 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# RFC-105: Trino Hudi Connector — Shim/Bundle Refactor
+
+## Proposers
+
+- @yihua
+- @voonhous
+
+## Approvers
+
+- @codope
+- @vinothchandar
+
+## Status
+
+Issue: [HUDI-18780](https://github.com/apache/hudi/issues/18780)
+
+> Please keep the status updated in `rfc/README.md`.
+
+## Motivation
+
+The Trino-Hudi connector currently lives in `trinodb/trino` at `plugin/trino-hudi`. Maintaining and evolving the connector through the Trino-OSS-only path has stalled in practice, and the cost falls on Hudi users:
+
+- **Hudi-side improvement PRs to the Trino Hudi connector are not landing.** Four stacked PRs targeting the Trino Hudi connector were closed by Trino's stale-bot for lack of review:
+ - [trinodb/trino#28518](https://github.com/trinodb/trino/pull/28518)
+ - [trinodb/trino#28533](https://github.com/trinodb/trino/pull/28533)
+ - [trinodb/trino#28644](https://github.com/trinodb/trino/pull/28644)
+ - [trinodb/trino#28645](https://github.com/trinodb/trino/pull/28645)
+- **Significant Hudi-side work for the Trino connector is ready but cannot land** through the current path: metadata-table-driven partition listing, eight `HudiIndexSupport` strategies (column stats, partition stats, record-level, secondary, expression, bloom, bucket, partition bloom), MOR snapshot-isolation fixes (worker-side use of the latest commit time from the table handle), and file-system caching integration.
+- **The current arrangement does not scale.** Connector evolution must go through Trino-side review for every change, while the expertise and the source-of-truth for Hudi internals live in this project. Hudi releases cannot directly deliver improvements to Hudi users querying via Trino.
+
+Following alignment between the Hudi and Trino communities, the agreed direction is to split the connector into a thin Trino-side shim plus a Hudi-published artifact carrying the connector logic. This lets the Hudi project ship Trino-Hudi improvements with each Hudi release, while Trino picks them up via a one-line dependency-version bump.
+
+The single requirement carried over from the Trino side is that a comprehensive test suite for the connector continues to be maintained on the Trino side. This RFC documents the agreed approach and the implementation plan.
+
+## Abstract
+
+We split the Trino-Hudi connector into two Maven artifacts:
+
+1. **`io.trino:trino-hudi`** stays in Trino OSS (`plugin/trino-hudi`) as a thin shim — a `HudiPlugin` class that registers the `io.trino.spi.Plugin` SPI entry point — plus the test harness (smoke tests, query runners, MinIO-backed integration tests). This module mostly does not change once landed.
+2. **`org.apache.hudi:hudi-trino`** is a new Hudi-published Maven artifact (regular, non-shaded JAR) containing the actual connector logic at `io.trino.plugin.hudi.*` — `HudiConnectorFactory`, `HudiConnector`, `HudiMetadata`, `HudiSplitManager`, `HudiPageSourceProvider`, all index-support strategies, the `HoodieStorage`/`HoodieIOFactory` bridges to Trino's filesystem, etc. The artifact is built against the latest Trino release's SPI; it declares `hudi-common`, `hudi-io`, etc. as transiti [...]
+
+The first publication ships in **Hudi 1.3.0**. The Trino-side shim PR pins `org.apache.hudi:hudi-trino:1.3.0`. Going forward, all Trino-Hudi connector evolution happens in Hudi OSS; Trino picks up changes by bumping the dependency version. To support this integration model, **Hudi will increase its release cadence**.
+
+## Background
+
+### State of the Trino-side connector today
+
+`plugin/trino-hudi` in `trinodb/trino` is the baseline: it implements the standard Trino SPI (`Plugin`, `ConnectorFactory`, `Connector`, `ConnectorMetadata`, `ConnectorSplitManager`, `ConnectorPageSourceProvider`, etc.), depends on `hudi-common` and `hudi-io`, and uses Hudi's `HoodieStorage` abstraction (RFC-74) over Trino's `TrinoFileSystem`. No direct Hadoop imports.
+
+### State of the Hudi-side `hudi-trino-plugin` work
+
+A more advanced version of the connector exists in Hudi-side branches under `hudi-trino-plugin/` (same `io.trino.plugin.hudi.*` package, built against a recent Trino release). On top of the Trino-OSS baseline it adds:
+
+- Eight `HudiIndexSupport` strategies (column stats, partition stats, record-level, secondary, expression, bloom, bucket, partition bloom) for file- and partition-level pruning via metadata tables.
+- Metadata-table-driven partition discovery (async, resumable).
+- MOR record-level merging via `HoodieFileGroupReader` (`HudiTrinoReaderContext`).
+- Lazy commit-time on `HudiTableHandle` for snapshot-isolated MOR reads across workers.
+- Background, weighted split generation; size-based split weighting; multi-reader routing (`HudiPageSource` for MOR, `HudiBaseFileOnlyPageSource` for COW/RO).
+- File-system cache integration.
+- HoodieStorage / HoodieIOFactory bridges over `TrinoFileSystem` (`HudiTrinoStorage`, `HudiTrinoInlineStorage`, `HudiTrinoIOFactory`).
+
+This is the body of code that will move into the `hudi-trino` Maven module on the Hudi side.
+
+### Why a "shim + Hudi-published artifact" pattern
+
+This pattern decouples Trino-Hudi connector evolution from the Trino-side release cycle:
+
+- The Hudi project can publish Trino-Hudi improvements with each Hudi release, without waiting for Trino-side reviews of every change.
+- The Trino-side surface shrinks to a stable plugin-registration shim, so Trino-side review burden is minimal — typically a one-line version bump per Hudi release.
+- All Hudi-Trino integration code (`io.trino.plugin.hudi.*`) is co-located with the Hudi core libraries it depends on. Changes that cross the Hudi-internal / connector boundary can land atomically.
+- The artifact is **purpose-built for Trino** and implements Trino's SPI directly, so no intermediate adapter layer is needed between the published artifact and the Trino plugin.
+
+Trino's `trino-spi` is governed by `revapi-maven-plugin` (see `core/trino-spi/pom.xml`) which enforces backward compatibility on the SPI surface. This is what makes a single `hudi-trino` artifact targeting the latest Trino release viable across multiple subsequent Trino releases.
+
+Trino loads each plugin in an isolated `URLClassLoader`. Transitive dependencies of `hudi-trino` (Avro, Parquet, etc.) are isolated to the plugin's classloader and cannot conflict with other plugins.
+
+## Implementation
+
+### Architecture
+
+```
+trinodb/trino : plugin/trino-hudi (packaging = trino-plugin)
+ HudiPlugin.java ← thin shim: trivial Plugin SPI registration
+ META-INF/services/io.trino.spi.Plugin
+ src/test/java/... ← full Trino-side test suite
+ pom.xml ← depends on org.apache.hudi:hudi-trino:1.3.0
+ │
+ │ Maven Central
+ ▼
+apache/hudi : hudi-trino-plugin/ (Maven profile -Phudi-trino,
+ excluded from default reactor,
+ JDK 25 required)
+ io.trino.plugin.hudi.* ← all connector logic:
+ HudiConnectorFactory, HudiConnector, HudiMetadata,
+ HudiSplitManager, HudiPageSourceProvider,
+ cache/, file/, io/, partition/,
+ query/ (incl. 8 index-support strategies),
+ reader/, split/, stats/, storage/, util/
+ src/test/java/... ← full duplicated + expanded suite
+ Published as: org.apache.hudi:hudi-trino:1.3.0
+```
+
+### What lives where
+
+#### Trino-side `plugin/trino-hudi` (the shim)
+
+| File | Purpose |
+|---|---|
+| `src/main/java/io/trino/plugin/hudi/HudiPlugin.java` | Implements `io.trino.spi.Plugin`. Single method returning `new HudiConnectorFactory()` (from the `hudi-trino` artifact). ~10 lines. |
+| `src/main/resources/META-INF/services/io.trino.spi.Plugin` | Service-loader pointer to `io.trino.plugin.hudi.HudiPlugin`. |
+| `pom.xml` | `<packaging>trino-plugin</packaging>`; pins `org.apache.hudi:hudi-trino:<version>`; SPI deps as `provided`. |
+| `src/test/java/...` | All current Trino-side tests stay: `HudiQueryRunner`, `TestHudiSmokeTest`, `TestHudiMinioConnectorSmokeTest`, `TestHudiConnectorTest`, `TestHudiSharedMetastore`, `TestHudiSystemTables`, `TestHudiPlugin`, `TestHudiConfig`, plus data initializers. Required by the Trino-side test-coverage commitment. |
+
+#### Hudi-side `hudi-trino-plugin/` (the engine)
+
+Everything else from the current `hudi-trino-plugin/` work, organized exactly as it is today:
+
+| Subpackage | Responsibility |
+|---|---|
+| `io.trino.plugin.hudi` | `HudiConnectorFactory`, `HudiConnector`, `HudiMetadata`, `HudiSplitManager`, `HudiPageSourceProvider`, `HudiSplit`, `HudiTableHandle`, `HudiModule`, `HudiConfig`, `HudiSessionProperties`, `HudiTableProperties`, `HudiTransactionManager`, `HudiMetadataFactory`. |
+| `.cache` | `HudiCacheKeyProvider` for file-system cache integration. |
+| `.file` | `HudiBaseFile`, `HudiLogFile`, file metadata abstractions. |
+| `.io` | `HudiTrinoIOFactory` (extends `HoodieIOFactory`), `HudiTrinoFileReaderFactory`, `TrinoSeekableDataInputStream`. |
+| `.partition` | `HudiPartitionInfo`, `HiveHudiPartitionInfo`, `HudiPartitionInfoLoader` (async resumable task). |
+| `.query` | `HudiDirectoryLister`, `HudiReadOptimizedDirectoryLister`, `HudiSnapshotDirectoryLister`; `query.index` package with 8 `HudiIndexSupport` strategies. |
+| `.reader` | `HudiTrinoReaderContext extends HoodieReaderContext<IndexedRecord>` for MOR record merging. |
+| `.split` | `HudiSplitFactory`, `HudiBackgroundSplitLoader`, `HudiSplitSource`, `HudiSplitWeightProvider`, `SizeBasedSplitWeightProvider`. |
+| `.stats` | `HudiTableStatistics`, `TableStatisticsReader`. |
+| `.storage` | `HudiTrinoStorage` (extends `HoodieStorage`), `HudiTrinoInlineStorage`, `TrinoStorageConfiguration`. |
+| `.util` | Serialization helpers, column synthesis, tuple-domain conversion, table-type utilities. |
+
+### API boundary
+
+The boundary between the shim and the published artifact is **Trino's SPI itself** — no intermediate API layer is introduced.
+
+- **Shim → artifact:** `HudiPlugin.getConnectorFactories()` returns `new HudiConnectorFactory()` defined in the artifact. Trino's runtime then calls `factory.create(catalogName, config, context)`. The `ConnectorContext` argument carries everything the artifact needs — `TypeManager`, `NodeManager`, `MetadataProvider`, `PageSorter`, `PageIndexerFactory`, `OpenTelemetry`, `Tracer`, `CatalogHandle` — without the artifact importing implementation classes.
+- **Artifact → Trino:** the artifact's `HudiConnector` exposes the standard SPI providers (`ConnectorMetadata`, `ConnectorSplitManager`, `ConnectorPageSourceProvider`, etc.). Trino calls these. Classloader context is handled by the standard `ClassLoaderSafe*` wrappers (`io.trino.plugin.base.classloader.*`) — already used today.
+
+### Maven dependencies for `hudi-trino`
+
+- **`compile`:** Hudi libs (`hudi-common`, `hudi-io`, `hudi-hive-sync`, `hudi-sync-common`) and Trino libs (`trino-filesystem`, `trino-hive`, `trino-metastore`, `trino-parquet`, `trino-cache`), Guice, Airlift, Caffeine.
+- **`provided`:** `trino-spi`, `slice`, Jackson, OpenTelemetry API, JOL (supplied by Trino at runtime).
+- **`runtime`:** log-manager, Dropwizard metrics, OpenTelemetry SDK, `trino-hive-formats`.
+- **`test`:** Trino testing libs (`trino-testing`, `trino-main`, `trino-testing-containers`, `trino-hdfs`), AssertJ, JUnit 5, Hudi test JARs.
+
+**Version alignment policy.** Trino versions are authoritative for shared libraries (Avro, Parquet, Jackson, Airlift). The `hudi-trino` POM pins these via `<dependencyManagement>` to whatever the targeted Trino release uses. If Hudi internals need a newer version, the fix is on the Hudi side or via a Trino-version bump — never by shipping divergent classpath versions.
+
+### Build target on Hudi side
+
+Trino requires Java 25, while the rest of Hudi targets a lower Java floor. `hudi-trino-plugin` therefore lives behind a Maven profile (`-Phudi-trino`) and is **excluded from the default `mvn install` reactor**:
+
+```xml
+<profile>
+ <id>hudi-trino</id>
+ <modules>
+ <module>hudi-trino-plugin</module>
+ </modules>
+</profile>
+```
+
+Default build (`mvn install`) skips it; Trino-targeted build (`mvn install -Phudi-trino`) requires JDK 25.
+
+### CI
+
+Two new GitHub Actions on the Hudi side, required for any change touching `hudi-trino-plugin/**`:
+
+1. **`hudi-trino-ci.yml`** — runs the full test suite via `mvn verify -Phudi-trino` on JDK 25. Catches regressions before they ship in a Hudi release.
+2. **`hudi-trino-compat.yml`** — nightly: pulls latest `trinodb/trino` master and latest `apache/hudi` master, builds Trino's relevant modules, then compiles `hudi-trino-plugin` against them and runs the `hudi-trino-plugin` test suite. Catches both SPI drift and behavioral incompatibilities before the next Trino release.
+
+On the Trino side, existing CI continues to build and test `plugin/trino-hudi`, exercising the published `hudi-trino` artifact end-to-end on every Trino PR.
+
+### Test strategy
+
+**Full test duplication.** The Trino-side smoke tests (`TestHudiSmokeTest`, `TestHudiMinioConnectorSmokeTest`, `TestHudiConnectorTest`, etc.) are mirrored on the Hudi side and additionally extended.
+
+- **Trino side runs them** on every Trino PR — fulfilling the Trino-side test-coverage commitment.
+- **Hudi side runs them** on every Hudi PR touching `hudi-trino-plugin` — so Hudi contributors catch regressions before they ship in a Hudi release. The Hudi-side suite is also **expanded** with more granular unit tests covering split generation edge cases, all eight index-support strategies, the MOR record-merging path, lazy-commit-time snapshot isolation, and the cache-key provider.
+
+This duplication has a known cost — two places to update when adding tests — but is the right trade-off given:
+- The Trino-side suite must remain comprehensive as agreed with the Trino community.
+- Hudi-side contributors need fast feedback without waiting for a Trino-side PR cycle.
+
+### Risks & caveats
+
+- **Trino SPI drift.** A future Trino SPI change could break the pre-built `hudi-trino` artifact at runtime. Mitigation: the nightly compat CI compiles and runs the `hudi-trino-plugin` test suite against Trino master, flagging both compile-time and behavioral incompatibilities before a Trino release ships.
+- **Avro / Parquet / Jackson version skew.** Resolved by policy: Trino's versions are authoritative, pinned via `<dependencyManagement>` in the `hudi-trino` POM. Hudi-side fixes or Trino-version bumps adjust to it.
+- **Test-infrastructure coupling.** `hudi-trino-plugin`'s test scope depends on `trino-testing`, `trino-main`, etc., coupling the Hudi build to Trino artifacts on Maven Central. Acceptable cost.
+- **Release coordination.** A critical fix in `hudi-trino` ships only via a Hudi release. Mitigation: keep the Trino-side shim trivial so virtually all fixes can land in `hudi-trino`, and increase Hudi release cadence.
+- **License / ASF process.** Cross-project releases between two ASF projects; covered by standard PMC announcements at first release.
+
+## Rollout/Adoption Plan
+
+**Step 1 — Hudi 1.3.0 publishes `hudi-trino`.** Land the `hudi-trino-plugin` work in `apache/hudi` master behind the `-Phudi-trino` profile, land the two CI workflows, then publish `org.apache.hudi:hudi-trino:1.3.0` to Maven Central as part of the 1.3.0 release. Hudi commits to a more frequent release cadence going forward: to start, a major release roughly every month, with more frequent minor releases to stabilize this module. A `hudi-trino` artifact is cut whenever there are enough im [...]
+
+**Step 2 — Trino-side shim PR.** A small PR against `trinodb/trino` that replaces the contents of `plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/` with a single `HudiPlugin.java`, keeps `META-INF/services/io.trino.spi.Plugin` and all current tests, and adds `org.apache.hudi:hudi-trino:1.3.0` as a `compile` dependency. The PR is small by design — deletes connector code, points at the published artifact — so Trino-maintainer review burden is minimal.
+
+**Step 3 — Steady state.** Trino-Hudi feature work and bug fixes happen on the Hudi side. Each Hudi release publishes a new `hudi-trino` artifact. Trino picks up changes by bumping the pinned version — a one-line PR per release. Because Trino pins `hudi-trino` as a compile-time dependency, the `hudi-trino` ↔ Trino release mapping is **one-to-one**: each `hudi-trino` version is the version that ships embedded in a specific Trino release. This one-to-one mapping makes feature support and b [...]
+
+**Impact on existing users.** No behavioral change: same `HudiPlugin` registration, same catalog config. The first Trino release picking up `hudi-trino:1.3.0` gains the features that the previously-stalled PRs covered (metadata-table partition listing, index support, MOR snapshot-isolation correctness, file-system caching, advanced split generation). No migration tools needed.
+
+## Test Plan
+
+The RFC is validated when:
+
+- [ ] `hudi-trino-plugin` builds and its full test suite passes on the Hudi side via `mvn verify -Phudi-trino` (JDK 25), covering smoke tests, MinIO/Alluxio caching tests, MOR/COW tests, page source tests, split-factory tests, index support tests, system-table tests, and plugin/config tests.
+- [ ] The `hudi-trino-compat.yml` workflow compiles `hudi-trino-plugin` against `trinodb/trino` master successfully.
+- [ ] `org.apache.hudi:hudi-trino:1.3.0` is published to Maven Central.
+- [ ] The Trino-side shim PR is green: Trino's CI for `plugin/trino-hudi` passes against the published `1.3.0` artifact, including MinIO/S3 integration and plugin-loading tests.
+- [ ] At least one subsequent Hudi-side patch release exercises the "bump version in Trino" steady-state flow end-to-end.