You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "kotwal-itpro (via GitHub)" <gi...@apache.org> on 2026/10/07 17:37:21 UTC
[PR] Issue #5524 : Move the lake table transforms into a lakehouse plugin (hop)
kotwal-itpro opened a new pull request, #8792:
URL: https://github.com/apache/hop/pull/8792
First step for #5524, following the plan agreed in the issue with @hansva and @mattcasters.
This moves the lake table transforms (input, output, merge, maintenance) and the catalog metadata type out of the Spark engine plugin into a new plugin, `plugins/tech/lakehouse`. Nothing changes for users yet. The point is to give the local engine a place to implement the same transforms in the next PRs, so we end up with one set of transforms that runs on both engines.
## What moves
| From (Spark plugin) | To (`hop-tech-lakehouse`) |
|---|---|
| `SparkLakeTable{Input,Output,Merge,Maintenance}{,Meta,Dialog,Data}` | `org.apache.hop.lakehouse.transforms.LakeTable*` |
| `SparkCatalog`, `SparkCatalogEditor`, `SparkCatalogTemplate` | `org.apache.hop.lakehouse.metadata.LakeCatalog*` |
| Messages and icons for the above | same, under `org/apache/hop/lakehouse/...` |
| Meta injection tests and `SparkCatalogTemplateTest` | moved with the classes |
The Spark plugin keeps everything Spark-specific: the `SparkLakeTable*Handler` classes, `SparkLakeTableSupport`, the SQL builders, `SparkCatalogApplier` and `LakeSessionPlan`. It reaches the moved classes through `dependencies.xml` (`../../tech/lakehouse`), the same way it already uses memgroupby, mergejoin and sort.
## Compatibility
Existing pipelines and metadata load unchanged:
- Transform plugin IDs keep their values (`SparkLakeTableInput`, `SparkLakeTableOutput`, `SparkLakeTableMerge`, `SparkLakeTableMaintenance`). They now live in `LakehouseConst`, and `SparkConst` refers to them.
- The catalog keeps the metadata key `spark-catalog`, with `SparkCatalog` still as a legacy key, so existing objects stay in the same folder.
- XML and metadata property keys are unchanged, and so are the i18n keys. Display names still say "Spark lake table ..." for now. I'd rather rename them in the PR that makes them work on the local engine, so this one stays a pure move.
- The Spark handlers already rebuild each meta from the transform XML (`new LakeTableInputMeta()` + `loadXml`), so the Spark plugin having its own copy of the classes in its class loader causes no problems.
Two small things I had to decide:
- `SparkField` is also used by the Spark file input and SQL transforms, so it stays where it is. The lakehouse plugin gets its own `LakeField` with the same serialized keys, and `SparkLakeTableSupport.toSparkFields` converts at the one place the Spark side needs it.
- The merge action and maintenance operation constants moved onto `LakeTableMergeMeta` and `LakeTableMaintenanceMeta`. `SparkMergeSqlBuilder` and `SparkMaintenanceSqlBuilder` refer to those, so the values are defined once.
The new plugin is added to `assemblies/plugins` and `assemblies/debug`.
## Testing
- `hop-tech-lakehouse`: 15 tests (meta injection for all four transforms, catalog templates).
- `hop-engines-spark`: all 177 tests pass, including the Iceberg and Delta path, table mode, time travel, merge and maintenance tests.
- `apache-rat:check` and `spotless:apply` clean on both modules.
## Next
2. Local engine read support for Iceberg.
3. Local engine write support for Iceberg.
4. Integration tests for the local engine, docs and samples.
addresses #5524
- [x] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Issue #5524 : Move the lake table transforms into a lakehouse plugin (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on PR #8792:
URL: https://github.com/apache/hop/pull/8792#issuecomment-6055266062
Built and tested locally: both modules build, unit tests pass (lakehouse 15, spark 179), spotless/RAT clean. Also ran the `iceberg` and `lakehouse` integration tests on `spark-local` against an assembled client with the new `plugins/tech/lakehouse` plugin installed next to the Spark engine: 6/6 pass, no class loading errors. The catalog metadata path (TABLE mode with a catalog object) was reviewed in the code but not exercised by the ITs.
No blockers. LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Issue #5524 : Move the lake table transforms into a lakehouse plugin (hop)
Posted by "kotwal-itpro (via GitHub)" <gi...@apache.org>.
kotwal-itpro commented on PR #8792:
URL: https://github.com/apache/hop/pull/8792#issuecomment-6076145127
Closing: this change landed in main as part of #8802, which was stacked on it and squash-merged (52d54c69f4). Thanks again @bamaer for the review and the integration test run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Issue #5524 : Move the lake table transforms into a lakehouse plugin (hop)
Posted by "kotwal-itpro (via GitHub)" <gi...@apache.org>.
kotwal-itpro commented on PR #8792:
URL: https://github.com/apache/hop/pull/8792#issuecomment-6055710519
Thanks a lot @bamaer, especially for running the integration tests against an assembled client, that's the class loading case I was most worried about.
On TABLE mode with a catalog object: the next PR covers it on the local engine. `LakeTableInputLocalTest.tableModeWithHadoopCatalog` reads a table through a hadoop catalog object end to end, and `IcebergTablesTest` covers how hadoop and rest catalog objects are turned into Iceberg catalogs. I'm opening it now. It builds on this one, so until this is merged it shows both commits; only the last one is new.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Issue #5524 : Move the lake table transforms into a lakehouse plugin (hop)
Posted by "kotwal-itpro (via GitHub)" <gi...@apache.org>.
kotwal-itpro closed pull request #8792: Issue #5524 : Move the lake table transforms into a lakehouse plugin
URL: https://github.com/apache/hop/pull/8792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org