You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "mattcasters (via GitHub)" <gi...@apache.org> on 2026/07/10 20:47:22 UTC
[PR] issue #7393 : Merge Rows layout alignment option for CDC (hop)
mattcasters opened a new pull request, #7487:
URL: https://github.com/apache/hop/pull/7487
## Summary
- Adds an **Align input layouts** option to Merge Rows (diff), enabled by default (`align_input_layouts` with `defaultBoolean = true`), so reference and compare streams can use different field layouts for CDC-style diffs.
- Builds a **reference-authoritative** field union before comparing: missing fields are null-filled; shared fields keep the reference type (with conversion for compare values); compare-only fields are appended.
- Strict legacy mode remains available with `align_input_layouts=N`.
- Integration tests cover existing behavior (base, diff JSON, passthrough), new alignment, and strict align-off compatibility.
- **Integration test runner improvement:** `run-tests-docker.sh` / `run-tests.sh` accept optional **`TEST_FILTER`** to run only matching `main*.hwf` workflows (basename substring or shell glob, comma-separated). Speeds up Docker IT debugging, e.g.:
```bash
bash integration-tests/scripts/run-tests-docker.sh \
PROJECT_NAME=transforms \
TEST_FILTER=0077-merge-rows
```
## Context
Addresses [#7393](https://github.com/apache/hop/issues/7393). Ports the proven Merge Rows Plus layout-alignment approach from the neighbor hop-data-vault project into core Merge Rows.
## Test plan
- [x] `./mvnw -pl plugins/transforms/mergerows -am test` (17 unit tests)
- [x] Integration suite via Hop client: `sh hop run -j "IT transforms" -f main-0077-merge-rows.hwf -r local` — all five unit tests pass (base, diff, passthrough, align, align-off)
- [x] Docker IT with rebuilt client zip + `TEST_FILTER=0077-merge-rows`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7393 : Merge Rows layout alignment option for CDC (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on PR #7487:
URL: https://github.com/apache/hop/pull/7487#issuecomment-4979751364
Not sure we should force this to true for already existing transforms, can cause regressions, I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7393 : Merge Rows layout alignment option for CDC (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #7487:
URL: https://github.com/apache/hop/pull/7487#issuecomment-4979982875
I was one of them until I had enough of it and implemented this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7393 : Merge Rows layout alignment option for CDC (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #7487:
URL: https://github.com/apache/hop/pull/7487#issuecomment-4979944578
You're right. It could fix bugs in the sense that the old behavior silently did the wrong thing. I'll have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7393 : Merge Rows layout alignment option for CDC (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on PR #7487:
URL: https://github.com/apache/hop/pull/7487#issuecomment-4979958714
Yeah, but I fear there are a lot of people who use bugs as "features"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7393 : Merge Rows layout alignment option for CDC (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters closed pull request #7487: issue #7393 : Merge Rows layout alignment option for CDC
URL: https://github.com/apache/hop/pull/7487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org