fresh-borzoni commented on PR #27372:
URL: https://github.com/apache/flink/pull/27372#issuecomment-4359995004
Real-world case demonstrating the need.
Flink 2.2.0 + Fluss 0.9, classic parent/child CDC join - sale_items.PK = (provider_id, sale_id, id) strictly extends sales.PK = (provider_id, id), with bucket key on each side covering its join key. All documented DeltaJoin preconditions hold (Fluss PK on both sides, INNER JOIN, no DELETE, delta-join.strategy=AUTO).
``` sql
INSERT INTO sales_with_sale_items
SELECT s.provider_id, s.id AS sale_id, si.id AS sale_item_id, ...
FROM sale_items si INNER JOIN sales s
ON si.provider_id = s.provider_id AND si.sale_id = s.id;
```
Stock 2.2.0 falls back to a regular Join and injects a SinkUpsertMaterializer:
+- Sink(... upsertMaterialize=[true])
+- Calc(...)
+- Join(leftInputSpec=[HasUniqueKey], rightInputSpec=[JoinKeyContainsUniqueKey],
changelogMode=[I,UB,UA], duplicateChanges=[DISALLOW])
:- TableSourceScan([sale_items])
+- TableSourceScan([sales])
This PR cherry-picked onto release-2.2.0:
+- Sink(...)
+- DeltaJoin(...)
:- DropUpdateBefore +- TableSourceScan([sale_items])
+- DropUpdateBefore +- TableSourceScan([sales])
Why: Base getJoinUpsertKeys returns only {si.provider_id, si.sale_id, si.id} (left's PK). The natural SELECT projects s.provider_id, s.id - bit-identical under INNER JOIN, but the Calc only retains projected columns, so the only upsert key dies. No upstream key -> SinkUpsertMaterializer -> duplicateChanges=DISALLOW propagates -> DropUpdateBefore blocked → DeltaJoin rejected.
This PR's enrichment derives {si.id, s.provider_id, s.id} via the equi-pairs - survives the Calc, matches the sink PK, materializer drops, DeltaJoin applies.
The asymmetric parent/child shape (one PK strictly extends the join key, the other matches it) is the canonical CDC join - orders/order_items, users/sessions, conversations/messages, etc. A SELECT-rewrite workaround exists (project FK columns from the child instead of the parent), but the failure mode is silent and the gap isn't obvious to authors.
@lincoln-lil Can you take a look, pls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org