You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mjsax (via GitHub)" <gi...@apache.org> on 2025/09/29 01:33:58 UTC
[PR] MINOR: simplify FK-join logic [kafka]
mjsax opened a new pull request, #20605:
URL: https://github.com/apache/kafka/pull/20605
The existing FK join logic is very convoluted due to incremental changes and bug-fixes, and thus very hard to understand.
This PR rewrites the logic from scratch to make it easier to understanding, and fixes a minor bug on the side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2388371695
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java:
##########
@@ -128,47 +128,105 @@ public void process(final Record<KLeft, Change<VLeft>> record) {
}
private void leftJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
- if (oldForeignKey != null && !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ if (oldValue == null && newValue == null) {
+ // no output for idempotent left hand side deletes
+ return;
+ }
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final KRight newForeignKey = newValue == null ? null : foreignKeyExtractor.extract(record.key(), newValue);
+
+ final boolean maybeUnsubscribe = oldForeignKey != null;
+ if (maybeUnsubscribe) {
+ // delete old subscription only if FK changed
+ //
+ // if FK did change, we need to explicitly delete the old subscription,
+ // because the new subscription goes to a different partition
+ final boolean foreignKeyChanged = !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey));
+
+ if (foreignKeyChanged) {
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
forward(record, oldForeignKey, DELETE_KEY_NO_PROPAGATE);
}
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
- } else if (record.value().newValue != null) {
- final KRight newForeignKey = foreignKeyExtractor.extract(record.key(), record.value().newValue);
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
}
+
+ // for all cases (insert, update, and delete), we send a new subscription;
+ // we need to get a response back for all cases to always produce a left-join result
+ //
+ // note: for delete, `newForeignKey` is null, what is a "hack"
+ // no actual subscription will be added for null-FK on the right hand sice, but we still get the response back we need
+ //
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
+ forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
}
private void defaultJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final boolean needToUnsubscribe = oldForeignKey != null;
- if (oldForeignKey == null && newForeignKey == null) {
+ // if left row is inserted or updated, subscribe to new FK (if new FK is valid)
+ if (newValue != null) {
+ final KRight newForeignKey = foreignKeyExtractor.extract(record.key(), newValue);
+
+ if (newForeignKey == null) { // invalid FK
logSkippedRecordDueToNullForeignKey();
- } else if (oldForeignKey == null) {
- forward(record, newForeignKey, PROPAGATE_ONLY_IF_FK_VAL_AVAILABLE);
- } else if (newForeignKey == null) {
- forward(record, oldForeignKey, DELETE_KEY_AND_PROPAGATE);
- } else if (!Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
- //Different Foreign Key - delete the old key value and propagate the new one.
- //Delete it from the oldKey's state store
- forward(record, oldForeignKey, DELETE_KEY_NO_PROPAGATE);
- //Add to the newKey's state store. Additionally, propagate null if no FK is found there,
- //since we must "unset" any output set by the previous FK-join. This is true for both INNER
- //and LEFT join.
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
- } else { // unchanged FK
- forward(record, newForeignKey, PROPAGATE_ONLY_IF_FK_VAL_AVAILABLE);
+ if (needToUnsubscribe) {
+ // delete old subscription
+ //
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
+ forward(record, oldForeignKey, DELETE_KEY_AND_PROPAGATE);
+ }
+ } else { // valid FK
Review Comment:
Not sure if I can follow?
```
if (newForeignKey = null) {
...
} else { // what would we add here? -> `else if (newForeignKey != null)` would be redundant
...
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2386499441
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java:
##########
@@ -128,47 +128,101 @@ public void process(final Record<KLeft, Change<VLeft>> record) {
}
private void leftJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
Review Comment:
The diff is unfortunately a little bit messy, but if you compare new and old code, also because I added comments, it should become clear the the new code is much cleaner, especially for left-join case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2386497964
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionReceiveProcessorSupplier.java:
##########
@@ -82,7 +82,8 @@ public void init(final ProcessorContext<CombinedKey<KRight, KLeft>, Change<Value
@Override
public void process(final Record<KRight, SubscriptionWrapper<KLeft>> record) {
- if (record.key() == null && !SubscriptionWrapper.Instruction.PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE.equals(record.value().instruction())) {
+ final KRight foreignKey = record.key();
Review Comment:
Minor cleanup, to explain that `record.key()` is the FK -- it's not necessarily obvious form the existing code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2386698796
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java:
##########
@@ -128,47 +128,105 @@ public void process(final Record<KLeft, Change<VLeft>> record) {
}
private void leftJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
- if (oldForeignKey != null && !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ if (oldValue == null && newValue == null) {
+ // no output for idempotent left hand side deletes
+ return;
+ }
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final KRight newForeignKey = newValue == null ? null : foreignKeyExtractor.extract(record.key(), newValue);
+
+ final boolean maybeUnsubscribe = oldForeignKey != null;
+ if (maybeUnsubscribe) {
+ // delete old subscription only if FK changed
+ //
+ // if FK did change, we need to explicitly delete the old subscription,
+ // because the new subscription goes to a different partition
+ final boolean foreignKeyChanged = !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey));
+
+ if (foreignKeyChanged) {
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
forward(record, oldForeignKey, DELETE_KEY_NO_PROPAGATE);
}
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
- } else if (record.value().newValue != null) {
- final KRight newForeignKey = foreignKeyExtractor.extract(record.key(), record.value().newValue);
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
}
+
+ // for all cases (insert, update, and delete), we send a new subscription;
+ // we need to get a response back for all cases to always produce a left-join result
+ //
+ // note: for delete, `newForeignKey` is null, what is a "hack"
Review Comment:
a thought, if we split the logic into if equals null, delete(or pass null expricitly), if not do the normal stuff, will it look as hacky?
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java:
##########
@@ -128,47 +128,105 @@ public void process(final Record<KLeft, Change<VLeft>> record) {
}
private void leftJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
- if (oldForeignKey != null && !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ if (oldValue == null && newValue == null) {
+ // no output for idempotent left hand side deletes
+ return;
+ }
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final KRight newForeignKey = newValue == null ? null : foreignKeyExtractor.extract(record.key(), newValue);
+
+ final boolean maybeUnsubscribe = oldForeignKey != null;
+ if (maybeUnsubscribe) {
+ // delete old subscription only if FK changed
+ //
+ // if FK did change, we need to explicitly delete the old subscription,
+ // because the new subscription goes to a different partition
+ final boolean foreignKeyChanged = !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey));
+
+ if (foreignKeyChanged) {
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
forward(record, oldForeignKey, DELETE_KEY_NO_PROPAGATE);
}
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
- } else if (record.value().newValue != null) {
- final KRight newForeignKey = foreignKeyExtractor.extract(record.key(), record.value().newValue);
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
}
+
+ // for all cases (insert, update, and delete), we send a new subscription;
+ // we need to get a response back for all cases to always produce a left-join result
+ //
+ // note: for delete, `newForeignKey` is null, what is a "hack"
+ // no actual subscription will be added for null-FK on the right hand sice, but we still get the response back we need
+ //
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
+ forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
}
private void defaultJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final boolean needToUnsubscribe = oldForeignKey != null;
- if (oldForeignKey == null && newForeignKey == null) {
+ // if left row is inserted or updated, subscribe to new FK (if new FK is valid)
+ if (newValue != null) {
+ final KRight newForeignKey = foreignKeyExtractor.extract(record.key(), newValue);
+
+ if (newForeignKey == null) { // invalid FK
logSkippedRecordDueToNullForeignKey();
- } else if (oldForeignKey == null) {
- forward(record, newForeignKey, PROPAGATE_ONLY_IF_FK_VAL_AVAILABLE);
- } else if (newForeignKey == null) {
- forward(record, oldForeignKey, DELETE_KEY_AND_PROPAGATE);
- } else if (!Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
- //Different Foreign Key - delete the old key value and propagate the new one.
- //Delete it from the oldKey's state store
- forward(record, oldForeignKey, DELETE_KEY_NO_PROPAGATE);
- //Add to the newKey's state store. Additionally, propagate null if no FK is found there,
- //since we must "unset" any output set by the previous FK-join. This is true for both INNER
- //and LEFT join.
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
- } else { // unchanged FK
- forward(record, newForeignKey, PROPAGATE_ONLY_IF_FK_VAL_AVAILABLE);
+ if (needToUnsubscribe) {
+ // delete old subscription
+ //
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
+ forward(record, oldForeignKey, DELETE_KEY_AND_PROPAGATE);
+ }
+ } else { // valid FK
Review Comment:
nit: could be else if
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionReceiveProcessorSupplier.java:
##########
@@ -82,7 +82,8 @@ public void init(final ProcessorContext<CombinedKey<KRight, KLeft>, Change<Value
@Override
public void process(final Record<KRight, SubscriptionWrapper<KLeft>> record) {
- if (record.key() == null && !SubscriptionWrapper.Instruction.PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE.equals(record.value().instruction())) {
+ final KRight foreignKey = record.key();
Review Comment:
as a thought, what if we rename the `record` to something like `fkRecord` instead?
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java:
##########
@@ -128,47 +128,105 @@ public void process(final Record<KLeft, Change<VLeft>> record) {
}
private void leftJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
- if (oldForeignKey != null && !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ if (oldValue == null && newValue == null) {
+ // no output for idempotent left hand side deletes
+ return;
+ }
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final KRight newForeignKey = newValue == null ? null : foreignKeyExtractor.extract(record.key(), newValue);
+
+ final boolean maybeUnsubscribe = oldForeignKey != null;
+ if (maybeUnsubscribe) {
+ // delete old subscription only if FK changed
+ //
+ // if FK did change, we need to explicitly delete the old subscription,
+ // because the new subscription goes to a different partition
+ final boolean foreignKeyChanged = !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey));
Review Comment:
we can check for null here to avoid the unnecessary serialization and comparison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2388356518
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java:
##########
@@ -128,47 +128,105 @@ public void process(final Record<KLeft, Change<VLeft>> record) {
}
private void leftJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
- if (oldForeignKey != null && !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ if (oldValue == null && newValue == null) {
+ // no output for idempotent left hand side deletes
+ return;
+ }
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final KRight newForeignKey = newValue == null ? null : foreignKeyExtractor.extract(record.key(), newValue);
+
+ final boolean maybeUnsubscribe = oldForeignKey != null;
+ if (maybeUnsubscribe) {
+ // delete old subscription only if FK changed
+ //
+ // if FK did change, we need to explicitly delete the old subscription,
+ // because the new subscription goes to a different partition
+ final boolean foreignKeyChanged = !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey));
Review Comment:
Don't think it would buy us much? Just a few method calls, as `serialize()` would do the null checks and returns `null`, and if either `serialize` call does return null, `equals` is also very cheap.
I would rather keep it the way it is to keep fewer if/else checks what makes it easier to read, and I don't think we get any perf benefits if we make the code more complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2386497159
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -891,7 +891,10 @@ protected static Topology getTopology(final Properties streamsConfig,
rejoin,
leftVersioned,
rightVersioned,
- value -> value.split("\\|")[1]
+ value -> {
+ final String[] tokens = value.split("\\|");
+ return tokens.length == 2 ? tokens[1] : null;
+ }
Review Comment:
Some side cleanup... While playing with the code adding other test (which I removed later, as they were just for myself), this method failed if we did only get one token...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #20605:
URL: https://github.com/apache/kafka/pull/20605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2388366045
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionSendProcessorSupplier.java:
##########
@@ -128,47 +128,105 @@ public void process(final Record<KLeft, Change<VLeft>> record) {
}
private void leftJoinInstructions(final Record<KLeft, Change<VLeft>> record) {
- if (record.value().oldValue != null) {
- final KRight oldForeignKey = foreignKeyExtractor.extract(record.key(), record.value().oldValue);
- final KRight newForeignKey = record.value().newValue == null ? null : foreignKeyExtractor.extract(record.key(), record.value().newValue);
- if (oldForeignKey != null && !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey))) {
+ final VLeft oldValue = record.value().oldValue;
+ final VLeft newValue = record.value().newValue;
+
+ if (oldValue == null && newValue == null) {
+ // no output for idempotent left hand side deletes
+ return;
+ }
+
+ final KRight oldForeignKey = oldValue == null ? null : foreignKeyExtractor.extract(record.key(), oldValue);
+ final KRight newForeignKey = newValue == null ? null : foreignKeyExtractor.extract(record.key(), newValue);
+
+ final boolean maybeUnsubscribe = oldForeignKey != null;
+ if (maybeUnsubscribe) {
+ // delete old subscription only if FK changed
+ //
+ // if FK did change, we need to explicitly delete the old subscription,
+ // because the new subscription goes to a different partition
+ final boolean foreignKeyChanged = !Arrays.equals(serialize(newForeignKey), serialize(oldForeignKey));
+
+ if (foreignKeyChanged) {
+ // this may lead to unnecessary tombstones if the old FK did not join;
+ // however, we cannot avoid it as we have no means to know if the old FK joined or not
forward(record, oldForeignKey, DELETE_KEY_NO_PROPAGATE);
}
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
- } else if (record.value().newValue != null) {
- final KRight newForeignKey = foreignKeyExtractor.extract(record.key(), record.value().newValue);
- forward(record, newForeignKey, PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE);
}
+
+ // for all cases (insert, update, and delete), we send a new subscription;
+ // we need to get a response back for all cases to always produce a left-join result
+ //
+ // note: for delete, `newForeignKey` is null, what is a "hack"
Review Comment:
IMHO it's still hacky, as we keep sending a `null` to the right hand side. The hacky part is not about the variable being `null` (it contributes a little bit I guess, as it might seem unintuitive that it is a valid case), but about sending `null` at all (and sending `null` is also unintuitive by itself IMHO). Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: simplify FK-join logic [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20605:
URL: https://github.com/apache/kafka/pull/20605#discussion_r2388349078
##########
streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/SubscriptionReceiveProcessorSupplier.java:
##########
@@ -82,7 +82,8 @@ public void init(final ProcessorContext<CombinedKey<KRight, KLeft>, Change<Value
@Override
public void process(final Record<KRight, SubscriptionWrapper<KLeft>> record) {
- if (record.key() == null && !SubscriptionWrapper.Instruction.PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE.equals(record.value().instruction())) {
+ final KRight foreignKey = record.key();
Review Comment:
Happy to rename, but don't think it does buy us too much, as it clear from the class name and parameter type that it's a subscription record. But renaming does not help much to explain how a subscription record built up internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org