You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "AyoubOm (via GitHub)" <gi...@apache.org> on 2024/03/26 19:42:26 UTC
[PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
AyoubOm opened a new pull request, #15607:
URL: https://github.com/apache/kafka/pull/15607
This fixes the two issues reported in KAFKA-16394, which are:
- Null should not be propagated to the join result when changing a foreign key
- Null should not be propagated twice in the join result when deleting a primary key
Note: This branch is based on branch KAFKA-16343. It would be rebased after merging KAFKA-16343.
For review, only last commit is concerned by this bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r2029857497
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -795,6 +796,12 @@ protected static Map<String, String> asMap(final KeyValueStore<String, ValueAndT
return result;
}
+ protected static List<KeyValue<String, String>> asList(final KeyValueStore<String, ValueAndTimestamp<String>> store) {
+ final List<KeyValue<String, String>> result = new LinkedList<>();
+ store.all().forEachRemaining(kv -> result.add(new KeyValue<>(kv.key, kv.value.value()))); // TODO: find a better solution
Review Comment:
Removing this method as we will use Map to test store content
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -307,12 +310,13 @@ public void doJoinFromRightThenDeleteRightEntity(final boolean leftJoin,
left.pipeInput("lhs3", "lhsValue3|rhs1", baseTimestamp + 2);
assertThat(
- outputTopic.readKeyValuesToMap(),
+ outputTopic.readKeyValuesToList(),
is(leftJoin
- ? mkMap(mkEntry("lhs1", "(lhsValue1|rhs1,null)"),
- mkEntry("lhs2", "(lhsValue2|rhs2,null)"),
- mkEntry("lhs3", "(lhsValue3|rhs1,null)"))
- : emptyMap()
+ ? Arrays.asList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,null)"),
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r1592386867
##########
streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -244,21 +248,21 @@ public void doJoinFromLeftThenDeleteLeftEntity() {
);
}
}
- // Now delete one LHS entity such that one delete is propagated down to the output.
- left.pipeInput("lhs1", (String) null, baseTimestamp + 6);
+ // Now delete one LHS entity such that one delete is propagated down to the output.
+ left.pipeInput("lhs1", null, baseTimestamp + 6);
assertThat(
- outputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", null)
+ outputTopic.readKeyValuesToList(),
+ is(Collections.singletonList(
+ KeyValue.pair("lhs1", null)
))
);
if (rejoin) {
assertThat(
- rejoinOutputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", null)
- ))
+ rejoinOutputTopic.readKeyValuesToList(),
+ hasItem(
Review Comment:
Not testing all items because the test is not deterministic, it may or may not contain `("lhs1", "rejoin((lhsValue1|rhs1,rhsValue1),null)")` depending on whether it's the left hand or the right hand side of the final join which propagates first.
`rejoin` is computed this way in the test:
___________LEFT
| |
| RIGHT |
| | |
\ / /
\ / /
\ / /
FKJoin /
\ /
\ /
Rejoin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "wrwksexahatvani (via GitHub)" <gi...@apache.org>.
wrwksexahatvani commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2376945505
@mjsax @wcarlson5 @gongxuanzhang This is a major blocker as it leads to data loss as happened in dozens of my projects. When will the fix for this be integrated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "wrwksexahatvani (via GitHub)" <gi...@apache.org>.
wrwksexahatvani commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2574874282
@mjsax @wcarlson5 @gongxuanzhang Even if Kafka is a busy project critical issues leading to data loss like this should have a fast lane for the nearest minor or hot fix release - has this one been triaged properly yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2088000194
Can we update existing test to not use `Map` but `List` (as I mentioned on the duplicate ticket https://issues.apache.org/jira/browse/KAFKA-16644) which should expose the bug?
Need to think about the fix a little more (need to refresh my memory on the different "subscription types").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-3043394454
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2088456658
> Can we update existing test to not use `Map` but `List` (as I mentioned on the duplicate ticket https://issues.apache.org/jira/browse/KAFKA-16644) which should expose the bug?
>
> Need to think about the fix a little more (need to refresh my memory on the different "subscription types").
Ok, will check the integration test and update it accordingly.
For the subscription types, I've kept a summary that helped me write the unit tests and implement the fix. It can make it easier to review it first if I missed something before checking the code ;)
```
// All cases of SubscriptionSendProcessor
/* LEFT JOIN Tests Summary
old record -> new record result Instruction
(pk1, null) -> (pk1, null) -> NOTHING
(pk1, null) -> (pk1, (null, ...)) -> PROPAGATE_IF_NO_FK_AVAILABLE
(pk1, null) -> (pk1, (fk1, ...)) -> PROPAGATE_IF_NO_FK_AVAILABLE
(pk1, (null, ..)) -> (pk1, null) -> PROPAGATE_IF_NO_FK_AVAILABLE
(pk1, (null, ..)) -> (pk1, (null, ...)) -> PROPAGATE_IF_NO_FK_AVAILABLE
(pk1, (null, ..)) -> (pk1, (fk1, ...)) -> PROPAGATE_IF_NO_FK_AVAILABLE
(pk1, (fk1, ..)) -> (pk1, null) -> DELETE_AND_PROPAGATE
(pk1, (fk1, ..)) -> (pk1, (null, ...)) -> DELETE_AND_PROPAGATE
(pk1, (fk1, ..)) -> (pk1, (fk1, ...)) -> PROPAGATE_IF_NO_FK_AVAILABLE
(pk1, (fk1, ..)) -> (pk1, (fk2, ...)) -> DELETE_NO_PROPAGATE + PROPAGATE_IF_NO_FK_AVAILABLE
*/
```
I represent each record with (pk, (fk, ...)) where pk is the key, and the value is composed of the foreign key fk and the other fields represented by the dots (which do not impact the resulting instruction type)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2757816904
@mjsax rebase done. Ready to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2783873590
As discussed via DM on some other channel, I think the original issue of duplicate tombstones was already fixed via https://github.com/apache/kafka/pull/19005
I think we should re-purpose this PR though, to improve the FK-join implementation further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2670307552
@AyoubOm -- I am finally spending time on this. I am still muling over the problem etc... Not sure if you are still interested to contribute a fix? If not, also ok with me, and I can submit a PR myself. Just checking in, as I don't just want to silently assume that you dropped this and lost interest (what would totally be my/our fault, as we did not have the capacity to spent time on it so far).
If you still have interest, please let me know, and we can push it over the finish line together. As this PR did age, it would need an update to resolve the merge conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2676134830
@mjsax, I am still interested in getting this fix merged. I will try to find some time to resolve the conflicts (I have been very busy these last months, that's why I am not active in Kafka recently).
I think the existing conflicts don't impact this fix. If you can have a look at the implementation in the meantime ?
Anyway, I will do my best to spare some time for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r2026210742
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -644,16 +645,16 @@ public void shouldUnsubscribeOldForeignKeyIfLeftSideIsUpdated(final boolean left
left.pipeInput("lhs1", "lhsValue1|rhs1", baseTimestamp + 2);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|rhs1,rhsValue1)")
+ final List<KeyValue<String, String>> expected = Collections.singletonList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,rhsValue1)")
);
assertThat(
- outputTopic.readKeyValuesToMap(),
+ outputTopic.readKeyValuesToList(),
is(expected)
);
if (materialized) {
assertThat(
- asMap(store),
+ asList(store),
Review Comment:
as above
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -712,12 +713,12 @@ public void shouldEmitRecordOnNullForeignKeyForLeftJoins(final String optimizati
left.pipeInput("lhs1", "lhsValue1|rhs1", baseTimestamp);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|rhs1,null)")
+ final List<KeyValue<String, String>> expected = Collections.singletonList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,null)")
);
- assertThat(outputTopic.readKeyValuesToMap(), is(expected));
+ assertThat(outputTopic.readKeyValuesToList(), is(expected));
if (materialized) {
- assertThat(asMap(store), is(expected));
+ assertThat(asList(store), is(expected));
Review Comment:
as above
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -423,15 +428,15 @@ public void shouldEmitTombstoneWhenDeletingNonJoiningRecords(final boolean leftJ
left.pipeInput("lhs1", "lhsValue1|rhs1", baseTimestamp);
{
- final Map<String, String> expected =
- leftJoin ? mkMap(mkEntry("lhs1", "(lhsValue1|rhs1,null)")) : emptyMap();
+ final List<KeyValue<String, String>> expected =
+ leftJoin ? Collections.singletonList(KeyValue.pair("lhs1", "(lhsValue1|rhs1,null)")) : emptyList();
assertThat(
- outputTopic.readKeyValuesToMap(),
+ outputTopic.readKeyValuesToList(),
is(expected)
);
if (materialized) {
assertThat(
- asMap(store),
+ asList(store),
Review Comment:
as above
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -753,23 +754,23 @@ public void shouldEmitRecordWhenOldAndNewFkDiffer(final String optimization,
final Bytes key = subscriptionStoreKey("lhs1", "rhs1");
left.pipeInput("lhs1", "lhsValue1|rhs1", baseTimestamp);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|rhs1,null)")
+ final List<KeyValue<String, String>> expected = Collections.singletonList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,null)")
);
- assertThat(outputTopic.readKeyValuesToMap(), is(expected));
+ assertThat(outputTopic.readKeyValuesToList(), is(expected));
if (materialized) {
- assertThat(asMap(store), is(expected));
+ assertThat(asList(store), is(expected));
Review Comment:
as above
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -307,12 +310,13 @@ public void doJoinFromRightThenDeleteRightEntity(final boolean leftJoin,
left.pipeInput("lhs3", "lhsValue3|rhs1", baseTimestamp + 2);
assertThat(
- outputTopic.readKeyValuesToMap(),
+ outputTopic.readKeyValuesToList(),
is(leftJoin
- ? mkMap(mkEntry("lhs1", "(lhsValue1|rhs1,null)"),
- mkEntry("lhs2", "(lhsValue2|rhs2,null)"),
- mkEntry("lhs3", "(lhsValue3|rhs1,null)"))
- : emptyMap()
+ ? Arrays.asList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,null)"),
Review Comment:
nit: indention
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -795,6 +796,12 @@ protected static Map<String, String> asMap(final KeyValueStore<String, ValueAndT
return result;
}
+ protected static List<KeyValue<String, String>> asList(final KeyValueStore<String, ValueAndTimestamp<String>> store) {
+ final List<KeyValue<String, String>> result = new LinkedList<>();
+ store.all().forEachRemaining(kv -> result.add(new KeyValue<>(kv.key, kv.value.value()))); // TODO: find a better solution
Review Comment:
This look ok to me?
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -753,23 +754,23 @@ public void shouldEmitRecordWhenOldAndNewFkDiffer(final String optimization,
final Bytes key = subscriptionStoreKey("lhs1", "rhs1");
left.pipeInput("lhs1", "lhsValue1|rhs1", baseTimestamp);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|rhs1,null)")
+ final List<KeyValue<String, String>> expected = Collections.singletonList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,null)")
);
- assertThat(outputTopic.readKeyValuesToMap(), is(expected));
+ assertThat(outputTopic.readKeyValuesToList(), is(expected));
if (materialized) {
- assertThat(asMap(store), is(expected));
+ assertThat(asList(store), is(expected));
}
Assertions.assertNotNull(subscriptionStore.get(key));
}
left.pipeInput("lhs1", "lhsValue1|returnNull", baseTimestamp);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|returnNull,null)")
+ final List<KeyValue<String, String>> expected = Collections.singletonList(
+ KeyValue.pair("lhs1", "(lhsValue1|returnNull,null)")
);
- assertThat(outputTopic.readKeyValuesToMap(), is(expected));
+ assertThat(outputTopic.readKeyValuesToList(), is(expected));
if (materialized) {
- assertThat(asMap(store), is(expected));
+ assertThat(asList(store), is(expected));
Review Comment:
as above
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -662,16 +663,16 @@ public void shouldUnsubscribeOldForeignKeyIfLeftSideIsUpdated(final boolean left
// Change LHS foreign key reference
left.pipeInput("lhs1", "lhsValue1|rhs2", baseTimestamp + 3);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|rhs2,rhsValue2)")
+ final List<KeyValue<String, String>> expected = Collections.singletonList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs2,rhsValue2)")
);
assertThat(
- outputTopic.readKeyValuesToMap(),
+ outputTopic.readKeyValuesToList(),
is(expected)
);
if (materialized) {
assertThat(
- asMap(store),
+ asList(store),
Review Comment:
as above
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -203,26 +206,26 @@ public void doJoinFromLeftThenDeleteLeftEntity(final boolean leftJoin,
left.pipeInput("lhs2", "lhsValue2|rhs2", baseTimestamp + 4);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|rhs1,rhsValue1)"),
- mkEntry("lhs2", "(lhsValue2|rhs2,rhsValue2)")
- );
+ final List<KeyValue<String, String>> expected = Arrays.asList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,rhsValue1)"),
+ KeyValue.pair("lhs2", "(lhsValue2|rhs2,rhsValue2)"));
+
assertThat(
- outputTopic.readKeyValuesToMap(),
+ outputTopic.readKeyValuesToList(),
is(expected)
);
if (rejoin) {
assertThat(
- rejoinOutputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", "rejoin((lhsValue1|rhs1,rhsValue1),lhsValue1|rhs1)"),
- mkEntry("lhs2", "rejoin((lhsValue2|rhs2,rhsValue2),lhsValue2|rhs2)")
- ))
+ rejoinOutputTopic.readKeyValuesToList(),
+ is(Arrays.asList(
+ KeyValue.pair("lhs1", "rejoin((lhsValue1|rhs1,rhsValue1),lhsValue1|rhs1)"),
+ KeyValue.pair("lhs2", "rejoin((lhsValue2|rhs2,rhsValue2),lhsValue2|rhs2)"))
+ )
);
}
if (materialized) {
assertThat(
- asMap(store),
+ asList(store),
Review Comment:
This seems to be potentially brittle? The problem is, we don't have any guarantee in what order the store would add the record to the list, right? And it could change. So for checking the store content it seems we should keep using Map, and only switch to List when verifying the result topic.
Cf https://github.com/apache/kafka/pull/19005/files#diff-ec133e43438f0cc66fc57a58758594758ec81cdb8e1039bcc949e613aa4d7b18R230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2571488608
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-3157316976
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r1592389047
##########
streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -244,21 +248,21 @@ public void doJoinFromLeftThenDeleteLeftEntity() {
);
}
}
- // Now delete one LHS entity such that one delete is propagated down to the output.
- left.pipeInput("lhs1", (String) null, baseTimestamp + 6);
+ // Now delete one LHS entity such that one delete is propagated down to the output.
+ left.pipeInput("lhs1", null, baseTimestamp + 6);
assertThat(
- outputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", null)
+ outputTopic.readKeyValuesToList(),
+ is(Collections.singletonList(
+ KeyValue.pair("lhs1", null)
))
);
if (rejoin) {
assertThat(
- rejoinOutputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", null)
- ))
+ rejoinOutputTopic.readKeyValuesToList(),
+ hasItem(
Review Comment:
Not testing all items because the test is not deterministic, it may or may not contain `("lhs1", "rejoin((lhsValue1|rhs1,rhsValue1),null)")` depending on whether it's the left hand or the right hand side of the final join which propagates first.
`rejoin` is computed this way in the test:
___________LEFT
| |
| RIGHT |
| | |
\ / /
\ / /
\ / /
FKJoin /
\ /
\ /
Rejoin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2378187158
Thanks for the ping -- I have this PR on my mind, but just don't find time to review... The 3.9 ship did sail. I hope (but cannot promise) to get this into 4.0 release... :(
Kafka is really a busy project and we don't have enough committers to review all PRs in a timely manner unfortunately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2670309222
Btw, there is also https://github.com/apache/kafka/pull/18887 which fixes something similar... We need to coordinate the two PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r2029522029
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -203,26 +206,26 @@ public void doJoinFromLeftThenDeleteLeftEntity(final boolean leftJoin,
left.pipeInput("lhs2", "lhsValue2|rhs2", baseTimestamp + 4);
{
- final Map<String, String> expected = mkMap(
- mkEntry("lhs1", "(lhsValue1|rhs1,rhsValue1)"),
- mkEntry("lhs2", "(lhsValue2|rhs2,rhsValue2)")
- );
+ final List<KeyValue<String, String>> expected = Arrays.asList(
+ KeyValue.pair("lhs1", "(lhsValue1|rhs1,rhsValue1)"),
+ KeyValue.pair("lhs2", "(lhsValue2|rhs2,rhsValue2)"));
+
assertThat(
- outputTopic.readKeyValuesToMap(),
+ outputTopic.readKeyValuesToList(),
is(expected)
);
if (rejoin) {
assertThat(
- rejoinOutputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", "rejoin((lhsValue1|rhs1,rhsValue1),lhsValue1|rhs1)"),
- mkEntry("lhs2", "rejoin((lhsValue2|rhs2,rhsValue2),lhsValue2|rhs2)")
- ))
+ rejoinOutputTopic.readKeyValuesToList(),
+ is(Arrays.asList(
+ KeyValue.pair("lhs1", "rejoin((lhsValue1|rhs1,rhsValue1),lhsValue1|rhs1)"),
+ KeyValue.pair("lhs2", "rejoin((lhsValue2|rhs2,rhsValue2),lhsValue2|rhs2)"))
+ )
);
}
if (materialized) {
assertThat(
- asMap(store),
+ asList(store),
Review Comment:
Agree, will change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r2029852092
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -795,6 +796,12 @@ protected static Map<String, String> asMap(final KeyValueStore<String, ValueAndT
return result;
}
+ protected static List<KeyValue<String, String>> asList(final KeyValueStore<String, ValueAndTimestamp<String>> store) {
+ final List<KeyValue<String, String>> result = new LinkedList<>();
+ store.all().forEachRemaining(kv -> result.add(new KeyValue<>(kv.key, kv.value.value()))); // TODO: find a better solution
Review Comment:
Yes, removing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r2029852092
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -795,6 +796,12 @@ protected static Map<String, String> asMap(final KeyValueStore<String, ValueAndT
return result;
}
+ protected static List<KeyValue<String, String>> asList(final KeyValueStore<String, ValueAndTimestamp<String>> store) {
+ final List<KeyValue<String, String>> result = new LinkedList<>();
+ store.all().forEachRemaining(kv -> result.add(new KeyValue<>(kv.key, kv.value.value()))); // TODO: find a better solution
Review Comment:
Yes, removing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "AyoubOm (via GitHub)" <gi...@apache.org>.
AyoubOm commented on code in PR #15607:
URL: https://github.com/apache/kafka/pull/15607#discussion_r1592386867
##########
streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java:
##########
@@ -244,21 +248,21 @@ public void doJoinFromLeftThenDeleteLeftEntity() {
);
}
}
- // Now delete one LHS entity such that one delete is propagated down to the output.
- left.pipeInput("lhs1", (String) null, baseTimestamp + 6);
+ // Now delete one LHS entity such that one delete is propagated down to the output.
+ left.pipeInput("lhs1", null, baseTimestamp + 6);
assertThat(
- outputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", null)
+ outputTopic.readKeyValuesToList(),
+ is(Collections.singletonList(
+ KeyValue.pair("lhs1", null)
))
);
if (rejoin) {
assertThat(
- rejoinOutputTopic.readKeyValuesToMap(),
- is(mkMap(
- mkEntry("lhs1", null)
- ))
+ rejoinOutputTopic.readKeyValuesToList(),
+ hasItem(
Review Comment:
Not testing all items because the test is not deterministic, it may or may not contain `("lhs1", "rejoin((lhsValue1|rhs1,rhsValue1),null)")` depending on whether it's the left hand or the right hand side of the final join which propagates first.
`rejoin` is computed this way in the test:
___________LEFT
| |
| RIGHT |
| | |
\ / /
\ / /
\ / /
FKJoin /
\ /
\ /
Rejoin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "soarez (via GitHub)" <gi...@apache.org>.
soarez commented on PR #15607:
URL: https://github.com/apache/kafka/pull/15607#issuecomment-2026208622
@mjsax I think this one should also be tagged 'streams'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16394: Fix null propagation in foreign key join result [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #15607: KAFKA-16394: Fix null propagation in foreign key join result
URL: https://github.com/apache/kafka/pull/15607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org