You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/06/28 10:13:03 UTC
[GitHub] [kafka] nicktelford opened a new pull request, #12356: KAFKA-10493: Drop out-of-order records in KTable
nicktelford opened a new pull request, #12356:
URL: https://github.com/apache/kafka/pull/12356
Instead of emitting a warning when adding out-of-order (by timestamp)
records to a `KTable`, we should discard these records silently.
Unfortunately, this creates a problem for `KTable`s produced from a
source-topic that has been optimized with the source-topic optimization,
i.e. when using `StreamsBuilder#table(...)`.
If a user topic that is being used as a changelog (by the source-topic
optimiation) has compaction enabled, it's possible that the
topic-compaction will eliminate newer (by timestamp) records, because
compaction is (currently) offset-based.
To ensure we don't produce a `KTable` that is inconsistent with the
source-topic in this scenario, we must continue to add all records
from these topics in to the materialized store, irrespective of
timestamp.
To facilitate this, `KTableSource` is now explicitly instructed by the
topology when it's processing records from an optimized source-topic,
via the new `sourceTopicOptimized` field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax closed pull request #12356: KAFKA-10493: Drop out-of-order records in KTable
Posted by GitBox <gi...@apache.org>.
mjsax closed pull request #12356: KAFKA-10493: Drop out-of-order records in KTable
URL: https://github.com/apache/kafka/pull/12356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford commented on pull request #12356: KAFKA-10493: Drop out-of-order records in KTable
Posted by GitBox <gi...@apache.org>.
nicktelford commented on PR #12356:
URL: https://github.com/apache/kafka/pull/12356#issuecomment-1168524097
Note: no tests yet as I wanted to sound-out whether this is an approach worth pursuing first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #12356: KAFKA-10493: Drop out-of-order records in KTable
Posted by GitBox <gi...@apache.org>.
mjsax commented on PR #12356:
URL: https://github.com/apache/kafka/pull/12356#issuecomment-1366975287
Given that we have https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores approved, I close the corresponding Jira (and the other open PR for it: https://github.com/apache/kafka/pull/10533)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org