You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/12/17 15:27:19 UTC
[GitHub] [kafka] nicktelford opened a new pull request #11610: KAFKA-13549: Add delete.interval.ms to Streams
nicktelford opened a new pull request #11610:
URL: https://github.com/apache/kafka/pull/11610
Records are explicitly deleted once they have been fully consumed.
Currently, this is done every time the Task is committed, resulting in
"delete records" requests being sent every `commit.interval.ms`
milliseconds.
When `commit.interval.ms` is set very low, for example when
`processing.guarantee` is set to `exactly_once_v2`, this causes delete
records requests to be sent extremely frequently, potentially reducing
throughput and causing a high volume of log messages to be logged by the
brokers.
Disconnecting delete records requests from the commit interval resolves
this problem.
We now only explicitly delete records for a repartition topic when we
commit, if it's been at least `delete.interval.ms` milliseconds since
the last time we deleted records.
Because we still require a commit to trigger record deletion, the
lower-bound of `delete.interval.ms` is effectively capped at the
`commit.interval.ms`.
For compatibility, the default `delete.interval.ms` is set to 30
seconds, the same as the default `commit.interval.ms`. Users who have
configured a different `commit.interval.ms` may need to review and
change `delete.interval.ms`.
Unlike `commit.interval.ms`, we don't dynamically change the default for
`delete.interval.ms` when EOS processing is enabled, as it's important
not to flood brokers with the record deletions, and we want a sensible
default.
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
This code is my own work and is licensed to the Apache Kafka project under the terms of the same license (ASL 2) as the project itself.
### Committer Checklist (excluded from commit message)
- [X] Verify design and implementation
- [X] Verify test coverage and CI build status
- [X] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford commented on pull request #11610: KAFKA-13549: Add repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
nicktelford commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1043171527
@mjsax I've added some tests as suggested and re-based against `trunk`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #11610: KAFKA-13549: Add repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1068542894
Merged to `trunk`. Thanks for the KIP, PR, and for adding test!
Very nice addition!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #11610: KAFKA-13549: Add repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
cadonna commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1042923168
@nicktelford and @mjsax Do you think we can get this PR merged by March 16th so that we can ship this feature with AK 3.2.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford commented on pull request #11610: KAFKA-13549: Add repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
nicktelford commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1042935934
@cadonna I believe the only missing piece is the test(s), which should be done well before March 16th.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford commented on pull request #11610: KAFKA-13549: Add delete.interval.ms to Streams
Posted by GitBox <gi...@apache.org>.
nicktelford commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-996810481
@mjsax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax merged pull request #11610: KAFKA-13549: Add repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
mjsax merged pull request #11610:
URL: https://github.com/apache/kafka/pull/11610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford edited a comment on pull request #11610: KAFKA-13549: Add min.repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
nicktelford edited a comment on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1010106545
KIP available at: https://cwiki.apache.org/confluence/display/KAFKA/KIP-811%3A+Add+config+repartition.purge.interval.ms+to+Kafka+Streams
The commit and PR description have been updated to the latest specification from the KIP, based on the dev list discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford commented on pull request #11610: KAFKA-13549: Add repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
nicktelford commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1043126322
@mjsax I don't see a `StreamThreadTest`. Do you mean `GlobalStreamThreadTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford removed a comment on pull request #11610: KAFKA-13549: Add repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
nicktelford removed a comment on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1043126322
@mjsax I don't see a `StreamThreadTest`. Do you mean `GlobalStreamThreadTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #11610: KAFKA-13549: Add delete.interval.ms to Streams
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-997113605
Thanks for the PR. Can you do a KIP for this? Happy to review the PR after the KIP is approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford commented on pull request #11610: KAFKA-13549: Add min.repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
nicktelford commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1010106545
KIP available at: https://cwiki.apache.org/confluence/display/KAFKA/KIP-811%3A+Add+separate+min.repartition.purge.interval.ms+to+Kafka+Streams
The commit and PR description have been updated to the latest specification from the KIP, based on the dev list discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] nicktelford commented on pull request #11610: KAFKA-13549: Add min.repartition.purge.interval.ms
Posted by GitBox <gi...@apache.org>.
nicktelford commented on pull request #11610:
URL: https://github.com/apache/kafka/pull/11610#issuecomment-1015328560
@mjsax The KIP has now been adopted and I've updated this PR to reflect the latest version of the KIP. Should now be ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org