You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Greg Harris (Jira)" <ji...@apache.org> on 2023/11/27 19:10:00 UTC
[jira] [Created] (KAFKA-15906) Emit offset syncs more often than offset.lag.max for low-throughput/finite partitions
Greg Harris created KAFKA-15906:
-----------------------------------
Summary: Emit offset syncs more often than offset.lag.max for low-throughput/finite partitions
Key: KAFKA-15906
URL: https://issues.apache.org/jira/browse/KAFKA-15906
Project: Kafka
Issue Type: Improvement
Components: mirrormaker
Reporter: Greg Harris
Right now, the offset.lag.max configuration limits the number of offset syncs are emitted by the MirrorSourceTask, along with a fair rate-limiting semaphore. After 100 records have been emitted for a partition, _and_ the semaphore is available, an offset sync can be emitted.
For low-volume topics, the `offset.lag.max` default of 100 is much more restrictive than the rate-limiting semaphore. For example, a topic which mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset sync. If the topic is actually finite, the last offset sync will never arrive, and the translation will have a persistent lag.
Instead, we can periodically flush the offset syncs for partitions that are under the offset.lag.max limit, but have not received an offset sync recently. This could be a new configuration, be a hard-coded time, or be based on the existing emit.checkpoints.interval.seconds and sync.group.offsets.interval.seconds configurations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)