[jira] [Created] (KAFKA-9952) Reconsider immediate propagation of hw when replica selector is used

["Jason Gustafson (Jira)" <ji...@apache.org>]

Jason Gustafson created KAFKA-9952:
--------------------------------------

             Summary: Reconsider immediate propagation of hw when replica selector is used
                 Key: KAFKA-9952
                 URL: https://issues.apache.org/jira/browse/KAFKA-9952
             Project: Kafka
          Issue Type: Improvement
            Reporter: Jason Gustafson


For KIP-392, we added logic to make sure that high watermark changes are propagated to followers without delay in order to improve end to end latency when fetching from followers. The downside of this change is that it increases the rate of fetch requests from followers which can have a noticeable impact on performance (see KAFKA-9731). To fix that problem, we modified the code so that we only propagate high watermark changes immediately when a replica selector is used (which is not the default). However, we should consider whether we want to go even further and disable immediate propagation of the high watermark more generally since it makes this feature risky to use. Instead, users can use the max wait time in order to control the worst-case latency. Note that this is typically only a problem for low-throughput topics since otherwise we will have a steady rate of high watermark updates.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)