[jira] [Commented] (KAFKA-6144) Allow state stores to serve stale reads during rebalance

["Vinoth Chandar (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947205#comment-16947205 ] 

Vinoth Chandar commented on KAFKA-6144:
---------------------------------------

[~NaviBrar] curious to know if you got around to the KIP. I am looking into the HA story for interactive queries atm 

> Allow state stores to serve stale reads during rebalance
> --------------------------------------------------------
>
>                 Key: KAFKA-6144
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6144
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: Antony Stubbs
>            Priority: Major
>              Labels: needs-kip
>
> Currently when expanding the KS cluster, the new node's partitions will be unavailable during the rebalance, which for large states can take a very long time, or for small state stores even more than a few ms can be a deal breaker for micro service use cases.
> One workaround is to allow stale data to be read from the state stores when use case allows.
> Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks - potentially a two phase rebalance
> This is the description from KAFKA-6031 (keeping this JIRA as the title is more descriptive):
> {quote}
> Currently reads for a key are served by single replica, which has 2 drawbacks:
>  - if replica is down there is a down time in serving reads for keys it was responsible for until a standby replica takes over
>  - in case of semantic partitioning some replicas might become hot and there is no easy way to scale the read load
> If standby replicas would have endpoints that are exposed in StreamsMetadata it would enable serving reads from several replicas, which would mitigate the above drawbacks. 
> Due to the lag between replicas reading from multiple replicas simultaneously would have weaker (eventual) consistency comparing to reads from single replica. This however should be acceptable tradeoff in many cases.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)