[jira] [Created] (KAFKA-10199) Separate state restoration into separate threads

["Guozhang Wang (Jira)" <ji...@apache.org>]

Guozhang Wang created KAFKA-10199:
-------------------------------------

             Summary: Separate state restoration into separate threads
                 Key: KAFKA-10199
                 URL: https://issues.apache.org/jira/browse/KAFKA-10199
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Guozhang Wang
            Assignee: Guozhang Wang


As part of the restoration optimization effort, we would like to move the restoration process to separate threads such that:

1. Stream threads would not be restricted by the main consumer `poll` frequency to keep as part of the group.

2. We can allow larger batches of data to be written into the restoration.

Besides this, we'd also like to fix the known issues that for piggy-backed source topics as changelog topics, the serde exception / extra processing logic would be skipped.

We would also cleanup the global update tasks as part of this effort to consolidate to the separate restoration threads, and would also gear them up with corresponding monitoring metrics (KIPs in progress).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)