[jira] [Created] (KAFKA-13523) Implement IQv2 support in global stores

["John Roesler (Jira)" <ji...@apache.org>]

John Roesler created KAFKA-13523:
------------------------------------

             Summary: Implement IQv2 support in global stores
                 Key: KAFKA-13523
                 URL: https://issues.apache.org/jira/browse/KAFKA-13523
             Project: Kafka
          Issue Type: Sub-task
            Reporter: John Roesler


Global stores pose one significant problem for IQv2: when they start up, they skip the regular ingest pipeline and instead use the "restoration" pipeline to read up until the current end offset. Then, they switch over to the regular ingest pipeline.

IQv2 position tracking expects to track the position of each record from the input topic through the ingest pipeline and then get the position headers through the restoration pipeline via the changelog topic. The fact that global stores "restore" the input topic instead of ingesting it violates our expectations.

It has also caused other problems, so we may want to consider switching the global store processing to use the normal paradigm rather than adding special-case logic to track positions in global stores.

 

Note: there are two primary reasons that global stores behave this way:
 # We can write in batches during restoration, so the I/O may be more efficient
 # The global thread does not transition to RUNNING state until it reaches the (current) end of the input topic, which blocks other threads from joining against it, thereby improving the time synchronization of global KTable joins.

If we want to propose changing the bootstrapping pipeline for global threads, we should have some kind of answer to these concerns.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)