[jira] [Created] (KAFKA-6106) Postpone normal processing of tasks within a thread until restoration of all tasks have completed

["Guozhang Wang (JIRA)" <ji...@apache.org>]

Guozhang Wang created KAFKA-6106:
------------------------------------

             Summary: Postpone normal processing of tasks within a thread until restoration of all tasks have completed
                 Key: KAFKA-6106
                 URL: https://issues.apache.org/jira/browse/KAFKA-6106
             Project: Kafka
          Issue Type: Improvement
          Components: streams
    Affects Versions: 0.11.0.1, 1.0.0
            Reporter: Guozhang Wang
            Assignee: Guozhang Wang


Let's say a stream thread hosts multiple tasks, A and B. At the very beginning when A and B are assigned to the thread, the thread state is {{TASKS_ASSIGNED}}, and the thread start restoring these two tasks during this state using the restore consumer while using normal consumer for heartbeating.

If task A's restoration has completed earlier than task B, then the thread will start processing A immediately even when it is still in the {{TASKS_ASSIGNED}} phase. But processing task A will slow down restoration of task B since it is single-thread. So the thread's transition to {{RUNNING}} when all of its assigned tasks have completed restoring and now can be processed will be delayed.

Note that the streams instance's state will only transit to {{RUNNING}} when all of its threads have transit to {{RUNNING}}, so the instance's transition will also be delayed by this scenario.

We'd better to not start processing ready tasks immediately, but instead focus on restoration during the {{TASKS_ASSIGNED}} state to shorten the overall time of the instance's state transition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)