[jira] [Created] (KAFKA-9113) Clean up task management

["Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

Sophie Blee-Goldman created KAFKA-9113:
------------------------------------------

             Summary: Clean up task management
                 Key: KAFKA-9113
                 URL: https://issues.apache.org/jira/browse/KAFKA-9113
             Project: Kafka
          Issue Type: Improvement
    Affects Versions: 2.4.0
            Reporter: Sophie Blee-Goldman


Along KIP-429 we did a lot of refactoring of the task management classes, including the TaskManager and AssignedTasks (and children).  While hopefully easier to reason about there's still significant opportunity for further cleanup including safer state tracking.  Some potential improvements:

1) Verify that no tasks are ever in more than one state at once. One possibility is to just check that the suspended, created, restoring, and running maps are all disjoint, but this begs the question of when and where to do those checks, and how often. Another idea might be to put all tasks into a single map and just track their state on a per-task basis. Whatever it is should be aware that some methods are on the critical code path, and should not be burdened with excessive safety checks (ie AssignedStreamTasks#process)

2) Cleanup of closing and/or shutdown logic – there are some potential improvements to be made here as well, for example AssignedTasks currently implements a closeZombieTask method despite the fact that standby tasks are never zombies. 

3)  The StoreChangelogReader also interacts with (only) the AssignedStreamsTasks class, through the TaskManager. It can be difficult to reason about these interactions and the state of the changelog reader.

4) All 4 classes and their state have very strict consistency requirements that currently are almost impossible to verify, which has already resulted in several bugs that we were lucky to catch in time. We should tighten up how these classes manage their own state, and how the overall state is managed between them, so that it is easy to make changes without introducing new bugs because one class updated its own state without knowing it needed to tell another class to also update its



--
This message was sent by Atlassian Jira
(v8.3.4#803005)