[jira] [Created] (KAFKA-5578) Streams Task Assignor should consider the staleness of state directories when allocating tasks

["Damian Guy (JIRA)" <ji...@apache.org>]

Damian Guy created KAFKA-5578:
---------------------------------

             Summary: Streams Task Assignor should consider the staleness of state directories when allocating tasks
                 Key: KAFKA-5578
                 URL: https://issues.apache.org/jira/browse/KAFKA-5578
             Project: Kafka
          Issue Type: Bug
            Reporter: Damian Guy


During task assignment we use the presence of a state directory to assign precedence to which instances should be assigned the task. We first chose previous active tasks, but then fall back to the existence of a state dir. Unfortunately we don't take into account the recency of the data from the available state dirs. So in the case where a task has run on many instances, it may be that we chose an instance that has relatively old data.

When doing task assignment we should take into consideration the age of the data in the state dirs. We could use the data from the checkpoint files to determine which instance is most up-to-date and attempt to assign accordingly (obviously making sure that tasks are still balanced across available instances)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)