[jira] [Created] (KAFKA-13555) Consider number if input topic partitions for task assignment

["Matthias J. Sax (Jira)" <ji...@apache.org>]

Matthias J. Sax created KAFKA-13555:
---------------------------------------

             Summary: Consider number if input topic partitions for task assignment
                 Key: KAFKA-13555
                 URL: https://issues.apache.org/jira/browse/KAFKA-13555
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Matthias J. Sax


StreamsAssignor tries to distribute tasks evenly across all instances/threads of a Kafka Streams application. It knows about instances/thread (to give more capacity to instances with more thread), and it distinguishes between stateless and stateful tasks. We also try to not move state around but to use a sticky assignment if possible. However, the assignment does not take the number of input topic partitions into account.

For example, an upstream tasks could compute two joins, and thus has 3 input partitions, while a downstream task compute a follow up aggregation with a single input partitions (from the repartition topic). It could happen that one thread gets the 3 input partition tasks assigned, while the other thread get the single input partition tasks assigned resulting to an uneven partition assignment across both threads.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)