[jira] [Created] (KAFKA-20998) Add HA task assignor

["Matthias J. Sax (Jira)" <ji...@apache.org>]

Matthias J. Sax created KAFKA-20998:
---------------------------------------

             Summary: Add HA task assignor
                 Key: KAFKA-20998
                 URL: https://issues.apache.org/jira/browse/KAFKA-20998
             Project: Kafka
          Issue Type: Sub-task
          Components: group-coordinator, streams
            Reporter: Matthias J. Sax


The "classic" protocol offers two built-in assignors, "sticky" and "ha_assignor". So far, "streams" only offers the sticky assignor.

KIP-1017 defines both "sticky" and "ha_assignor" as options, too, and we can now close this gap.

However, we should amend the KIP slightly, and call the second built-in assignor "balanced" (instead of "ha_assignor" as currently specified on the KIP). The reason is, that high-availability features are delivered independently on the assignment strategy with "streams" protocol, ie, "warmup tasks" are inserted by the new refiner step, not the task assignor. Thus, the new assignor is not a 1:1 of the classic HA-assignor, but only need to deliver a different ("balanced") task assignment compare to sticky assignor.

We can start with a 1:1 translation of he existing task assignment algorithm, but should use JMH benchmarks, and refine as needed, similar to what we did for the sticky assignor. Because task assignment is running inside the GC, we need to ensure that the task assignor performance is good, so it doesn't put too much load on the broker.

The classic HA-assignor also supports rack-aware standby task assignment, which we should exclude in this ticket. We rather want so deliver rack-aware standby task assignment for both "sticky" and "balanced" assignor, and are tracking this in it's own sub-task.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)