Unexpected cluster rebalancing

[omer bar lev <om...@gmail.com>]

As part of our load tests on Kafka, we are trying to create a certain
number of sink connectors, each one with a unique name and topic and
therefore, each sink connector is in his own consumer group.
For every new connector added, a rebalance is triggered for all the
connectors (the more connectors created before, the longer the current
rebalance).
Our creation is done by sending parallel create requests to the connector
RestAPI (we have tried doing this in several ways – 1000 requests, the
desired amount, at once, and also in smaller “batches” – 10/20 requests).

We have a few questions:

Why is the rebalance happening?
Why does it take so long (can be hours)?
Is the rebalance configurable?

The cluster is deployed using strimzi on kuberentes.

The connectors are created with the following template:

POST connectors/HttpSinkConnector-{index}
{

  "name": "HttpSinkConnector-{index}",

  "config": {

    "connector.class": "com. connectors.rest.sink.HttpSinkConnector",

    "tasks.max": "1",

    "topics": "http-sink-connector-{index}",

    "http.api.url": "{ some url }",

    "headers": "Content-Type:application/json"

  }

}

Connector Worker configuration:

            offset.storage.topic=connect-cluster-offsets

            value.converter=org.apache.kafka.connect.json.JsonConverter

            config.storage.topic=connect-cluster-configs

            key.converter=org.apache.kafka.connect.json.JsonConverter

            group.id=connect-cluster

            status.storage.topic=connect-cluster-status

            config.storage.replication.factor=3

            connector.client.config.override.policy=All

            consumer.override.auto.offset.reset=latest

            heartbeat.interval.ms=100000

            key.converter.schemas.enable=false

            offset.storage.replication.factor=3

            session.timeout.ms=300000

            status.storage.replication.factor=3
            value.converter.schemas.enable=false