[jira] [Created] (KAFKA-13450) Kafka broker to return alternating sets of listeners to clients for multipathing

["Sudarshan Pathalam (Jira)" <ji...@apache.org>]

Sudarshan Pathalam created KAFKA-13450:
------------------------------------------

             Summary: Kafka broker to return alternating sets of listeners to clients for multipathing
                 Key: KAFKA-13450
                 URL: https://issues.apache.org/jira/browse/KAFKA-13450
             Project: Kafka
          Issue Type: Improvement
          Components: network
            Reporter: Sudarshan Pathalam


It is not uncommon to deploy Kafka for high throughput use cases. In this context operators could have multiple network paths for client to broker traffic.

The use of multiple network paths is not only to cover for failure conditions but also to balance load between the available paths. Common link speeds (currently) in data centers/cloud are 10Gbps/40Gbps and when there is a need to scale past the link speed multiple paths (multipathing) to the target (Kafka) is desired. The option of bundling multiple links in to a single channel is not commonly offered as a user facing feature in the Cloud.

However when using DNS to load balance between the available network paths, TTL caching creates traffic skew between the network paths causing either deterioration or failure. 

It would be ideal for the broker to return alternating sets of listener names to the metadata requests made by the client for even distribution of traffic between the available network paths. KIP 302 provides the ability for the clients to attempt to use all IP address for a name but does not cover situations when there is a need for multiple names each pointing to a separate network path. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)