[jira] [Commented] (KAFKA-6669) KIP-271: Add NetworkClient redirector

["Sergey Korytnikov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401859#comment-16401859 ] 

Sergey Korytnikov commented on KAFKA-6669:
------------------------------------------

*Usa-case example:* 

Kafka cluster is running in OpenShift as a distributed setup of 5 brokers. Each broker's pod service port is exposed as a separate port on localhost interface. KafkaProducer connects to cluster from outside of OpenShift environment and specifies bootstrap servers as:
|{{bootstrap.servers=localhost:}}{{9092}}{{,localhost:}}{{9093}}{{,localhost:}}{{9094}}{{,localhost:}}{{9095}}{{,localhost:}}{{9096}}|

When message is getting sent, Kafka client recieves clusters metadata with pods internal DNS names and delivery is failed:
|{{DEBUG org.apache.kafka.clients.NetworkClient[kafka-producer-network-thread \| producer-}}{{1}}{{] - [Producer clientId=producer-}}{{1}}{{] Initiating connection to node kafka-}}{{2}}{{.kafka.myproject.svc.cluster.local:}}{{9092}} {{(id: }}{{1003}} {{rack: }}{{null}}{{)}}
{{DEBUG org.apache.kafka.clients.NetworkClient[kafka-producer-network-thread \| producer-}}{{1}}{{] - [Producer clientId=producer-}}{{1}}{{] Error connecting to node kafka-}}{{2}}{{.kafka.myproject.svc.cluster.local:}}{{9092}} {{(id: }}{{1003}} {{rack: }}{{null}}{{)}}
{{java.io.IOException: Can't resolve address: kafka-}}{{2}}{{.kafka.myproject.svc.cluster.local:}}{{9092}}
{{    }}{{at org.apache.kafka.common.network.Selector.doConnect(Selector.java:}}{{258}}{{)}}
{{    }}{{at org.apache.kafka.common.network.Selector.connect(Selector.java:}}{{237}}{{)}}
{{    }}{{at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:}}{{792}}{{)}}
{{    }}{{at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:}}{{230}}{{)}}
{{    }}{{at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:}}{{263}}{{)}}
{{    }}{{at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:}}{{238}}{{)}}
{{    }}{{at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:}}{{163}}{{)}}
{{    }}{{at java.base/java.lang.Thread.run(Thread.java:}}{{844}}{{)}}
{{Caused by: java.nio.channels.UnresolvedAddressException}}
{{    }}{{at java.base/sun.nio.ch.Net.checkAddress(Net.java:}}{{112}}{{)}}
{{    }}{{at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:}}{{622}}{{)}}
{{    }}{{at org.apache.kafka.common.network.Selector.doConnect(Selector.java:}}{{256}}{{)}}
{{    }}{{... }}{{7}} {{more}}|

 

> KIP-271: Add NetworkClient redirector
> -------------------------------------
>
>                 Key: KAFKA-6669
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6669
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients
>            Reporter: Sergey Korytnikov
>            Priority: Major
>
> "KIP-271 Add NetworkClient redirector" purposes the change to accommodate environments without proper DNS support, when KafkaProducer or KafkaConsumer tries to connect to Kafka brokers that are inside another network.
> Currently, Kafka client connection fails with "[java.io|http://java.io/].IOException: Can't resolve address" after Kafka cluster metadata is updated with internal DNS names of brokers, unreachable by Client. The example of such configuration might be a Java Client calling Kafka Cluster from outside of Kubernetes cluster or AWS network.
> The KIP improves NetworkClient to redirect the call to alternative network address and provides developers with additional Configurable interface to specify redirection rules.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)