[jira] [Commented] (KAFKA-5635) KIP-181 Kafka-Connect integrate with kafka ReST Proxy

["Ewen Cheslack-Postava (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458273#comment-16458273 ] 

Ewen Cheslack-Postava commented on KAFKA-5635:
----------------------------------------------

[~dhananjaydp] I'm inclined to close this as WONTFIX as long as REST proxies are third party – there's not really a clean way to support this generically since even with Confluent's REST proxy there are differences between v1 and v2 APIs (due to old/new Java clients) that are difficult to generalize.

I think the intended goal here is great (i.e., provide compatibility), but ultimately it seems to mostly expose a limitation of the service you're working with. A REST API is a good compatibility layer to provide, but I would think most realistic cloud Kafka services that care about things like performance and scalability that are core to Kafka's value prop would support a native interface as well. Without it, many services layered on top of those protocols wouldn't work – Connect and Streams from Apache Kafka itself, as well as numerous other services such as KSQL, Spark Streaming, Flink, etc. I'm not convinced that adding REST proxy support to *all* of these in order to make them compatible with cloud services that don't expose Kafka's basic protocol layer would be in the interest of the project.

> KIP-181 Kafka-Connect integrate with kafka ReST Proxy
> -----------------------------------------------------
>
>                 Key: KAFKA-5635
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5635
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Dhananjay Patkar
>            Priority: Major
>              Labels: features, newbie
>
> Kafka connect currently uses kafka clients which directly connect to kafka brokers. 
> In a use case, wherein I have many kafka connect [producers] running remotely its a challenge to configure broker information on every connect agent.
> Also, in case of IP change [upgrade or cluster re-creation], we need to update every remote connect configuration.
> If kafka connect source connectors talk to ReST endpoint then client is unaware of broker details. This way we can transparently upgrade / re-create kafka cluster as long as ReST endpoint remains same.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)