KIPs to broaden the utilization of Kafka by making it more user friendly

[Werner Daehn <we...@gmail.com>]

I would have a couple of ideas about Kafka and would love to get your
input. If I am totally off, if it is possible today without me noticing or
if you can embrace/do disregard the idea - appreciate all your feedback.

Statement 1: A typical scenario for Kafka can be coupling on-premise system
with a Kafka based cloud application. Today Kafka requires a direct
connection but the on-prem side does allow http only.

Kafka Rest is obviously one solution, but it is slow and has overhead.
Suggestion: In Kafka Connect add the ability to run connectors on-prem.
Each opens one long running http connection with the Kafka server streaming
the data. Or maybe a WebSocket as a bridge.


Statement 2: For above use cases you need to support security and
multi-tenant.

Suggestion: Above http connection should validate that a connector has the
proper certificate and is entitled to write into a Kafka topic.


Statement 3: Connect needs an UI for being configured by a non-Kafka user.

Suggestion: The connectors are hosted by a Tomcat which also acts as the
configuration, administration and monitoring UI for the on-prem side.


I have a few more thoughts about metadata (schema registry), a mapping
(Kafka Connect Transform) UI,... but above would be the most pressing
points for me.

Thanks in advance