You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Kondrát Bertalan <kb...@gmail.com> on 2025/03/14 11:38:17 UTC
[DISCUSS] KIP-1144: Exposing a new public REST API for MirrorMaker2
Hi All,
I would like to start a discussion thread about the KIP-1144: Exposing a
new public REST API for MirrorMaker2
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1144%3A+Exposing+a+new+public+REST+API+for+MirrorMaker+2>
This KIP introduces a new REST API tailored for MirrorMaker2 operational
requirements. Tries to make the interaction with MirrorMaker2 easier by
providing a new abstraction layer on top of Connect, that can hide the
implementation details, and avoid managing the connectors directly.
The original REST API implementation (before KIP-710 removed the complete
public REST API) exposed the full Connect API, including features like
creating, modifying and removing connectors, manipulating tasks. This is
not only increased the operational complexity of MirrorMaker2 but allowed
users to manipulate the underlying connect clusters in an unintended way,
and also opened up attack vectors for malicious actors.
This proposal introduces a set of REST endpoints for MirrorMaker2 to expose
operational status and control information for replication flows. The
design greatly reduce the amount of endpoints compared to the Connect REST
API, and keep only the essential ones. Built on top of the Connect
functionality, this KIP introduces new endpoints built specifically for
MirrorMaker2 while applying the same BasicAuthSecurityRestExtension based
on JAAS.
With these endpoints, operators and tools can query the state of
MirrorMaker2 deployments, enabling easier monitoring.
I appreciate all your feedbacks and comments.
Thank you,
Berci
--
*Bertalan Kondrat*
------------------------------
Re: [DISCUSS] KIP-1144: Exposing a new public REST API for MirrorMaker2
Posted by Kondrát Bertalan <kb...@gmail.com>.
Hi Devs,
I still want to sell this idea :) I think it would be really beneficial for
the MM2 to have a proper REST API, to monitor the state of the replication
flows.
Please share your opinion on this.
Thank you!
On Tue, Mar 25, 2025 at 1:46 PM Kondrát Bertalan <kb...@gmail.com> wrote:
> Hi Team,
>
> I appreciate any feedback on this KIP, KIP-1144: Exposing a new public
> REST API for MirrorMaker2
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1144%3A+Exposing+a+new+public+REST+API+for+MirrorMaker+2>
> .
>
> Thank you!
>
>
> On Fri, Mar 14, 2025 at 4:07 PM Kondrát Bertalan <kb...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I have to make some correction to the description: so the KIP-710
>> introduced the REST API to solve the problem with the inter worker
>> communication. Previously, there were no REST APIs in MM2. Exposing the
>> full Connect API is a rejected alternative.
>>
>>
>>
>> On Fri, Mar 14, 2025 at 12:38 PM Kondrát Bertalan <kb...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I would like to start a discussion thread about the KIP-1144: Exposing
>>> a new public REST API for MirrorMaker2
>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1144%3A+Exposing+a+new+public+REST+API+for+MirrorMaker+2>
>>>
>>> This KIP introduces a new REST API tailored for MirrorMaker2 operational
>>> requirements. Tries to make the interaction with MirrorMaker2 easier by
>>> providing a new abstraction layer on top of Connect, that can hide the
>>> implementation details, and avoid managing the connectors directly.
>>>
>>> The original REST API implementation (before KIP-710 removed the
>>> complete public REST API) exposed the full Connect API, including features
>>> like creating, modifying and removing connectors, manipulating tasks. This
>>> is not only increased the operational complexity of MirrorMaker2 but
>>> allowed users to manipulate the underlying connect clusters in an
>>> unintended way, and also opened up attack vectors for malicious actors.
>>>
>>> This proposal introduces a set of REST endpoints for MirrorMaker2 to
>>> expose operational status and control information for replication flows.
>>> The design greatly reduce the amount of endpoints compared to the Connect
>>> REST API, and keep only the essential ones. Built on top of the Connect
>>> functionality, this KIP introduces new endpoints built specifically for
>>> MirrorMaker2 while applying the same BasicAuthSecurityRestExtension based
>>> on JAAS.
>>> With these endpoints, operators and tools can query the state of
>>> MirrorMaker2 deployments, enabling easier monitoring.
>>>
>>> I appreciate all your feedbacks and comments.
>>>
>>> Thank you,
>>> Berci
>>>
>>> --
>>> *Bertalan Kondrat*
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>
>>
>> --
>> *Bertalan Kondrat*
>> t. +36(70) 413-4801
>>
>>
>>
>>
>> ------------------------------
>>
>
>
> --
> *Bertalan Kondrat*
> t. +36(70) 413-4801
>
>
>
>
> ------------------------------
>
--
*Bertalan Kondrat*
t. +36(70) 413-4801
------------------------------
Re: [DISCUSS] KIP-1144: Exposing a new public REST API for MirrorMaker2
Posted by Kondrát Bertalan <kb...@gmail.com>.
Hi Team,
I appreciate any feedback on this KIP, KIP-1144: Exposing a new public REST
API for MirrorMaker2
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1144%3A+Exposing+a+new+public+REST+API+for+MirrorMaker+2>
.
Thank you!
On Fri, Mar 14, 2025 at 4:07 PM Kondrát Bertalan <kb...@gmail.com> wrote:
> Hi All,
>
> I have to make some correction to the description: so the KIP-710
> introduced the REST API to solve the problem with the inter worker
> communication. Previously, there were no REST APIs in MM2. Exposing the
> full Connect API is a rejected alternative.
>
>
>
> On Fri, Mar 14, 2025 at 12:38 PM Kondrát Bertalan <kb...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I would like to start a discussion thread about the KIP-1144: Exposing a
>> new public REST API for MirrorMaker2
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1144%3A+Exposing+a+new+public+REST+API+for+MirrorMaker+2>
>>
>> This KIP introduces a new REST API tailored for MirrorMaker2 operational
>> requirements. Tries to make the interaction with MirrorMaker2 easier by
>> providing a new abstraction layer on top of Connect, that can hide the
>> implementation details, and avoid managing the connectors directly.
>>
>> The original REST API implementation (before KIP-710 removed the complete
>> public REST API) exposed the full Connect API, including features like
>> creating, modifying and removing connectors, manipulating tasks. This is
>> not only increased the operational complexity of MirrorMaker2 but allowed
>> users to manipulate the underlying connect clusters in an unintended way,
>> and also opened up attack vectors for malicious actors.
>>
>> This proposal introduces a set of REST endpoints for MirrorMaker2 to
>> expose operational status and control information for replication flows.
>> The design greatly reduce the amount of endpoints compared to the Connect
>> REST API, and keep only the essential ones. Built on top of the Connect
>> functionality, this KIP introduces new endpoints built specifically for
>> MirrorMaker2 while applying the same BasicAuthSecurityRestExtension based
>> on JAAS.
>> With these endpoints, operators and tools can query the state of
>> MirrorMaker2 deployments, enabling easier monitoring.
>>
>> I appreciate all your feedbacks and comments.
>>
>> Thank you,
>> Berci
>>
>> --
>> *Bertalan Kondrat*
>>
>>
>>
>>
>>
>> ------------------------------
>>
>
>
> --
> *Bertalan Kondrat*
> t. +36(70) 413-4801
>
>
>
>
> ------------------------------
>
--
*Bertalan Kondrat*
t. +36(70) 413-4801
------------------------------
Re: [DISCUSS] KIP-1144: Exposing a new public REST API for MirrorMaker2
Posted by Kondrát Bertalan <kb...@gmail.com>.
Hi All,
I have to make some correction to the description: so the KIP-710
introduced the REST API to solve the problem with the inter worker
communication. Previously, there were no REST APIs in MM2. Exposing the
full Connect API is a rejected alternative.
On Fri, Mar 14, 2025 at 12:38 PM Kondrát Bertalan <kb...@gmail.com> wrote:
> Hi All,
>
> I would like to start a discussion thread about the KIP-1144: Exposing a
> new public REST API for MirrorMaker2
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1144%3A+Exposing+a+new+public+REST+API+for+MirrorMaker+2>
>
> This KIP introduces a new REST API tailored for MirrorMaker2 operational
> requirements. Tries to make the interaction with MirrorMaker2 easier by
> providing a new abstraction layer on top of Connect, that can hide the
> implementation details, and avoid managing the connectors directly.
>
> The original REST API implementation (before KIP-710 removed the complete
> public REST API) exposed the full Connect API, including features like
> creating, modifying and removing connectors, manipulating tasks. This is
> not only increased the operational complexity of MirrorMaker2 but allowed
> users to manipulate the underlying connect clusters in an unintended way,
> and also opened up attack vectors for malicious actors.
>
> This proposal introduces a set of REST endpoints for MirrorMaker2 to
> expose operational status and control information for replication flows.
> The design greatly reduce the amount of endpoints compared to the Connect
> REST API, and keep only the essential ones. Built on top of the Connect
> functionality, this KIP introduces new endpoints built specifically for
> MirrorMaker2 while applying the same BasicAuthSecurityRestExtension based
> on JAAS.
> With these endpoints, operators and tools can query the state of
> MirrorMaker2 deployments, enabling easier monitoring.
>
> I appreciate all your feedbacks and comments.
>
> Thank you,
> Berci
>
> --
> *Bertalan Kondrat*
>
>
>
>
>
> ------------------------------
>
--
*Bertalan Kondrat*
t. +36(70) 413-4801
------------------------------