You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jason Gustafson <ja...@confluent.io> on 2016/03/31 18:31:32 UTC
[DISCUSS] KIP-52 - Add Connector Control APIs
Hi All, I've written a short KIP to add control APIs to Kafka Connect to
make administration easier:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs.
Please let me know your thoughts.
Thanks,
Jason
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Gwen Shapira <gw...@confluent.io>.
Looks awesome to me, and looks like no one objects. Maybe just start a vote?
On Thu, Mar 31, 2016 at 9:31 AM, Jason Gustafson <ja...@confluent.io> wrote:
> Hi All, I've written a short KIP to add control APIs to Kafka Connect to
> make administration easier:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> .
> Please let me know your thoughts.
>
> Thanks,
> Jason
>
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Ismael, thanks for the comments. I've updated the wiki:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
.
-Jason
On Wed, Apr 6, 2016 at 4:53 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Jason,
>
> The suggested changes make sense. I think it's also worth clarifying the
> documentation for the endpoints whose behaviour will remain the same.
>
> Ismael
>
> On Wed, Apr 6, 2016 at 8:39 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hey Ismael,
> >
> > The intent is only to indicate that we've accepted the request, so the
> > no-op case has no separate response code in the current proposal. The
> > question is whether that is sufficient? For pause/resume, you can verify
> > the effect using the status endpoint which seems OK, but restart is
> > trickier since the final state of the task may match the current state.
> One
> > option is to make the restart API synchronous. In that case, successful
> > completion of the request would indicate a successful restart of the
> task.
> > If there is another restart in progress, maybe we could just return 409
> > (Conflict) so the client knows a restart is already in progress. I
> > considered doing something more complex like blocking until the other
> > restart had completed, but it seems simpler to let the client retry if it
> > wants to. Does that make sense?
> >
> > Thanks,
> > Jason
> >
> > On Tue, Apr 5, 2016 at 11:49 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Thank you Jason. One more question, the response codes are described as
> > for
> > > all endpoints:
> > >
> > > Response Codes: 202 (Accepted) on successful restart initiation, 404 if
> > the
> > > connector doesn't exist
> > >
> > > What is the response code in the no-op case?
> > >
> > > Ismael
> > >
> > >
> > > On Mon, Apr 4, 2016 at 6:52 PM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > Hey Ismael, thanks for having a look. I've changed pause/resume to
> use
> > > PUT.
> > > >
> > > > -Jason
> > > >
> > > > On Sun, Apr 3, 2016 at 7:30 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > Ismael,
> > > > >
> > > > > Great point. Pause and resume should be idempotent and actually
> > > represent
> > > > > updating a resource that gets written to Kafka (although I must
> > admit I
> > > > > don't know if the use of 202/Accepted should affect this at all),
> the
> > > > > restart endpoints seem a bit different as they are one-off
> immediate
> > > > > commands.
> > > > >
> > > > > -Ewen
> > > > >
> > > > > On Fri, Apr 1, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Do I understand correctly that these requests are idempotent? If
> > so,
> > > > why
> > > > > > are they POSTs instead of PUTs?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Mar 31, 2016 at 5:31 PM, Jason Gustafson <
> > jason@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All, I've written a short KIP to add control APIs to Kafka
> > > Connect
> > > > > to
> > > > > > > make administration easier:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> > > > > > > .
> > > > > > > Please let me know your thoughts.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Ewen
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jason,
The suggested changes make sense. I think it's also worth clarifying the
documentation for the endpoints whose behaviour will remain the same.
Ismael
On Wed, Apr 6, 2016 at 8:39 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Ismael,
>
> The intent is only to indicate that we've accepted the request, so the
> no-op case has no separate response code in the current proposal. The
> question is whether that is sufficient? For pause/resume, you can verify
> the effect using the status endpoint which seems OK, but restart is
> trickier since the final state of the task may match the current state. One
> option is to make the restart API synchronous. In that case, successful
> completion of the request would indicate a successful restart of the task.
> If there is another restart in progress, maybe we could just return 409
> (Conflict) so the client knows a restart is already in progress. I
> considered doing something more complex like blocking until the other
> restart had completed, but it seems simpler to let the client retry if it
> wants to. Does that make sense?
>
> Thanks,
> Jason
>
> On Tue, Apr 5, 2016 at 11:49 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Thank you Jason. One more question, the response codes are described as
> for
> > all endpoints:
> >
> > Response Codes: 202 (Accepted) on successful restart initiation, 404 if
> the
> > connector doesn't exist
> >
> > What is the response code in the no-op case?
> >
> > Ismael
> >
> >
> > On Mon, Apr 4, 2016 at 6:52 PM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > Hey Ismael, thanks for having a look. I've changed pause/resume to use
> > PUT.
> > >
> > > -Jason
> > >
> > > On Sun, Apr 3, 2016 at 7:30 PM, Ewen Cheslack-Postava <
> ewen@confluent.io
> > >
> > > wrote:
> > >
> > > > Ismael,
> > > >
> > > > Great point. Pause and resume should be idempotent and actually
> > represent
> > > > updating a resource that gets written to Kafka (although I must
> admit I
> > > > don't know if the use of 202/Accepted should affect this at all), the
> > > > restart endpoints seem a bit different as they are one-off immediate
> > > > commands.
> > > >
> > > > -Ewen
> > > >
> > > > On Fri, Apr 1, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Do I understand correctly that these requests are idempotent? If
> so,
> > > why
> > > > > are they POSTs instead of PUTs?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Mar 31, 2016 at 5:31 PM, Jason Gustafson <
> jason@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi All, I've written a short KIP to add control APIs to Kafka
> > Connect
> > > > to
> > > > > > make administration easier:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> > > > > > .
> > > > > > Please let me know your thoughts.
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ewen
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Ismael,
The intent is only to indicate that we've accepted the request, so the
no-op case has no separate response code in the current proposal. The
question is whether that is sufficient? For pause/resume, you can verify
the effect using the status endpoint which seems OK, but restart is
trickier since the final state of the task may match the current state. One
option is to make the restart API synchronous. In that case, successful
completion of the request would indicate a successful restart of the task.
If there is another restart in progress, maybe we could just return 409
(Conflict) so the client knows a restart is already in progress. I
considered doing something more complex like blocking until the other
restart had completed, but it seems simpler to let the client retry if it
wants to. Does that make sense?
Thanks,
Jason
On Tue, Apr 5, 2016 at 11:49 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Thank you Jason. One more question, the response codes are described as for
> all endpoints:
>
> Response Codes: 202 (Accepted) on successful restart initiation, 404 if the
> connector doesn't exist
>
> What is the response code in the no-op case?
>
> Ismael
>
>
> On Mon, Apr 4, 2016 at 6:52 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hey Ismael, thanks for having a look. I've changed pause/resume to use
> PUT.
> >
> > -Jason
> >
> > On Sun, Apr 3, 2016 at 7:30 PM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> > wrote:
> >
> > > Ismael,
> > >
> > > Great point. Pause and resume should be idempotent and actually
> represent
> > > updating a resource that gets written to Kafka (although I must admit I
> > > don't know if the use of 202/Accepted should affect this at all), the
> > > restart endpoints seem a bit different as they are one-off immediate
> > > commands.
> > >
> > > -Ewen
> > >
> > > On Fri, Apr 1, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Do I understand correctly that these requests are idempotent? If so,
> > why
> > > > are they POSTs instead of PUTs?
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Mar 31, 2016 at 5:31 PM, Jason Gustafson <jason@confluent.io
> >
> > > > wrote:
> > > >
> > > > > Hi All, I've written a short KIP to add control APIs to Kafka
> Connect
> > > to
> > > > > make administration easier:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> > > > > .
> > > > > Please let me know your thoughts.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
>
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Ismael Juma <is...@juma.me.uk>.
Thank you Jason. One more question, the response codes are described as for
all endpoints:
Response Codes: 202 (Accepted) on successful restart initiation, 404 if the
connector doesn't exist
What is the response code in the no-op case?
Ismael
On Mon, Apr 4, 2016 at 6:52 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Ismael, thanks for having a look. I've changed pause/resume to use PUT.
>
> -Jason
>
> On Sun, Apr 3, 2016 at 7:30 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > Ismael,
> >
> > Great point. Pause and resume should be idempotent and actually represent
> > updating a resource that gets written to Kafka (although I must admit I
> > don't know if the use of 202/Accepted should affect this at all), the
> > restart endpoints seem a bit different as they are one-off immediate
> > commands.
> >
> > -Ewen
> >
> > On Fri, Apr 1, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Jason,
> > >
> > > Do I understand correctly that these requests are idempotent? If so,
> why
> > > are they POSTs instead of PUTs?
> > >
> > > Ismael
> > >
> > > On Thu, Mar 31, 2016 at 5:31 PM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > Hi All, I've written a short KIP to add control APIs to Kafka Connect
> > to
> > > > make administration easier:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> > > > .
> > > > Please let me know your thoughts.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Ismael, thanks for having a look. I've changed pause/resume to use PUT.
-Jason
On Sun, Apr 3, 2016 at 7:30 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Ismael,
>
> Great point. Pause and resume should be idempotent and actually represent
> updating a resource that gets written to Kafka (although I must admit I
> don't know if the use of 202/Accepted should affect this at all), the
> restart endpoints seem a bit different as they are one-off immediate
> commands.
>
> -Ewen
>
> On Fri, Apr 1, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Jason,
> >
> > Do I understand correctly that these requests are idempotent? If so, why
> > are they POSTs instead of PUTs?
> >
> > Ismael
> >
> > On Thu, Mar 31, 2016 at 5:31 PM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > Hi All, I've written a short KIP to add control APIs to Kafka Connect
> to
> > > make administration easier:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> > > .
> > > Please let me know your thoughts.
> > >
> > > Thanks,
> > > Jason
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Ismael,
Great point. Pause and resume should be idempotent and actually represent
updating a resource that gets written to Kafka (although I must admit I
don't know if the use of 202/Accepted should affect this at all), the
restart endpoints seem a bit different as they are one-off immediate
commands.
-Ewen
On Fri, Apr 1, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Jason,
>
> Do I understand correctly that these requests are idempotent? If so, why
> are they POSTs instead of PUTs?
>
> Ismael
>
> On Thu, Mar 31, 2016 at 5:31 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi All, I've written a short KIP to add control APIs to Kafka Connect to
> > make administration easier:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> > .
> > Please let me know your thoughts.
> >
> > Thanks,
> > Jason
> >
>
--
Thanks,
Ewen
Re: [DISCUSS] KIP-52 - Add Connector Control APIs
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jason,
Do I understand correctly that these requests are idempotent? If so, why
are they POSTs instead of PUTs?
Ismael
On Thu, Mar 31, 2016 at 5:31 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hi All, I've written a short KIP to add control APIs to Kafka Connect to
> make administration easier:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs
> .
> Please let me know your thoughts.
>
> Thanks,
> Jason
>