You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ivan Yurchenko <iv...@gmail.com> on 2019/06/13 14:20:53 UTC
[DISCUSS] KIP-477: Add PATCH method for connector config in Connect
REST API
Hello,
I'd like to start the discussion of KIP-477: Add PATCH method for connector
config in Connect REST API.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
There is also a draft PR: https://github.com/apache/kafka/pull/6934.
Thank you.
Ivan
Re: [DISCUSS] KIP-477: Add PATCH method for connector config in
Connect REST API
Posted by Chris Egerton <ch...@confluent.io.INVALID>.
Hi all,
Know it's been a while for this KIP but in my personal experience the value
of a PATCH method in the REST API has actually increased over time and I'd
love to have this option for quick, stop-the-bleeding remediation efforts.
One thought that comes to mind is that sometimes it may be useful to
explicitly remove properties from a connector configuration. We might
permit this by allowing users to specify null (the JSON literal, not a
string containing the characters "null") as the value for to-be-removed
properties.
I'd love to see this change if you're still interested in driving it, Ivan.
Hopefully we can give it the attention it deserves in the upcoming months!
Cheers,
Chris
On Fri, Jun 28, 2019 at 4:56 AM Ivan Yurchenko <iv...@gmail.com>
wrote:
> Thank you for your feedback Ryanne!
> These are all surely valid concerns and PATCH isn't really necessary or
> suitable for normal production configuration management. However, there are
> cases where quick patching of the configuration is useful, such as hot
> fixes of production or in development.
>
> Overall, the change itself is really tiny and if the cost-benefit balance
> is positive, I'd still like to drive it further.
>
> Ivan
>
> On Wed, 26 Jun 2019 at 17:45, Ryanne Dolan <ry...@gmail.com> wrote:
>
> > Ivan, I looked at adding PATCH a while ago as well. I decided not to
> pursue
> > the idea for a few reasons:
> >
> > 1) PATCH is still racy. For example, if you want to add a topic to the
> > "topics" property, you still need to read, modify, and write the existing
> > value. To handle this, you'd need to support atomic sub-document
> > operations, which I don't see happening.
> >
> > 2) A common pattern is to store your configurations in git or something,
> > and then apply them via PUT. Throw in some triggers or jenkins etc, and
> you
> > have a more robust solution than PATCH provides.
> >
> > 3) For properties that change a lot, it's possible to use an out-of-band
> > data source, e.g. Kafka or Zookeeper, and then have your Connector
> > subscribe to changes. I've done something like this to enable dynamic
> > reconfiguration of Connectors from command-line tools and dashboards
> > without involving the Connect REST API at all. Moreover, I've done so in
> an
> > atomic, non-racy way.
> >
> > So I don't think PATCH is strictly necessary nor sufficient for atomic
> > partial updates. That said, it doesn't hurt and I'm happy to support the
> > KIP.
> >
> > Ryanne
> >
> > On Tue, Jun 25, 2019 at 12:15 PM Ivan Yurchenko <
> ivan0yurchenko@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Since Kafka 2.3 has just been release and more people may have time to
> > look
> > > at this now, I'd like to bump this discussion.
> > > Thanks.
> > >
> > > Ivan
> > >
> > >
> > > On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko <ivan0yurchenko@gmail.com
> >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I'd like to start the discussion of KIP-477: Add PATCH method for
> > > > connector config in Connect REST API.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
> > > >
> > > > There is also a draft PR: https://github.com/apache/kafka/pull/6934.
> > > >
> > > > Thank you.
> > > >
> > > > Ivan
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-477: Add PATCH method for connector config in
Connect REST API
Posted by Ivan Yurchenko <iv...@gmail.com>.
Thank you for your feedback Ryanne!
These are all surely valid concerns and PATCH isn't really necessary or
suitable for normal production configuration management. However, there are
cases where quick patching of the configuration is useful, such as hot
fixes of production or in development.
Overall, the change itself is really tiny and if the cost-benefit balance
is positive, I'd still like to drive it further.
Ivan
On Wed, 26 Jun 2019 at 17:45, Ryanne Dolan <ry...@gmail.com> wrote:
> Ivan, I looked at adding PATCH a while ago as well. I decided not to pursue
> the idea for a few reasons:
>
> 1) PATCH is still racy. For example, if you want to add a topic to the
> "topics" property, you still need to read, modify, and write the existing
> value. To handle this, you'd need to support atomic sub-document
> operations, which I don't see happening.
>
> 2) A common pattern is to store your configurations in git or something,
> and then apply them via PUT. Throw in some triggers or jenkins etc, and you
> have a more robust solution than PATCH provides.
>
> 3) For properties that change a lot, it's possible to use an out-of-band
> data source, e.g. Kafka or Zookeeper, and then have your Connector
> subscribe to changes. I've done something like this to enable dynamic
> reconfiguration of Connectors from command-line tools and dashboards
> without involving the Connect REST API at all. Moreover, I've done so in an
> atomic, non-racy way.
>
> So I don't think PATCH is strictly necessary nor sufficient for atomic
> partial updates. That said, it doesn't hurt and I'm happy to support the
> KIP.
>
> Ryanne
>
> On Tue, Jun 25, 2019 at 12:15 PM Ivan Yurchenko <iv...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Since Kafka 2.3 has just been release and more people may have time to
> look
> > at this now, I'd like to bump this discussion.
> > Thanks.
> >
> > Ivan
> >
> >
> > On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko <iv...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I'd like to start the discussion of KIP-477: Add PATCH method for
> > > connector config in Connect REST API.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
> > >
> > > There is also a draft PR: https://github.com/apache/kafka/pull/6934.
> > >
> > > Thank you.
> > >
> > > Ivan
> > >
> >
>
Re: [DISCUSS] KIP-477: Add PATCH method for connector config in
Connect REST API
Posted by Ryanne Dolan <ry...@gmail.com>.
Ivan, I looked at adding PATCH a while ago as well. I decided not to pursue
the idea for a few reasons:
1) PATCH is still racy. For example, if you want to add a topic to the
"topics" property, you still need to read, modify, and write the existing
value. To handle this, you'd need to support atomic sub-document
operations, which I don't see happening.
2) A common pattern is to store your configurations in git or something,
and then apply them via PUT. Throw in some triggers or jenkins etc, and you
have a more robust solution than PATCH provides.
3) For properties that change a lot, it's possible to use an out-of-band
data source, e.g. Kafka or Zookeeper, and then have your Connector
subscribe to changes. I've done something like this to enable dynamic
reconfiguration of Connectors from command-line tools and dashboards
without involving the Connect REST API at all. Moreover, I've done so in an
atomic, non-racy way.
So I don't think PATCH is strictly necessary nor sufficient for atomic
partial updates. That said, it doesn't hurt and I'm happy to support the
KIP.
Ryanne
On Tue, Jun 25, 2019 at 12:15 PM Ivan Yurchenko <iv...@gmail.com>
wrote:
> Hi,
>
> Since Kafka 2.3 has just been release and more people may have time to look
> at this now, I'd like to bump this discussion.
> Thanks.
>
> Ivan
>
>
> On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko <iv...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I'd like to start the discussion of KIP-477: Add PATCH method for
> > connector config in Connect REST API.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
> >
> > There is also a draft PR: https://github.com/apache/kafka/pull/6934.
> >
> > Thank you.
> >
> > Ivan
> >
>
Re: [DISCUSS] KIP-477: Add PATCH method for connector config in
Connect REST API
Posted by Ivan Yurchenko <iv...@gmail.com>.
Hi,
Since Kafka 2.3 has just been release and more people may have time to look
at this now, I'd like to bump this discussion.
Thanks.
Ivan
On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko <iv...@gmail.com>
wrote:
> Hello,
>
> I'd like to start the discussion of KIP-477: Add PATCH method for
> connector config in Connect REST API.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
>
> There is also a draft PR: https://github.com/apache/kafka/pull/6934.
>
> Thank you.
>
> Ivan
>