You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Viktor Somogyi-Vass <vi...@gmail.com> on 2018/09/27 10:34:41 UTC
[DISCUSS] KIP-377: TopicCommand to use AdminClient
Hi All,
This is the continuation of the old KIP-375 with the same title:
https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
The problem there was that two KIPs were created around the same time and I
chose to reorganize mine a bit and give it a new number to avoid
duplication.
The KIP summary here once again:
I wrote up a relatively simple KIP about improving the Kafka protocol and
the TopicCommand tool to support the new Java based AdminClient and
hopefully to deprecate the Zookeeper side of it.
I would be happy to receive some opinions about this. In general I think
this would be an important addition as this is one of the few left but
important tools that still uses direct Zookeeper connection.
Here is the link for the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
Cheers,
Viktor
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi All,
Colin, thanks for the heads-up. I'll rethink this metadata protocol thing
as in a global sense there might be other options as you mentioned and
start separate a discussion.
I'll start a vote soon as the KIP itself is relatively simple.
Viktor
On Tue, Oct 23, 2018 at 3:33 AM Kevin Lu <lu...@berkeley.edu> wrote:
> Hi Viktor,
>
> +1 to this KIP.
>
> I would very much like to see AdminClient in TopicCommand. This would also
> allow us to efficiently implement new features like the "--under-min-isr"
> option I proposed in KIP-351
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--under-min-isr+option+to+describe+topics+command
> >
> .
>
> Thanks.
>
> Regards,
> Kevin
>
> On Sat, Oct 20, 2018 at 10:52 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Viktor,
> >
> > Sounds good. If you want to propose a way of improving the metadata
> > protocol so that "[deleted]" could be supported, you could probably
> create
> > that KIP in parallel.
> >
> > The last KIP in that area that I can remember is KIP-142, which didn't
> get
> > adopted (yet?)
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster
> >
> > There have been other discussions though. In general there are a lot of
> > features that would be nice to have in the metadata protocol (pagniation,
> > regexes, skip stuff we don't need).
> >
> > best,
> > Colin
> >
> >
> > On Tue, Oct 16, 2018, at 10:11, Viktor Somogyi-Vass wrote:
> > > Hi Colin,
> > >
> > > Thanks, it makes sense and simplifies this KIP tremendously. I'll move
> > this
> > > section to the rejected alternatives with a note that KIP-142 will have
> > > this feature.
> > > On a similar note: is there a KIP for describe topics protocol or have
> > you
> > > been thinking about it? I guess there it's the same problem, we often
> > don't
> > > want to forward the entire metadata.
> > >
> > > Viktor
> > >
> > > On Fri, Oct 12, 2018 at 12:03 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > >
> > > > Hi Viktor,
> > > >
> > > > Thanks for bumping this thread.
> > > >
> > > > I think we should just focus on transitioning the TopicCommand to use
> > > > AdminClient, and talk about protocol changes in a separate KIP.
> > Protocol
> > > > changes often involve a lot of discussion. This does mean that we
> > couldn't
> > > > implement the "list topics under deletion" feature when using
> > AdminClient
> > > > at the moment. We could add a note to the tool output indicating
> this.
> > > >
> > > > We should move the protocol discussion to a separate thread.
> Probably
> > > > also look at KIP-142 as well.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote:
> > > > > Hi All,
> > > > >
> > > > > Would like to bump this as the conversation sank a little bit, but
> > more
> > > > > importantly I'd like to validate my plans/ideas on extending the
> > Metadata
> > > > > protocol. I was thinking about two other alternatives, namely:
> > > > > 1. Create a ListTopicUnderDeletion protocol. This however would be
> > > > > unnecessary: it'd have one very narrow functionality which we can't
> > > > extend.
> > > > > I'd make sense to have a list topics or describe topics protocol
> > where we
> > > > > can list/describe topics under deletion but for normal
> > listing/describing
> > > > > we already use the metadata, so it would be a duplication of
> > > > functionality.
> > > > > 2. DeleteTopicsResponse could return the topics under deletion if
> the
> > > > > request's argument list is empty which might make sense at the
> first
> > > > look,
> > > > > but actually we'd mix the query functionality with the delete
> > > > functionality
> > > > > which is counterintuitive.
> > > > >
> > > > > Even though most clients won't need these "limbo" topics (which are
> > under
> > > > > deletion) in the foreseeable future, it can be considered as part
> of
> > the
> > > > > cluster state or metadata and to me it makes sense. Also it doesn't
> > have
> > > > a
> > > > > big overhead in the response size as typically users don't delete
> > topics
> > > > > too often as far as I experienced.
> > > > >
> > > > > I'd be happy to receive some ideas/feedback on this.
> > > > >
> > > > > Cheers,
> > > > > Viktor
> > > > >
> > > > >
> > > > > On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass <
> > > > viktorsomogyi@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I made an update to the KIP. Just in short:
> > > > > > Currently KafkaAdminClient.describeTopics() and
> > > > > > KafkaAdminClient.listTopics() uses the Metadata protocol to
> acquire
> > > > topic
> > > > > > information. The returned response however won't contain the
> topics
> > > > that
> > > > > > are under deletion but couldn't complete yet (for instance
> because
> > of
> > > > some
> > > > > > replicas offline), therefore it is not possible to implement the
> > > > current
> > > > > > command's "marked for deletion" feature. To get around this I
> > > > introduced
> > > > > > some changes in the Metadata protocol.
> > > > > >
> > > > > > Thanks,
> > > > > > Viktor
> > > > > >
> > > > > > On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass <
> > > > > > viktorsomogyi@gmail.com> wrote:
> > > > > >
> > > > > >> Hi Mickael,
> > > > > >>
> > > > > >> Thanks for the feedback, I also think that many customers wanted
> > this
> > > > for
> > > > > >> a long time.
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Viktor
> > > > > >>
> > > > > >> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi Viktor,
> > > > > >>> Thanks for taking this task!
> > > > > >>> This is a very nice change as it will allow users to use this
> > tool in
> > > > > >>> many Cloud environments where direct zookeeper access is not
> > > > > >>> available.
> > > > > >>>
> > > > > >>>
> > > > > >>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
> > > > > >>> <vi...@gmail.com> wrote:
> > > > > >>> >
> > > > > >>> > Hi All,
> > > > > >>> >
> > > > > >>> > This is the continuation of the old KIP-375 with the same
> > title:
> > > > > >>> >
> > > > > >>>
> > > >
> >
> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
> > > > > >>> >
> > > > > >>> > The problem there was that two KIPs were created around the
> > same
> > > > time
> > > > > >>> and I
> > > > > >>> > chose to reorganize mine a bit and give it a new number to
> > avoid
> > > > > >>> > duplication.
> > > > > >>> >
> > > > > >>> > The KIP summary here once again:
> > > > > >>> >
> > > > > >>> > I wrote up a relatively simple KIP about improving the Kafka
> > > > protocol
> > > > > >>> and
> > > > > >>> > the TopicCommand tool to support the new Java based
> > AdminClient and
> > > > > >>> > hopefully to deprecate the Zookeeper side of it.
> > > > > >>> >
> > > > > >>> > I would be happy to receive some opinions about this. In
> > general I
> > > > > >>> think
> > > > > >>> > this would be an important addition as this is one of the few
> > left
> > > > but
> > > > > >>> > important tools that still uses direct Zookeeper connection.
> > > > > >>> >
> > > > > >>> > Here is the link for the KIP:
> > > > > >>> >
> > > > > >>>
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> > > > > >>> >
> > > > > >>> > Cheers,
> > > > > >>> > Viktor
> > > > > >>>
> > > > > >>
> > > >
> >
>
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Kevin Lu <lu...@berkeley.edu>.
Hi Viktor,
+1 to this KIP.
I would very much like to see AdminClient in TopicCommand. This would also
allow us to efficiently implement new features like the "--under-min-isr"
option I proposed in KIP-351
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--under-min-isr+option+to+describe+topics+command>
.
Thanks.
Regards,
Kevin
On Sat, Oct 20, 2018 at 10:52 PM Colin McCabe <cm...@apache.org> wrote:
> Hi Viktor,
>
> Sounds good. If you want to propose a way of improving the metadata
> protocol so that "[deleted]" could be supported, you could probably create
> that KIP in parallel.
>
> The last KIP in that area that I can remember is KIP-142, which didn't get
> adopted (yet?)
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster
>
> There have been other discussions though. In general there are a lot of
> features that would be nice to have in the metadata protocol (pagniation,
> regexes, skip stuff we don't need).
>
> best,
> Colin
>
>
> On Tue, Oct 16, 2018, at 10:11, Viktor Somogyi-Vass wrote:
> > Hi Colin,
> >
> > Thanks, it makes sense and simplifies this KIP tremendously. I'll move
> this
> > section to the rejected alternatives with a note that KIP-142 will have
> > this feature.
> > On a similar note: is there a KIP for describe topics protocol or have
> you
> > been thinking about it? I guess there it's the same problem, we often
> don't
> > want to forward the entire metadata.
> >
> > Viktor
> >
> > On Fri, Oct 12, 2018 at 12:03 PM Colin McCabe <cm...@apache.org>
> wrote:
> >
> > > Hi Viktor,
> > >
> > > Thanks for bumping this thread.
> > >
> > > I think we should just focus on transitioning the TopicCommand to use
> > > AdminClient, and talk about protocol changes in a separate KIP.
> Protocol
> > > changes often involve a lot of discussion. This does mean that we
> couldn't
> > > implement the "list topics under deletion" feature when using
> AdminClient
> > > at the moment. We could add a note to the tool output indicating this.
> > >
> > > We should move the protocol discussion to a separate thread. Probably
> > > also look at KIP-142 as well.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote:
> > > > Hi All,
> > > >
> > > > Would like to bump this as the conversation sank a little bit, but
> more
> > > > importantly I'd like to validate my plans/ideas on extending the
> Metadata
> > > > protocol. I was thinking about two other alternatives, namely:
> > > > 1. Create a ListTopicUnderDeletion protocol. This however would be
> > > > unnecessary: it'd have one very narrow functionality which we can't
> > > extend.
> > > > I'd make sense to have a list topics or describe topics protocol
> where we
> > > > can list/describe topics under deletion but for normal
> listing/describing
> > > > we already use the metadata, so it would be a duplication of
> > > functionality.
> > > > 2. DeleteTopicsResponse could return the topics under deletion if the
> > > > request's argument list is empty which might make sense at the first
> > > look,
> > > > but actually we'd mix the query functionality with the delete
> > > functionality
> > > > which is counterintuitive.
> > > >
> > > > Even though most clients won't need these "limbo" topics (which are
> under
> > > > deletion) in the foreseeable future, it can be considered as part of
> the
> > > > cluster state or metadata and to me it makes sense. Also it doesn't
> have
> > > a
> > > > big overhead in the response size as typically users don't delete
> topics
> > > > too often as far as I experienced.
> > > >
> > > > I'd be happy to receive some ideas/feedback on this.
> > > >
> > > > Cheers,
> > > > Viktor
> > > >
> > > >
> > > > On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass <
> > > viktorsomogyi@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I made an update to the KIP. Just in short:
> > > > > Currently KafkaAdminClient.describeTopics() and
> > > > > KafkaAdminClient.listTopics() uses the Metadata protocol to acquire
> > > topic
> > > > > information. The returned response however won't contain the topics
> > > that
> > > > > are under deletion but couldn't complete yet (for instance because
> of
> > > some
> > > > > replicas offline), therefore it is not possible to implement the
> > > current
> > > > > command's "marked for deletion" feature. To get around this I
> > > introduced
> > > > > some changes in the Metadata protocol.
> > > > >
> > > > > Thanks,
> > > > > Viktor
> > > > >
> > > > > On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass <
> > > > > viktorsomogyi@gmail.com> wrote:
> > > > >
> > > > >> Hi Mickael,
> > > > >>
> > > > >> Thanks for the feedback, I also think that many customers wanted
> this
> > > for
> > > > >> a long time.
> > > > >>
> > > > >> Cheers,
> > > > >> Viktor
> > > > >>
> > > > >> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <
> > > mickael.maison@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi Viktor,
> > > > >>> Thanks for taking this task!
> > > > >>> This is a very nice change as it will allow users to use this
> tool in
> > > > >>> many Cloud environments where direct zookeeper access is not
> > > > >>> available.
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
> > > > >>> <vi...@gmail.com> wrote:
> > > > >>> >
> > > > >>> > Hi All,
> > > > >>> >
> > > > >>> > This is the continuation of the old KIP-375 with the same
> title:
> > > > >>> >
> > > > >>>
> > >
> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
> > > > >>> >
> > > > >>> > The problem there was that two KIPs were created around the
> same
> > > time
> > > > >>> and I
> > > > >>> > chose to reorganize mine a bit and give it a new number to
> avoid
> > > > >>> > duplication.
> > > > >>> >
> > > > >>> > The KIP summary here once again:
> > > > >>> >
> > > > >>> > I wrote up a relatively simple KIP about improving the Kafka
> > > protocol
> > > > >>> and
> > > > >>> > the TopicCommand tool to support the new Java based
> AdminClient and
> > > > >>> > hopefully to deprecate the Zookeeper side of it.
> > > > >>> >
> > > > >>> > I would be happy to receive some opinions about this. In
> general I
> > > > >>> think
> > > > >>> > this would be an important addition as this is one of the few
> left
> > > but
> > > > >>> > important tools that still uses direct Zookeeper connection.
> > > > >>> >
> > > > >>> > Here is the link for the KIP:
> > > > >>> >
> > > > >>>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> > > > >>> >
> > > > >>> > Cheers,
> > > > >>> > Viktor
> > > > >>>
> > > > >>
> > >
>
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Colin McCabe <cm...@apache.org>.
Hi Viktor,
Sounds good. If you want to propose a way of improving the metadata protocol so that "[deleted]" could be supported, you could probably create that KIP in parallel.
The last KIP in that area that I can remember is KIP-142, which didn't get adopted (yet?)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster
There have been other discussions though. In general there are a lot of features that would be nice to have in the metadata protocol (pagniation, regexes, skip stuff we don't need).
best,
Colin
On Tue, Oct 16, 2018, at 10:11, Viktor Somogyi-Vass wrote:
> Hi Colin,
>
> Thanks, it makes sense and simplifies this KIP tremendously. I'll move this
> section to the rejected alternatives with a note that KIP-142 will have
> this feature.
> On a similar note: is there a KIP for describe topics protocol or have you
> been thinking about it? I guess there it's the same problem, we often don't
> want to forward the entire metadata.
>
> Viktor
>
> On Fri, Oct 12, 2018 at 12:03 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Viktor,
> >
> > Thanks for bumping this thread.
> >
> > I think we should just focus on transitioning the TopicCommand to use
> > AdminClient, and talk about protocol changes in a separate KIP. Protocol
> > changes often involve a lot of discussion. This does mean that we couldn't
> > implement the "list topics under deletion" feature when using AdminClient
> > at the moment. We could add a note to the tool output indicating this.
> >
> > We should move the protocol discussion to a separate thread. Probably
> > also look at KIP-142 as well.
> >
> > best,
> > Colin
> >
> >
> > On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote:
> > > Hi All,
> > >
> > > Would like to bump this as the conversation sank a little bit, but more
> > > importantly I'd like to validate my plans/ideas on extending the Metadata
> > > protocol. I was thinking about two other alternatives, namely:
> > > 1. Create a ListTopicUnderDeletion protocol. This however would be
> > > unnecessary: it'd have one very narrow functionality which we can't
> > extend.
> > > I'd make sense to have a list topics or describe topics protocol where we
> > > can list/describe topics under deletion but for normal listing/describing
> > > we already use the metadata, so it would be a duplication of
> > functionality.
> > > 2. DeleteTopicsResponse could return the topics under deletion if the
> > > request's argument list is empty which might make sense at the first
> > look,
> > > but actually we'd mix the query functionality with the delete
> > functionality
> > > which is counterintuitive.
> > >
> > > Even though most clients won't need these "limbo" topics (which are under
> > > deletion) in the foreseeable future, it can be considered as part of the
> > > cluster state or metadata and to me it makes sense. Also it doesn't have
> > a
> > > big overhead in the response size as typically users don't delete topics
> > > too often as far as I experienced.
> > >
> > > I'd be happy to receive some ideas/feedback on this.
> > >
> > > Cheers,
> > > Viktor
> > >
> > >
> > > On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass <
> > viktorsomogyi@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I made an update to the KIP. Just in short:
> > > > Currently KafkaAdminClient.describeTopics() and
> > > > KafkaAdminClient.listTopics() uses the Metadata protocol to acquire
> > topic
> > > > information. The returned response however won't contain the topics
> > that
> > > > are under deletion but couldn't complete yet (for instance because of
> > some
> > > > replicas offline), therefore it is not possible to implement the
> > current
> > > > command's "marked for deletion" feature. To get around this I
> > introduced
> > > > some changes in the Metadata protocol.
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > > On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass <
> > > > viktorsomogyi@gmail.com> wrote:
> > > >
> > > >> Hi Mickael,
> > > >>
> > > >> Thanks for the feedback, I also think that many customers wanted this
> > for
> > > >> a long time.
> > > >>
> > > >> Cheers,
> > > >> Viktor
> > > >>
> > > >> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi Viktor,
> > > >>> Thanks for taking this task!
> > > >>> This is a very nice change as it will allow users to use this tool in
> > > >>> many Cloud environments where direct zookeeper access is not
> > > >>> available.
> > > >>>
> > > >>>
> > > >>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
> > > >>> <vi...@gmail.com> wrote:
> > > >>> >
> > > >>> > Hi All,
> > > >>> >
> > > >>> > This is the continuation of the old KIP-375 with the same title:
> > > >>> >
> > > >>>
> > https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
> > > >>> >
> > > >>> > The problem there was that two KIPs were created around the same
> > time
> > > >>> and I
> > > >>> > chose to reorganize mine a bit and give it a new number to avoid
> > > >>> > duplication.
> > > >>> >
> > > >>> > The KIP summary here once again:
> > > >>> >
> > > >>> > I wrote up a relatively simple KIP about improving the Kafka
> > protocol
> > > >>> and
> > > >>> > the TopicCommand tool to support the new Java based AdminClient and
> > > >>> > hopefully to deprecate the Zookeeper side of it.
> > > >>> >
> > > >>> > I would be happy to receive some opinions about this. In general I
> > > >>> think
> > > >>> > this would be an important addition as this is one of the few left
> > but
> > > >>> > important tools that still uses direct Zookeeper connection.
> > > >>> >
> > > >>> > Here is the link for the KIP:
> > > >>> >
> > > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> > > >>> >
> > > >>> > Cheers,
> > > >>> > Viktor
> > > >>>
> > > >>
> >
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi Colin,
Thanks, it makes sense and simplifies this KIP tremendously. I'll move this
section to the rejected alternatives with a note that KIP-142 will have
this feature.
On a similar note: is there a KIP for describe topics protocol or have you
been thinking about it? I guess there it's the same problem, we often don't
want to forward the entire metadata.
Viktor
On Fri, Oct 12, 2018 at 12:03 PM Colin McCabe <cm...@apache.org> wrote:
> Hi Viktor,
>
> Thanks for bumping this thread.
>
> I think we should just focus on transitioning the TopicCommand to use
> AdminClient, and talk about protocol changes in a separate KIP. Protocol
> changes often involve a lot of discussion. This does mean that we couldn't
> implement the "list topics under deletion" feature when using AdminClient
> at the moment. We could add a note to the tool output indicating this.
>
> We should move the protocol discussion to a separate thread. Probably
> also look at KIP-142 as well.
>
> best,
> Colin
>
>
> On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote:
> > Hi All,
> >
> > Would like to bump this as the conversation sank a little bit, but more
> > importantly I'd like to validate my plans/ideas on extending the Metadata
> > protocol. I was thinking about two other alternatives, namely:
> > 1. Create a ListTopicUnderDeletion protocol. This however would be
> > unnecessary: it'd have one very narrow functionality which we can't
> extend.
> > I'd make sense to have a list topics or describe topics protocol where we
> > can list/describe topics under deletion but for normal listing/describing
> > we already use the metadata, so it would be a duplication of
> functionality.
> > 2. DeleteTopicsResponse could return the topics under deletion if the
> > request's argument list is empty which might make sense at the first
> look,
> > but actually we'd mix the query functionality with the delete
> functionality
> > which is counterintuitive.
> >
> > Even though most clients won't need these "limbo" topics (which are under
> > deletion) in the foreseeable future, it can be considered as part of the
> > cluster state or metadata and to me it makes sense. Also it doesn't have
> a
> > big overhead in the response size as typically users don't delete topics
> > too often as far as I experienced.
> >
> > I'd be happy to receive some ideas/feedback on this.
> >
> > Cheers,
> > Viktor
> >
> >
> > On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass <
> viktorsomogyi@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I made an update to the KIP. Just in short:
> > > Currently KafkaAdminClient.describeTopics() and
> > > KafkaAdminClient.listTopics() uses the Metadata protocol to acquire
> topic
> > > information. The returned response however won't contain the topics
> that
> > > are under deletion but couldn't complete yet (for instance because of
> some
> > > replicas offline), therefore it is not possible to implement the
> current
> > > command's "marked for deletion" feature. To get around this I
> introduced
> > > some changes in the Metadata protocol.
> > >
> > > Thanks,
> > > Viktor
> > >
> > > On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass <
> > > viktorsomogyi@gmail.com> wrote:
> > >
> > >> Hi Mickael,
> > >>
> > >> Thanks for the feedback, I also think that many customers wanted this
> for
> > >> a long time.
> > >>
> > >> Cheers,
> > >> Viktor
> > >>
> > >> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <
> mickael.maison@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Viktor,
> > >>> Thanks for taking this task!
> > >>> This is a very nice change as it will allow users to use this tool in
> > >>> many Cloud environments where direct zookeeper access is not
> > >>> available.
> > >>>
> > >>>
> > >>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
> > >>> <vi...@gmail.com> wrote:
> > >>> >
> > >>> > Hi All,
> > >>> >
> > >>> > This is the continuation of the old KIP-375 with the same title:
> > >>> >
> > >>>
> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
> > >>> >
> > >>> > The problem there was that two KIPs were created around the same
> time
> > >>> and I
> > >>> > chose to reorganize mine a bit and give it a new number to avoid
> > >>> > duplication.
> > >>> >
> > >>> > The KIP summary here once again:
> > >>> >
> > >>> > I wrote up a relatively simple KIP about improving the Kafka
> protocol
> > >>> and
> > >>> > the TopicCommand tool to support the new Java based AdminClient and
> > >>> > hopefully to deprecate the Zookeeper side of it.
> > >>> >
> > >>> > I would be happy to receive some opinions about this. In general I
> > >>> think
> > >>> > this would be an important addition as this is one of the few left
> but
> > >>> > important tools that still uses direct Zookeeper connection.
> > >>> >
> > >>> > Here is the link for the KIP:
> > >>> >
> > >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> > >>> >
> > >>> > Cheers,
> > >>> > Viktor
> > >>>
> > >>
>
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Colin McCabe <cm...@apache.org>.
Hi Viktor,
Thanks for bumping this thread.
I think we should just focus on transitioning the TopicCommand to use AdminClient, and talk about protocol changes in a separate KIP. Protocol changes often involve a lot of discussion. This does mean that we couldn't implement the "list topics under deletion" feature when using AdminClient at the moment. We could add a note to the tool output indicating this.
We should move the protocol discussion to a separate thread. Probably also look at KIP-142 as well.
best,
Colin
On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote:
> Hi All,
>
> Would like to bump this as the conversation sank a little bit, but more
> importantly I'd like to validate my plans/ideas on extending the Metadata
> protocol. I was thinking about two other alternatives, namely:
> 1. Create a ListTopicUnderDeletion protocol. This however would be
> unnecessary: it'd have one very narrow functionality which we can't extend.
> I'd make sense to have a list topics or describe topics protocol where we
> can list/describe topics under deletion but for normal listing/describing
> we already use the metadata, so it would be a duplication of functionality.
> 2. DeleteTopicsResponse could return the topics under deletion if the
> request's argument list is empty which might make sense at the first look,
> but actually we'd mix the query functionality with the delete functionality
> which is counterintuitive.
>
> Even though most clients won't need these "limbo" topics (which are under
> deletion) in the foreseeable future, it can be considered as part of the
> cluster state or metadata and to me it makes sense. Also it doesn't have a
> big overhead in the response size as typically users don't delete topics
> too often as far as I experienced.
>
> I'd be happy to receive some ideas/feedback on this.
>
> Cheers,
> Viktor
>
>
> On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass <vi...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I made an update to the KIP. Just in short:
> > Currently KafkaAdminClient.describeTopics() and
> > KafkaAdminClient.listTopics() uses the Metadata protocol to acquire topic
> > information. The returned response however won't contain the topics that
> > are under deletion but couldn't complete yet (for instance because of some
> > replicas offline), therefore it is not possible to implement the current
> > command's "marked for deletion" feature. To get around this I introduced
> > some changes in the Metadata protocol.
> >
> > Thanks,
> > Viktor
> >
> > On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass <
> > viktorsomogyi@gmail.com> wrote:
> >
> >> Hi Mickael,
> >>
> >> Thanks for the feedback, I also think that many customers wanted this for
> >> a long time.
> >>
> >> Cheers,
> >> Viktor
> >>
> >> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <mi...@gmail.com>
> >> wrote:
> >>
> >>> Hi Viktor,
> >>> Thanks for taking this task!
> >>> This is a very nice change as it will allow users to use this tool in
> >>> many Cloud environments where direct zookeeper access is not
> >>> available.
> >>>
> >>>
> >>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
> >>> <vi...@gmail.com> wrote:
> >>> >
> >>> > Hi All,
> >>> >
> >>> > This is the continuation of the old KIP-375 with the same title:
> >>> >
> >>> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
> >>> >
> >>> > The problem there was that two KIPs were created around the same time
> >>> and I
> >>> > chose to reorganize mine a bit and give it a new number to avoid
> >>> > duplication.
> >>> >
> >>> > The KIP summary here once again:
> >>> >
> >>> > I wrote up a relatively simple KIP about improving the Kafka protocol
> >>> and
> >>> > the TopicCommand tool to support the new Java based AdminClient and
> >>> > hopefully to deprecate the Zookeeper side of it.
> >>> >
> >>> > I would be happy to receive some opinions about this. In general I
> >>> think
> >>> > this would be an important addition as this is one of the few left but
> >>> > important tools that still uses direct Zookeeper connection.
> >>> >
> >>> > Here is the link for the KIP:
> >>> >
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> >>> >
> >>> > Cheers,
> >>> > Viktor
> >>>
> >>
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi All,
Would like to bump this as the conversation sank a little bit, but more
importantly I'd like to validate my plans/ideas on extending the Metadata
protocol. I was thinking about two other alternatives, namely:
1. Create a ListTopicUnderDeletion protocol. This however would be
unnecessary: it'd have one very narrow functionality which we can't extend.
I'd make sense to have a list topics or describe topics protocol where we
can list/describe topics under deletion but for normal listing/describing
we already use the metadata, so it would be a duplication of functionality.
2. DeleteTopicsResponse could return the topics under deletion if the
request's argument list is empty which might make sense at the first look,
but actually we'd mix the query functionality with the delete functionality
which is counterintuitive.
Even though most clients won't need these "limbo" topics (which are under
deletion) in the foreseeable future, it can be considered as part of the
cluster state or metadata and to me it makes sense. Also it doesn't have a
big overhead in the response size as typically users don't delete topics
too often as far as I experienced.
I'd be happy to receive some ideas/feedback on this.
Cheers,
Viktor
On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass <vi...@gmail.com>
wrote:
> Hi All,
>
> I made an update to the KIP. Just in short:
> Currently KafkaAdminClient.describeTopics() and
> KafkaAdminClient.listTopics() uses the Metadata protocol to acquire topic
> information. The returned response however won't contain the topics that
> are under deletion but couldn't complete yet (for instance because of some
> replicas offline), therefore it is not possible to implement the current
> command's "marked for deletion" feature. To get around this I introduced
> some changes in the Metadata protocol.
>
> Thanks,
> Viktor
>
> On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass <
> viktorsomogyi@gmail.com> wrote:
>
>> Hi Mickael,
>>
>> Thanks for the feedback, I also think that many customers wanted this for
>> a long time.
>>
>> Cheers,
>> Viktor
>>
>> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <mi...@gmail.com>
>> wrote:
>>
>>> Hi Viktor,
>>> Thanks for taking this task!
>>> This is a very nice change as it will allow users to use this tool in
>>> many Cloud environments where direct zookeeper access is not
>>> available.
>>>
>>>
>>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
>>> <vi...@gmail.com> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > This is the continuation of the old KIP-375 with the same title:
>>> >
>>> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
>>> >
>>> > The problem there was that two KIPs were created around the same time
>>> and I
>>> > chose to reorganize mine a bit and give it a new number to avoid
>>> > duplication.
>>> >
>>> > The KIP summary here once again:
>>> >
>>> > I wrote up a relatively simple KIP about improving the Kafka protocol
>>> and
>>> > the TopicCommand tool to support the new Java based AdminClient and
>>> > hopefully to deprecate the Zookeeper side of it.
>>> >
>>> > I would be happy to receive some opinions about this. In general I
>>> think
>>> > this would be an important addition as this is one of the few left but
>>> > important tools that still uses direct Zookeeper connection.
>>> >
>>> > Here is the link for the KIP:
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
>>> >
>>> > Cheers,
>>> > Viktor
>>>
>>
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi All,
I made an update to the KIP. Just in short:
Currently KafkaAdminClient.describeTopics() and
KafkaAdminClient.listTopics() uses the Metadata protocol to acquire topic
information. The returned response however won't contain the topics that
are under deletion but couldn't complete yet (for instance because of some
replicas offline), therefore it is not possible to implement the current
command's "marked for deletion" feature. To get around this I introduced
some changes in the Metadata protocol.
Thanks,
Viktor
On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass <vi...@gmail.com>
wrote:
> Hi Mickael,
>
> Thanks for the feedback, I also think that many customers wanted this for
> a long time.
>
> Cheers,
> Viktor
>
> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
>> Hi Viktor,
>> Thanks for taking this task!
>> This is a very nice change as it will allow users to use this tool in
>> many Cloud environments where direct zookeeper access is not
>> available.
>>
>>
>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
>> <vi...@gmail.com> wrote:
>> >
>> > Hi All,
>> >
>> > This is the continuation of the old KIP-375 with the same title:
>> >
>> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
>> >
>> > The problem there was that two KIPs were created around the same time
>> and I
>> > chose to reorganize mine a bit and give it a new number to avoid
>> > duplication.
>> >
>> > The KIP summary here once again:
>> >
>> > I wrote up a relatively simple KIP about improving the Kafka protocol
>> and
>> > the TopicCommand tool to support the new Java based AdminClient and
>> > hopefully to deprecate the Zookeeper side of it.
>> >
>> > I would be happy to receive some opinions about this. In general I think
>> > this would be an important addition as this is one of the few left but
>> > important tools that still uses direct Zookeeper connection.
>> >
>> > Here is the link for the KIP:
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
>> >
>> > Cheers,
>> > Viktor
>>
>
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi Mickael,
Thanks for the feedback, I also think that many customers wanted this for a
long time.
Cheers,
Viktor
On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison <mi...@gmail.com>
wrote:
> Hi Viktor,
> Thanks for taking this task!
> This is a very nice change as it will allow users to use this tool in
> many Cloud environments where direct zookeeper access is not
> available.
>
>
> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
> <vi...@gmail.com> wrote:
> >
> > Hi All,
> >
> > This is the continuation of the old KIP-375 with the same title:
> >
> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
> >
> > The problem there was that two KIPs were created around the same time
> and I
> > chose to reorganize mine a bit and give it a new number to avoid
> > duplication.
> >
> > The KIP summary here once again:
> >
> > I wrote up a relatively simple KIP about improving the Kafka protocol and
> > the TopicCommand tool to support the new Java based AdminClient and
> > hopefully to deprecate the Zookeeper side of it.
> >
> > I would be happy to receive some opinions about this. In general I think
> > this would be an important addition as this is one of the few left but
> > important tools that still uses direct Zookeeper connection.
> >
> > Here is the link for the KIP:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> >
> > Cheers,
> > Viktor
>
Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Hi Viktor,
Thanks for taking this task!
This is a very nice change as it will allow users to use this tool in
many Cloud environments where direct zookeeper access is not
available.
On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
<vi...@gmail.com> wrote:
>
> Hi All,
>
> This is the continuation of the old KIP-375 with the same title:
> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
>
> The problem there was that two KIPs were created around the same time and I
> chose to reorganize mine a bit and give it a new number to avoid
> duplication.
>
> The KIP summary here once again:
>
> I wrote up a relatively simple KIP about improving the Kafka protocol and
> the TopicCommand tool to support the new Java based AdminClient and
> hopefully to deprecate the Zookeeper side of it.
>
> I would be happy to receive some opinions about this. In general I think
> this would be an important addition as this is one of the few left but
> important tools that still uses direct Zookeeper connection.
>
> Here is the link for the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
>
> Cheers,
> Viktor