You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by David Jacot <dj...@confluent.io> on 2020/12/14 16:10:22 UTC
[DISCUSS] KIP-700: Add Describe Cluster API
Hi all,
I'd like to propose a small KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
Please let me know what you think.
Best,
David
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by David Jacot <dj...@confluent.io>.
Hi all,
Thank you for your feedback. I have updated the KIP to incorporate it. I
have
changed the following:
* Added a few ideas in the future work section about metadata which could be
added in the future. This strengthens the motivation of the KIP.
* Added a section about the Metadata API to clean up the cluster authorized
fields in the request/response.
I think that the updated version should address all your comments. Please
let
me know if you have any further comments. I will start a vote later this
week if
I don't hear from you.
Best,
David
On Fri, Dec 18, 2020 at 10:52 PM Ismael Juma <is...@juma.me.uk> wrote:
> Thanks for the KIP. I agree with keeping it simple in the first iteration.
> If we can include the authorized operations clean-up without too much
> effort, that would be great. But if it increases the scope a lot, I'd punt
> it to the future.
>
> Ismael
>
> On Mon, Dec 14, 2020 at 8:10 AM David Jacot <dj...@confluent.io> wrote:
>
> > Hi all,
> >
> > I'd like to propose a small KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
> >
> > Please let me know what you think.
> >
> > Best,
> > David
> >
>
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks for the KIP. I agree with keeping it simple in the first iteration.
If we can include the authorized operations clean-up without too much
effort, that would be great. But if it increases the scope a lot, I'd punt
it to the future.
Ismael
On Mon, Dec 14, 2020 at 8:10 AM David Jacot <dj...@confluent.io> wrote:
> Hi all,
>
> I'd like to propose a small KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
>
> Please let me know what you think.
>
> Best,
> David
>
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by David Jacot <dj...@confluent.io>.
Hi Colin,
Thanks for your feedback.
That is correct. I mention this in the future work section. I thought that
we
could remove it when we could also remove TopicAuthorizedOperations
in order to remain consistent in the API. We can also do it immediately.
I don't have a strong preference for any ways. What do you think?
Best,
David
On Fri, Dec 18, 2020 at 1:51 PM David Jacot <dj...@confluent.io> wrote:
> Hi Ryan,
>
> Thanks for your feedback. That's an interesting idea but that raises more
> questions. At the moment, `describeCluster` only requires to be
> authenticated
> (if authentication is enabled) to get the basic information about the
> cluster.
> If we add the JMX port, is it something that we would provide without
> requiring
> more permissions? I am not sure about this.
>
> I lean towards keeping this KIP focused on introducing the new API and to
> add new information with separate KIPs. There might be more information
> that we want to add as part of KIP-500.
>
> I will be happy to hear what other members of the community think about
> this.
>
> Best,
> David
>
> On Thu, Dec 17, 2020 at 5:57 AM Colin McCabe <cm...@apache.org> wrote:
>
>> Hi David,
>>
>> This seems reasonable. It would be nice to have an API specifically for
>> describeCluster, so that we could extend this API without adding more
>> fields to the already large MetadataRequest.
>>
>> As you mention in the KIP, KIP-700 would allow us to deprecate
>> MetadataRequest#ClusterAuthorizedOperations. So it seems like this KIP
>> should specify a new version of MetadataRequest where the related fields
>> are absent, right?
>>
>> best,
>> Colin
>>
>>
>> On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
>> > Hi all,
>> >
>> > I'd like to propose a small KIP:
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
>> >
>> > Please let me know what you think.
>> >
>> > Best,
>> > David
>> >
>>
>
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by Gwen Shapira <gw...@confluent.io>.
On Fri, Dec 18, 2020, 12:48 PM Colin McCabe <cm...@apache.org> wrote:
> On Fri, Dec 18, 2020, at 08:32, Gwen Shapira wrote:
> > Agree. Once we have the basic API we can decide what belongs and what
> > doesn't. I remember requests for broker status, version, and various
> > other things. Considering all those options together will be too much
> > at this point.
> >
>
> Hi Gwen,
>
> I agree that there are a lot of potential improvements, and we want to
> avoid putting too much in this KIP. But I still think it would be helpful
> to have one or two new user-visible things in this KIP, just to demonstrate
> why we need a new RPC. Or if that's too much, can we identify a few things
> in a "future work" section? I think it adds a lot to the motivation for
> this KIP. After all, if you don't think we're going to add anything more
> to describeCluster, there is not a strong case for a new RPC.
>
I think the complexity of MetadataResponse alone justifies a cleaner API,
but if adding JMX port (or broker version, or broker status) is a must-have
in your mind, I have no objection - I have few use-cases for the
broker-status response.
> > One thing that may be worth considering is whether you want to include
> > not just the controller ID but also its epoch. This will allow
> > resolving cases where brokers respond with outdated information. I
> > haven't thought it through, so maybe it doesn't make sense here, but I
> > was wondering if this was considered.
>
> I think we should hold off on adding more epochs right now until we have a
> more general solution. With KIP-500 we can potentially put an epoch on the
> whole metadata response, which would allow us to have read-after-write
> consistency for metadata-- something people have wanted for a while.
>
Oh, this is wonderful news! We can add the metadata epoch separately (I did
not suggest a new epoch, I suggested publishing the existing broker and
controller epochs in the response, but we can add this later... as I just
argued above).
>
> best,
> Colin
>
>
> >
> > On Fri, Dec 18, 2020 at 4:51 AM David Jacot <dj...@confluent.io> wrote:
> > >
> > > Hi Ryan,
> > >
> > > Thanks for your feedback. That's an interesting idea but that raises
> more
> > > questions. At the moment, `describeCluster` only requires to be
> > > authenticated
> > > (if authentication is enabled) to get the basic information about the
> > > cluster.
> > > If we add the JMX port, is it something that we would provide without
> > > requiring
> > > more permissions? I am not sure about this.
> > >
> > > I lean towards keeping this KIP focused on introducing the new API and
> to
> > > add new information with separate KIPs. There might be more information
> > > that we want to add as part of KIP-500.
> > >
> > > I will be happy to hear what other members of the community think about
> > > this.
> > >
> > > Best,
> > > David
> > >
> > > On Thu, Dec 17, 2020 at 5:57 AM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > Hi David,
> > > >
> > > > This seems reasonable. It would be nice to have an API specifically
> for
> > > > describeCluster, so that we could extend this API without adding more
> > > > fields to the already large MetadataRequest.
> > > >
> > > > As you mention in the KIP, KIP-700 would allow us to deprecate
> > > > MetadataRequest#ClusterAuthorizedOperations. So it seems like this
> KIP
> > > > should specify a new version of MetadataRequest where the related
> fields
> > > > are absent, right?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
> > > > > Hi all,
> > > > >
> > > > > I'd like to propose a small KIP:
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
> > > > >
> > > > > Please let me know what you think.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > >
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Dec 18, 2020, at 08:32, Gwen Shapira wrote:
> Agree. Once we have the basic API we can decide what belongs and what
> doesn't. I remember requests for broker status, version, and various
> other things. Considering all those options together will be too much
> at this point.
>
Hi Gwen,
I agree that there are a lot of potential improvements, and we want to avoid putting too much in this KIP. But I still think it would be helpful to have one or two new user-visible things in this KIP, just to demonstrate why we need a new RPC. Or if that's too much, can we identify a few things in a "future work" section? I think it adds a lot to the motivation for this KIP. After all, if you don't think we're going to add anything more to describeCluster, there is not a strong case for a new RPC.
> One thing that may be worth considering is whether you want to include
> not just the controller ID but also its epoch. This will allow
> resolving cases where brokers respond with outdated information. I
> haven't thought it through, so maybe it doesn't make sense here, but I
> was wondering if this was considered.
I think we should hold off on adding more epochs right now until we have a more general solution. With KIP-500 we can potentially put an epoch on the whole metadata response, which would allow us to have read-after-write consistency for metadata-- something people have wanted for a while.
best,
Colin
>
> On Fri, Dec 18, 2020 at 4:51 AM David Jacot <dj...@confluent.io> wrote:
> >
> > Hi Ryan,
> >
> > Thanks for your feedback. That's an interesting idea but that raises more
> > questions. At the moment, `describeCluster` only requires to be
> > authenticated
> > (if authentication is enabled) to get the basic information about the
> > cluster.
> > If we add the JMX port, is it something that we would provide without
> > requiring
> > more permissions? I am not sure about this.
> >
> > I lean towards keeping this KIP focused on introducing the new API and to
> > add new information with separate KIPs. There might be more information
> > that we want to add as part of KIP-500.
> >
> > I will be happy to hear what other members of the community think about
> > this.
> >
> > Best,
> > David
> >
> > On Thu, Dec 17, 2020 at 5:57 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi David,
> > >
> > > This seems reasonable. It would be nice to have an API specifically for
> > > describeCluster, so that we could extend this API without adding more
> > > fields to the already large MetadataRequest.
> > >
> > > As you mention in the KIP, KIP-700 would allow us to deprecate
> > > MetadataRequest#ClusterAuthorizedOperations. So it seems like this KIP
> > > should specify a new version of MetadataRequest where the related fields
> > > are absent, right?
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
> > > > Hi all,
> > > >
> > > > I'd like to propose a small KIP:
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
> > > >
> > > > Please let me know what you think.
> > > >
> > > > Best,
> > > > David
> > > >
> > >
>
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by Gwen Shapira <gw...@confluent.io>.
Agree. Once we have the basic API we can decide what belongs and what
doesn't. I remember requests for broker status, version, and various
other things. Considering all those options together will be too much
at this point.
One thing that may be worth considering is whether you want to include
not just the controller ID but also its epoch. This will allow
resolving cases where brokers respond with outdated information. I
haven't thought it through, so maybe it doesn't make sense here, but I
was wondering if this was considered.
On Fri, Dec 18, 2020 at 4:51 AM David Jacot <dj...@confluent.io> wrote:
>
> Hi Ryan,
>
> Thanks for your feedback. That's an interesting idea but that raises more
> questions. At the moment, `describeCluster` only requires to be
> authenticated
> (if authentication is enabled) to get the basic information about the
> cluster.
> If we add the JMX port, is it something that we would provide without
> requiring
> more permissions? I am not sure about this.
>
> I lean towards keeping this KIP focused on introducing the new API and to
> add new information with separate KIPs. There might be more information
> that we want to add as part of KIP-500.
>
> I will be happy to hear what other members of the community think about
> this.
>
> Best,
> David
>
> On Thu, Dec 17, 2020 at 5:57 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi David,
> >
> > This seems reasonable. It would be nice to have an API specifically for
> > describeCluster, so that we could extend this API without adding more
> > fields to the already large MetadataRequest.
> >
> > As you mention in the KIP, KIP-700 would allow us to deprecate
> > MetadataRequest#ClusterAuthorizedOperations. So it seems like this KIP
> > should specify a new version of MetadataRequest where the related fields
> > are absent, right?
> >
> > best,
> > Colin
> >
> >
> > On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
> > > Hi all,
> > >
> > > I'd like to propose a small KIP:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
> > >
> > > Please let me know what you think.
> > >
> > > Best,
> > > David
> > >
> >
--
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by David Jacot <dj...@confluent.io>.
Hi Ryan,
Thanks for your feedback. That's an interesting idea but that raises more
questions. At the moment, `describeCluster` only requires to be
authenticated
(if authentication is enabled) to get the basic information about the
cluster.
If we add the JMX port, is it something that we would provide without
requiring
more permissions? I am not sure about this.
I lean towards keeping this KIP focused on introducing the new API and to
add new information with separate KIPs. There might be more information
that we want to add as part of KIP-500.
I will be happy to hear what other members of the community think about
this.
Best,
David
On Thu, Dec 17, 2020 at 5:57 AM Colin McCabe <cm...@apache.org> wrote:
> Hi David,
>
> This seems reasonable. It would be nice to have an API specifically for
> describeCluster, so that we could extend this API without adding more
> fields to the already large MetadataRequest.
>
> As you mention in the KIP, KIP-700 would allow us to deprecate
> MetadataRequest#ClusterAuthorizedOperations. So it seems like this KIP
> should specify a new version of MetadataRequest where the related fields
> are absent, right?
>
> best,
> Colin
>
>
> On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
> > Hi all,
> >
> > I'd like to propose a small KIP:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
> >
> > Please let me know what you think.
> >
> > Best,
> > David
> >
>
Re: [DISCUSS] KIP-700: Add Describe Cluster API
Posted by Colin McCabe <cm...@apache.org>.
Hi David,
This seems reasonable. It would be nice to have an API specifically for describeCluster, so that we could extend this API without adding more fields to the already large MetadataRequest.
As you mention in the KIP, KIP-700 would allow us to deprecate MetadataRequest#ClusterAuthorizedOperations. So it seems like this KIP should specify a new version of MetadataRequest where the related fields are absent, right?
best,
Colin
On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
> Hi all,
>
> I'd like to propose a small KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
>
> Please let me know what you think.
>
> Best,
> David
>